The study evaluates LMMs on the OB rejoining task using a new dataset (OBI-Rejoin). Models were assessed on their ability to determine if two fragments are rejoinable, using both 'yes-or-no' queries (measuring accuracy) and 'how' queries (measuring rejoinable probability with acc@k). GPT-4o and Gemini 1.5 Pro achieved acc@10 scores over 76%, suggesting a potential for assisting scholars by narrowing down candidates. However, open-source models performed poorly, with average acc@1, acc@5, and acc@10 scores of 3.85%, 11.39%, and 18.65%, respectively.
Evidence
correlational
Key metric
acc@10: GPT-4o 78.47%, Gemini 1.5 Pro 76.67%; acc@1: GPT-4o 29.13%, Gemini 1.5 Pro 24.88%
Caveat
Performance is significantly better with 'how' queries (probabilistic output) than 'yes-or-no' queries, suggesting that the format of the answer greatly influences the results. Open-source models remain far from practical utility.