IC-343GPT-4o, Claude-3.5 Sonnet, and GeminiPro-1.5 score below BigDocs-trained open models on BigDocs-Bench tasks requiring long structured code generation
Juan A. Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte Suresh, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-Andre Noel, Mats Leon Richter, Saverio Vadacchino, Shubham Agarwal, Sanket Biswas, Sara Shanian, Ying Zhang, Sathwik Tejaswi Madhusudhan, Joao Monteiro, Krishnamurthy Dj Dvijotham, Torsten Scholak, Nicolas Chapados, Sepideh Kharaghani, Sean Hughes, M. Özsu, Siva Reddy, Marco Pedersoli, Yoshua Bengio, Christopher Pal, Issam H. Laradji, Spandana Gella, Perouz Taslakian, David Vazquez, Sai Rajeswar
The paper evaluates released models on BigDocs-Bench, a suite of 10 tasks involving generating HTML, LaTeX, SVG, and flowchart code from images, plus GUI reasoning. GPT-4o achieves the highest average among non-BigDocs models at 36.84, but all BigDocs-trained open models (49.52 to 56.34) outperform it. Claude-3.5 Sonnet scores 30.55, GeminiPro-1.5 scores 32.32, Qwen2-VL-72B scores 31.36, and Llama-3.2-90B scores 25.66. The gap is largest on flow generation and GUI reasoning tasks.
BigDocs-Bench is a new benchmark introduced by this paper; the tasks may not be representative of general document understanding. Some tasks required 1-shot prompts for certain models (marked with dagger).