ROME

🏠Project Page & Leaderboard | 💻Code | 📄Paper | 🤗Data | 🤗Evaluation Response This repository contains a visual reasoning benchmark named ROME from the paper FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions. ROME include 8 subtasks (281 high-quality questions in total). Each sample has been verified to ensure that images are necessary to answer correctly: Academic * questions from college courses Diagrams * charts and figures collected from recent scientific papers, reports, or blog posts Puzzles and games * Raven's Progressive Matrices, rebus puzzles, and gameplay Memes * recreated memes Geo * geolocation inference Recognition * fine-grained recognition Multi-image * find-the-difference tasks or video frame reordering. Spatial * relative positions, depths/distances, heights, etc We plot the scatter of overall accuracy vs. token consumption for visual problems: 📰 News [09/10/2025]** 🚀 First release of Rome. We released our leaderboard on **30+ LLMs and MLLMs** that we have tested so far. We also released all model responses across 4 runs of evaluations(Model responses). 👋 Evaluation Findings We conduct a moderate-scale contamination-free (hopefully) evaluation of current LRMs with some preliminary findings. To highlight a few: With a few more thousands of thinking tokens, LRMs consistently show superior performance than their non-thinking counterparts in solving challenging problems or puzzles. LRMs achieving high metrics on previous benchmarks are also showing within-task generalization, thus benchmark saturation should not always be attributed to contamination or memorization. Many recently findings from LRMs might be model-specific or data-specific. For instance, we observe slight degradation in instruction following only on Claude Sonnet 4 and DeepSeek series, and on Qwen 3 and DeepSeek LRMs in multi-turn settings. There exists degradation in multi-turn settings for some LRMs agai