GAMUT

GAMUT🌈: Two-Level Meta-Rubrics for Evaluating Open-Ended Generation 📄 **Paper:** https://arxiv.org/abs/2607.19322 GAMUT** *(Grounded Assessment of Multimodal Factuality)* is a multimodal everyday deep-research benchmark** and a **two-level meta-rubric framework** for evaluating open-ended, long-form generation. Two-level meta-rubric framework** — resolving a tension in **rubric-based evaluation**. Judging open-ended generation requires **structure**, because a complete answer often does not decompose into independent, individually required facts: answers may draw from **open-ended pools** of valid alternatives where no single item is strictly required (the *supporting ingredients* in the figure); or describe **ordered processes** where the sequence itself matters (the *preparation steps*). But LLM judges are far more reliable and lower-variance scoring **flat, binary checks** — the very structure needed to *describe* a complete answer is what makes it hard to *grade* one consistently. GAMUT resolves this with a two-level rubric: a **structured meta-rubric** (left) for authoring the grading criteria — what a complete answer requires — which is then **mechanically compiled** into **flat binary checks** (right) that a judge scores one at a time. Everyday deep-research benchmark.** **1,813** image-grounded questions across 10 diverse domains, grounded in real wearable imagery. Complex, real-world questions that require multi-step information gathering and multi-paragraph answer synthesis. Questions and rubrics are built through a multi-round human + LLM annotation process, each rubric backed by cited web evidence and verified by expert annotators. Challenging and discriminative.** Evaluated 14 proprietary and open-weight models; the best (Gemini 3.1 Pro) reaches only **58.7%**, so the benchmark is far from saturated. Scores spread widely with large gaps between models, and the ranking is **robust to the choice of judge** (Gemini and Claude judges agree on an identical r