Context
MERRIN is a human-annotated 162-question benchmark testing search agents' ability to select appropriate modalities and sources, then reason over conflicting and noisy open-web evidence.
What changed
The authors evaluated ten model configurations under no-search, native-search and agentic multimodal search. They report 22.3% mean accuracy across runs and 40.1% for the strongest agentic configuration; additional queries and page visits did not reliably improve outcomes.
Why it matters
Multimodal AI discoverability requires that agents retrieve and correctly interpret the right source modality, not merely find a webpage. More query fan-out can introduce distracting evidence and impair final grounding.
Visibility OS interpretation
Visibility OS interpretation: evidence-quality, modality-selection and answer-convergence measures should complement citation and retrieval counts in agentic discovery audits. This does not imply a change to commercial search ranking algorithms.
What remains uncertain
How do results transfer beyond the 162 curated questions and Google-centered search? How much is caused by modality retrieval versus multimodal reasoning? Will changing web pages affect benchmark reproducibility?