notableReviewed Sat Oct 31

SIGIR research finds substantial source divergence between Google Search, AI Overviews and Gemini

An original SIGIR 2026-accepted study compared 11,500 user queries across Google Search, AI Overviews and Gemini Flash 2.5. Authors report that AI Overviews appeared for 51.5% of their sampled queries, while cross-system retrieved-source overlap averaged below 0.2 Jaccard similarity; AIO results also varied across repeat runs and minor query edits.

Context

An original SIGIR 2026-accepted study compared 11,500 user queries across Google Search, AI Overviews and Gemini Flash 2.5. Authors report that AI Overviews appeared for 51.5% of their sampled queries, while cross-system retrieved-source overlap averaged below 0.2 Jaccard similarity; AIO results also varied across repeat runs and minor query edits.

What changed

Original comparative benchmark quantifies cross-surface retrieval differences and prompt sensitivity, rather than assuming that organic Search results represent generative search source selection.

Why it matters

AI discovery measurement must distinguish organic Google rankings, AI Overview citations and Gemini results, repeat query measurements and document crawler policy separately.

Visibility OS interpretation

Visibility OS interprets the study as evidence against using a single Google Search rank as a surrogate for AI answer visibility. The reported source overlap and AIO rate are sample- and configuration-specific; crawler blocking correlation is not proof of a causal ranking effect.

What remains uncertain

How do the results change with newer Gemini and Search versions, geography and time? Which factors cause source overlap differences? Can crawler preference effects be isolated experimentally?

Evidence claims

Emerging Evidenceintroduces

In a 11,500-query comparison of Google Search, AI Overviews and Gemini Flash 2.5, Grossman and colleagues reported AI Overviews for 51.5% of sampled queries and average Jaccard overlap of retrieved sources below 0.2 across the compared systems.