notableReviewed Mon Nov 02

Retrieved but Not Read study separates search-result retrieval from source reading and citation faithfulness

A 2026 research preprint tests how search-augmented LLMs use current evidence across 225 changed facts and reports that snippet-only answers correlate with more stale answers and unsupported citations than answers citing opened pages.

Context

A 2026 research preprint tests how search-augmented LLMs use current evidence across 225 changed facts and reports that snippet-only answers correlate with more stale answers and unsupported citations than answers citing opened pages.

What changed

The authors trace whether assistants open retrieved source pages before answering and quantify stale answers and citation faithfulness by reading depth, using deployed and frozen-retrieval experiments.

Why it matters

Source retrieval and even citation do not guarantee that an AI answer read or faithfully used the underlying evidence. This adds a measurable distinction between discovery, source opening, evidence use and answer support.

Visibility OS interpretation

Visibility OS interpretation: measure the source-to-answer funnel as retrieval, page reading, evidence incorporation and citation faithfulness; do not treat citation counts as proof of answer influence.

What remains uncertain

Do the associations persist across other proprietary engines and natural free-form tasks? Would randomized page-opening interventions reduce failures? Can mechanistic claims be confirmed causally?

Evidence claims

Emerging Evidencesupports

Walters, Chen and Toni evaluated 225 post-cutoff changed facts across six domains using one proprietary search-enabled model and six open-weight model configurations under controlled retrieval.

Emerging Evidencesupports

In the study's pooled frozen-retrieval evaluation, 6.4% of snippet-only episodes (87/1366) produced stale answers versus 1.3% of episodes citing an opened page (8/632); this is an observational association, not a causal estimate.

Emerging Evidencesupports

In the same study, 6.4% of graded snippet-only answers (62/962) had unfaithful citations versus 2.3% of graded opened-and-cited answers (29/1258).

Retrieved but Not Read study separates search-result retrieval from source reading and citation faithfulness | Visibility OS