notableReviewed Sun Nov 01

SAGE benchmark finds agent query wording changes retrieval effectiveness

SAGE introduces 1,200 scientific-literature questions across four domains and approximately 200,000 papers. In a fixed-corpus DR Tulu comparison, lexical BM25 surpassed tested semantic retrievers on short-form exact match, while a semantic retriever led on open-ended weighted recall.

Context

SAGE introduces 1,200 scientific-literature questions across four domains and approximately 200,000 papers. In a fixed-corpus DR Tulu comparison, lexical BM25 surpassed tested semantic retrievers on short-form exact match, while a semantic retriever led on open-ended weighted recall.

What changed

The authors introduced a multi-domain benchmark covering short-form and open-ended agentic research queries, compared retrieval backends under a fixed agent, and tested corpus keyword and metadata enrichment.

Why it matters

Which sources enter an AI agent's evidence pool depends on its intermediate query formulations and retrieval configuration; semantic similarity alone does not predict exposure across tasks.

Visibility OS interpretation

Visibility OS interpretation: query fan-out, lexical/semantic retriever alignment, source exposure, and final answer citation are distinct stages. The benchmark informs measurement design but does not reveal production ChatGPT, Gemini, or Perplexity citation ranking.

What remains uncertain

Do these query-retriever interactions generalize to heterogeneous public web corpora, commerce tasks, and other agent backbones? Can independent groups reproduce the reported gains with human-written queries?

Evidence claims

introduces

In SAGE's controlled DR Tulu corpus-search experiment at top-k=10, BM25 achieved 81.2 average short-form exact match versus 63.0 for gte-Qwen and 49.3 for ReasonIR; on open-ended weighted recall, gte-Qwen scored 33.0 versus BM25 at 30.7.

SAGE benchmark finds agent query wording changes retrieval effectiveness | Visibility OS