notableReviewed Sun Nov 01

SearchHorizonBench separates answer success from the marginal value of another agent search

Ning, Li and Kong introduce SearchHorizonBench, a paired-continuation evaluation of SEARCH versus THINK from the same agent state, with 45,632 executions across 3,168 questions. The study finds that calibration of overall success can improve while predicting the incremental benefit of search remains difficult.

Context

Ning, Li and Kong introduce SearchHorizonBench, a paired-continuation evaluation of SEARCH versus THINK from the same agent state, with 45,632 executions across 3,168 questions. The study finds that calibration of overall success can improve while predicting the incremental benefit of search remains difficult.

What changed

The authors released an evaluation framework that separates question-level answer success, marginal search gain and decision utility with explicit retrieval costs and repeated outcomes.

Why it matters

A relevant page may not enter an AI agent's evidence pool because the agent decides not to search again. Retrieval-trigger and stopping decisions must be measured separately from ranking and final citations.

Visibility OS interpretation

Visibility OS interpretation: missing AI citations may originate before source selection, at the decision to search or stop. These are controlled research-agent findings, not observations of deployed ChatGPT Search, Google AI Mode or Perplexity.

What remains uncertain

How well do search-gain predictors transfer to production agents, different query populations and real-world cost structures? Can marginal retrieval decisions be audited externally?

Evidence claims

supports

SearchHorizonBench reports 45,632 distinct executions and 8,880 forecasts across 3,168 questions, using paired SEARCH and THINK continuations from the same agent state.

supports

On 1,632 held-out questions, SearchHorizonBench reports a 16.2% reduction in Brier error after historical calibration, while tested historical estimators explain less than 2% of question-specific search-gain variance.