notableReviewed Sun Nov 01

INSPIRE benchmark measures exposure, selection and ranking losses in research agents

INSPIRE evaluates scientific search agents on 476 solution-redacted computer-science research targets. Logged search trajectories separate relevant evidence exposure, retention in final selections and final ranking, with the largest observed deficit occurring before source selection.

Context

INSPIRE evaluates scientific search agents on 476 solution-redacted computer-science research targets. Logged search trajectories separate relevant evidence exposure, retention in final selections and final ranking, with the largest observed deficit occurring before source selection.

What changed

The authors introduced a date-gated open-corpus benchmark that scores exposure, selection and ranking against graded cited antecedents on the same trajectory and tests a matched hindsight-supervision intervention.

Why it matters

A source that an agent never retrieves cannot be cited, and retrieved evidence can still be omitted or poorly ranked. Citation visibility measurement should distinguish these failure stages.

Visibility OS interpretation

Visibility OS interpretation: adapt the exposure-selection-ranking funnel to diagnose AI discovery. The findings concern scholarly search tasks and do not establish internal behavior of commercial consumer AI search systems.

What remains uncertain

How much do citation-lineage labels miss relevant but uncited sources? Will exposure deficits persist across domains and live web discovery? How sensitive are findings to search interfaces and model versions?

Evidence claims

introduces

In INSPIRE's 476-target scholarly search evaluation, the strongest tested agent achieved 0.284 nDCG@10, and the rate of targets where no graded antecedent was exposed ranged from 17.4% to 56.1% across agents; the largest observed deficit was before source selection.