notableReviewed Sun Nov 01

Deep literature search study separates citation coverage from topical relevance

Original research on a 250-paper computer-science benchmark finds that breadth-first bibliography expansion greatly improves scholarly retrieval recall and that reference-list overlap alone does not capture topical citation relevance.

Context

Original research on a 250-paper computer-science benchmark finds that breadth-first bibliography expansion greatly improves scholarly retrieval recall and that reference-list overlap alone does not capture topical citation relevance.

What changed

A full-document, multi-query scholarly retrieval pipeline combines API seeds, bibliography expansion and reranking; the authors separately measure recall against human references, LLM-judged relevance, diversity and co-authorship proximity.

Why it matters

AI discovery measurement should separate whether sources were found from how they were ranked and judged relevant; existing human citations are not a complete or unambiguous ground truth for evaluating retrieval.

Visibility OS interpretation

Visibility OS interpretation: this controlled scholarly-search result suggests measuring retrieval coverage, citation relevance and evidence diversity separately. It is not a test of live commercial answer-engine citation ranking.

What remains uncertain

Will independent human judges reproduce the LLM-judged topical relevance gap? Do findings generalize beyond arXiv computer-science papers and beyond scholarly search? How much of recall improvement is due to larger candidate budgets?

Evidence claims

Emerging Evidenceintroduces

In the authors' RollingEval-Jun25 benchmark of 250 computer-science arXiv papers, vanilla scholarly API retrieval remained around or below 15% reference-list recall at very large candidate counts, whereas bibliography-expanding Deep Research variants exceeded 80% recall by K=10,000.