notableReviewed Mon Nov 02

EMNLP study traces deep-research citation errors to agent orchestration

Peer-reviewed research by Hirsch and colleagues evaluates citation faithfulness in three open-source multi-agent deep-research systems using local diagnostics of agent invocations. In AI-Q, the authors attribute 84.7% of final-report errors to the orchestrator; roughly 31% of those orchestrator errors are hallucinations and the remainder citation mistakes.

Context

Peer-reviewed research by Hirsch and colleagues evaluates citation faithfulness in three open-source multi-agent deep-research systems using local diagnostics of agent invocations. In AI-Q, the authors attribute 84.7% of final-report errors to the orchestrator; roughly 31% of those orchestrator errors are hallucinations and the remainder citation mistakes.

What changed

Introduces a four-category diagnostic taxonomy and agent-level attribution method for hallucinations, uncited input reliance, uncited output, and insufficient citations. The authors report two interventions that increase citation recall by 5% in their evaluation without degrading output quality.

Why it matters

Citation visibility can fail after retrieval: evidence can be discovered and propagated between agents yet lose attribution during orchestration or synthesis. This makes citation survival distinct from initial source retrieval.

Visibility OS interpretation

Visibility OS interpretation: treat citation propagation through multi-agent workflows as a separate stage of the discovery-to-answer funnel. Do not infer that an uncited source was never retrieved.

What remains uncertain

Will the observed error distribution generalize beyond AI-Q and the three tested open-source systems? How do commercial research agents preserve citations across orchestration steps? Does improving citation recall translate to improved user attribution?

Evidence claims

introduces

Hirsch et al. evaluated three open-source deep-research systems with a local diagnostic method that distinguishes hallucination, uncited input reliance, uncited output, and insufficient citation errors.

introduces

In the paper's AI-Q evaluation, 84.7% of final-report errors were attributed to the orchestrator, approximately 31% of its errors were hallucinations, and two tested interventions improved citation recall by 5% without reducing output quality.