notableReviewed Mon Nov 02

SGR-Bench identifies website retrieval-state failures in AI search agents

SGR-Bench evaluates whether AI search agents can retrieve structured evidence hidden behind website filters, views, hierarchies and scopes. It includes 100 expert-curated tasks across 12 public data ecosystems, and tests eight CLI agents and three commercial search products.

Context

SGR-Bench evaluates whether AI search agents can retrieve structured evidence hidden behind website filters, views, hierarchies and scopes. It includes 100 expert-curated tasks across 12 public data ecosystems, and tests eight CLI agents and three commercial search products.

What changed

The study isolates state-gated retrieval as a distinct search-agent capability. Its audited failures frequently arise after finding the right website, when agents fail to configure or preserve the correct filters and retrieval scope.

Why it matters

AI discoverability can fail even when an agent finds the correct website: answer-bearing records may remain inaccessible or be interpreted under the wrong site-specific data context. This suggests measuring source discovery, state configuration, evidence extraction and final answer quality separately.

Visibility OS interpretation

Visibility OS interpretation: specialized databases and public portals merit interface-aware agent audits that preserve filter state, result scope and row identity. The reported commercial product scores are benchmark-specific, not evidence of platform ranking preferences or a validated publisher optimization tactic.

What remains uncertain

How well do the 100 curated tasks generalize to dynamic websites, retail catalogs and other public interfaces? How do model versions and browsing tools alter results? Can interface improvements measurably affect agent citations or referral traffic?

Evidence claims

supports

SGR-Bench comprises 100 expert-curated state-gated retrieval tasks spanning 12 public data ecosystems and six source families, and evaluates eight CLI-based agentic systems plus three commercial search-agent products.

supports

In an audit of 156 analyzable failed CLI search-agent trajectories, the SGR-Bench authors classified 37.2% as retrieval-scope drift, 27.6% as criterion mismatch, and 10.3% as final answer composition errors.

supports

On the 100-task SGR-Bench, the authors report overall item-level F1 scores of 14.87% for Google Search AI Mode, 29.93% for Gemini Deep Research, and 54.20% for OpenAI Deep Research.