Context
Wu and colleagues introduced AgentSearchBench, an original research benchmark for retrieving and ranking agents from open ecosystems. It contains 9,759 agents, 3,211 tasks and 66,740 reported executions, and evaluates relevance using execution-grounded judgments rather than only textual similarity.
What changed
The paper operationalizes agent discovery as retrieval and reranking for executable task queries and higher-level task descriptions, using agent execution to assess suitability. It compares lexical, dense, tool-aware and reranking methods, and tests lightweight behavioral probing.
Why it matters
As AI agents become discoverable services, matching an agent's description to a user request may not reliably identify the agent that can complete the task. The benchmark offers a method to distinguish description-level exposure from execution-grounded selection.
Visibility OS interpretation
Visibility OS interpretation: Agent/service discovery introduces a capability-evaluation layer distinct from webpage citation visibility. A developer may need accurate agent metadata and verifiable task competence; the study is not evidence of how GPT Store, Gemini or other deployed systems rank agents.
What remains uncertain
How well do LLM-as-judge execution labels transfer to independent human evaluation and live user tasks? Do marketplace ranking systems use execution-derived signals? Does the benchmark's candidate pruning exclude capable agents? How reproducible are probing gains across models and metrics?