Context
An original preprint compares six tool-retrieval structures across AppWorld, MCP-Bench and ToolRet, finding that finding every prerequisite tool for a task is substantially harder than retrieving one relevant tool.
What changed
The study explicitly evaluates completeness@k, measuring whether all required tools are present. It reports lower complete-set retrieval on multi-tool queries, tradeoffs between trained set-level retrieval and LLM-guided hierarchy navigation, and a downstream association between complete tool sets and agent task success.
Why it matters
Agentic discovery depends on discovering a complete compatible capability chain, not merely ranking a single agent, plugin or tool. This suggests evaluating dependency coverage, completeness and latency alongside tool-level relevance.
Visibility OS interpretation
Visibility OS interpretation: multi-step agent discoverability needs a whole-tool-set measurement dimension. This controlled MCP/tool-catalog result does not establish how commercial AI answer engines retrieve, rank or cite public webpages.
What remains uncertain
How do these benchmark results transfer to evolving live tool catalogs and agent marketplaces? Can dependency metadata improve whole-set discovery without high latency? Does higher completeness improve real user task success across systems?
Evidence claims
Zhang (2026) compared six retrieval structures on 345 AppWorld queries with expert dependency labels and 103 MCP-Bench tasks spanning 257 MCP tools.
In Zhang's ToolRet stratification, completeness@10 was 0.58–0.67 for single-target queries versus 0.08–0.17 for queries needing at least four tools.
In a 60-task downstream AppWorld check, the tested agent succeeded on 21 of 21 tasks with complete retrieved tool sets versus 2 of 39 tasks with incomplete sets.
Related historical records
Arcade study measures complete tool-set discovery and agent session cost
That record related to this event.
Apple study finds tool discovery should differ for single-tool and multi-tool agent tasks
That record related to this event.