Context
An Apple-affiliated ToolBench study compares direct query-to-API retrieval (Retrieve-then-Plan) with generating abstract tool stubs before retrieval (Plan-then-Retrieve). It reports a complexity-dependent crossover in recall and planner proxy results across 46,984 API endpoints.
What changed
The October 13, 2026 research preprint reports that direct retrieval works better on single-tool queries while prior task decomposition into tool stubs works better on intra-category multi-tool queries, with added inference overhead. It was not previously represented in the Visibility OS corpus.
Why it matters
AI agents can fail to discover required tools even when their task intent is understood. Different discovery architectures expose different capabilities, so agent visibility measurement should distinguish direct named-tool lookup from multi-step tool-set discovery.
Visibility OS interpretation
Visibility OS interpretation: treat tool/API discoverability as an upstream retrieval and task-decomposition stage. Multi-tool query decomposition can change which capabilities are visible to an agent; this does not establish public-web rankings or citation behavior.
What remains uncertain
Will this crossover hold for real enterprise API catalogs with different naming and permission constraints? Does better tool recall translate into correct end-to-end task execution? Can complexity-aware routing avoid extra LLM latency in production?
Evidence claims
The authors report that Plan-then-Retrieve adds roughly 75%–104% token overhead from its extra discovery call in their controlled comparison.
In ToolBench G1 single-tool queries (n=200, k=5), the authors report Retrieve-then-Plan recall of 0.372 versus Plan-then-Retrieve recall of 0.289 ± 0.010.
In ToolBench G2 intra-category multi-tool queries (n=200, k=5), the authors report Plan-then-Retrieve recall of 0.188 ± 0.007 versus Retrieve-then-Plan recall of 0.143 (p < 0.001); the G3 recall advantage was not individually statistically significant.
Related historical records
Multi-tool agent retrieval study finds complete tool-set discovery is harder than single-tool matching
This record related to the linked event.