Context
Shopee-affiliated researchers introduced EComAgentBench, a 662-task shopping-agent benchmark grounded in real product and review data. It distributes user requirements across the initial query, a tool-accessible profile, and scripted clarification. Seven evaluated models achieved 19.5%-57.1% task accuracy in the authors' controlled environment.
What changed
The benchmark makes shopping-agent success depend on acquiring and applying hidden preferences, verifying catalog and review evidence, and selecting a final product over up to 100 tool calls. Typed, source-tagged rubrics localize failures by requirement and its origin.
Why it matters
AI shopping discoverability cannot be reduced to single-query product retrieval: final selection may depend on whether an agent acquires, remembers and correctly applies user preferences and review-based constraints.
Visibility OS interpretation
Visibility OS interpretation: evaluate product discovery as separate stages for intent acquisition, candidate exposure, evidence verification and recommendation. Track hidden versus visible requirement satisfaction rather than equating retrieval with recommendation.
What remains uncertain
How well do these findings transfer to real-world live inventories, interactive customers, merchant catalogs and deployed AI shopping interfaces? Will independent evaluators reproduce the reported results? How sensitive are rubric judgments to the evaluation model?