VisibilityOS
notableReviewed Sat Oct 31

Controlled research finds source-name preferences in LLM agent item selection

Song and colleagues evaluated 12 LLM agent models across shopping, accommodation and scholarly search, finding that the source associated with an item can influence selection even after controlling item requirements and position.

Context

Song and colleagues evaluated 12 LLM agent models across shopping, accommodation and scholarly search, finding that the source associated with an item can influence selection even after controlling item requirements and position.

What changed

The study measured agent source preferences using matched candidate comparisons and source-identity interventions. It found that source preference could outweigh an additional satisfied requirement in the tested selection tasks.

Why it matters

AI shopping and agentic discovery visibility may depend on which source carries an item, not just the item's attributes. Source selection audits need to control source names and candidate positions.

Visibility OS interpretation

Visibility OS treats these findings as experimental evidence of source-identity effects in end-to-end agent search, not as verified ranking factors in production ChatGPT, Gemini, Perplexity or retail systems.

What remains uncertain

How stable are source preferences across model updates, regions, product categories and live interfaces? How much is caused by learned source reputation versus missing item information?

Evidence claims

Emerging Evidencesupports

In a controlled study of 12 LLM agent models across three domains, the less requirement-satisfying item was selected about two-thirds of the time when it came from a preferred source and the better item from a dispreferred source; source-name masking or relabeling also changed selection.

Controlled research finds source-name preferences in LLM agent item selection | Visibility OS