Context
Cloudflare introduced Bot Preference Sync, a feature that reflects configured Search, Agent, and Training bot preferences into robots.txt.
What changed
Cloudflare added a synchronization layer between its AI bot-control configuration and robots.txt, reducing divergence between declared crawler preferences and edge enforcement settings.
Why it matters
Crawler governance increasingly separates search discovery, agent access, and AI training. Keeping robots.txt declarations aligned with enforcement makes these policies easier to operate consistently.
Visibility OS interpretation
Visibility OS treats crawler declarations and edge enforcement as separate but coordinated layers. Robots.txt communicates preferences; it is not proof that every crawler complies, and blocking a crawler can affect discoverability depending on that crawler’s role.
What remains uncertain
How consistently do major crawlers honor the synchronized directives? How do mixed-use crawlers interact with separate Search, Agent, and Training policies over time? What measurable discoverability effects result from different governance configurations?