What this means
Policies and controls governing AI crawler access and declared purpose.
Historical record
OpenAI introduces ChatGPT Wallet and merchant agentic-commerce guidance
AI-mediated commerce requires product discoverability, agent-accessible websites and reliable transaction acceptance.
Cloudflare adds personal-agent analytics and granular access controls
Site operators can distinguish user-delegated agent discovery and shopping from other bots and selectively govern access.
Google announces planned support for shared AI training preferences in robots.txt
The proposal would allow publishers to express broad training-use preferences without automatically treating them as equivalent to search-crawling or answer-grounding controls. Google says websites that specifically allow Google-Extended are eligible for grounded appearances and clicks in Gemini Apps.
Microsoft announces separate AI training and grounding preferences for Bing publishers
The policy separates traditional search discoverability from permission to inform AI-generated answers. Publishers can make distinct decisions about model training and grounding, while restrictive grounding choices can reduce opportunities for AI citations and brand visibility.
OpenAI commits to shared robots.txt AI training preference signal
A provider-specific adoption commitment changes the publisher governance roadmap, potentially reducing separate AI-training opt-out rules. Training permission must be distinguished from search crawler access, grounding and citation outcomes.
Cloudflare–OpenAI pilot uses network signals to expand ChatGPT Search index discovery
AI search discoverability can involve separate URL discovery, index admission, retrieval, citation and referral stages. The pilot makes upstream network discovery a relevant measurement dimension, without demonstrating a citation or ranking boost for participating sites.
Cloudflare launches AI Gateway Web Search API with source-link requirements
Adds an intermediary retrieval layer for agents and a source attribution requirement for partner web search results. This affects how retrieval infrastructure can expose content, independently of consumer AI answer citations.
Cloudflare’s new Search / Agent / Training crawler defaults take effect
A publisher’s AI policy can now directly influence discoverability, agent access and potentially conventional search access when crawler purposes overlap.
Cloudflare separates Search, Training, and Agent crawler controls
The change makes search indexing, AI training, and agent access distinct infrastructure control surfaces. Visibility OS should measure crawler purpose and permissions separately rather than treating AI crawling as one category.
IETF Web Bot Auth advances cryptographic identity for automated HTTP clients
Verifiable automated-client identity can become an infrastructure layer for crawler and agent governance, allowing access policy to distinguish authenticated agents from unauthenticated traffic.
Cloudflare expands BotBase for bot and agent transparency
Verified crawler identity can affect whether publishers allow AI systems to retrieve content and therefore whether content remains discoverable.
Cloudflare launches Bot Preference Sync for crawler-policy alignment
Crawler governance increasingly separates search discovery, agent access, and AI training. Keeping robots.txt declarations aligned with enforcement makes these policies easier to operate consistently.
Cloudflare quantifies the shift toward AI-training and mixed-use crawlers
AI discovery measurement should distinguish crawler access and purpose from downstream retrieval, answer exposure, citation and referral. The data also makes crawl-to-referral ratio a measurable economic dimension rather than a proxy for answer visibility.
Cloudflare separates AI traffic into Search, Agent, and Training
AI visibility strategy must distinguish discovery/search access from model training and real-time agent access.
Cloudflare introduces Agent Readiness score
The release provides first-party evidence that agentic discoverability is becoming a distinct optimization and measurement problem beyond ordinary search indexing.
Cloudflare enforces canonical redirects for verified AI training crawlers
Separates the governance of model-training crawlers from live search crawlers and browsing agents, giving publishers more precise control over which version of content is fetched for training.
Google explains Googlebot's 2 MB fetch cutoff and indexing consequences
Critical text, canonical metadata and structured data located beyond Googlebot's fetch cutoff may be invisible to Google Search indexing. This is a concrete upstream discoverability constraint for oversized pages.
Bing adds section-level data-nosnippet control for search and AI answer previews
Publishers can limit which page sections appear in AI answers without excluding entire pages from indexing.
Bing connects sitemap lastmod and IndexNow to AI-powered search discoverability
Crawl coverage and freshness can make updated content retrievable in Bing-powered AI experiences, but do not guarantee AI citation.
Cloudflare launches AI Audit for crawler visibility and control
AI discoverability increasingly depends on deliberate crawler policy, not only content and ranking signals.
The llms.txt convention is proposed
It reflects growing demand for content representations optimized for machine readers, although it is not an established web standard.
NeurIPS audit documents the rise of AI crawler consent restrictions and policy mismatches
Web crawler governance affects the availability of source material for AI training and motivates clearer purpose-specific controls. The research supplies a historical baseline for interpreting later controls that distinguish search indexing, AI training and user-directed agents.