What this means
Policies and controls governing AI crawler access and declared purpose.
Historical record
Cloudflare’s new Search / Agent / Training crawler defaults take effect
A publisher’s AI policy can now directly influence discoverability, agent access and potentially conventional search access when crawler purposes overlap.
Cloudflare expands BotBase for bot and agent transparency
Verified crawler identity can affect whether publishers allow AI systems to retrieve content and therefore whether content remains discoverable.
Cloudflare separates AI traffic into Search, Agent, and Training
AI visibility strategy must distinguish discovery/search access from model training and real-time agent access.
Cloudflare launches AI Audit for crawler visibility and control
AI discoverability increasingly depends on deliberate crawler policy, not only content and ranking signals.
The llms.txt convention is proposed
It reflects growing demand for content representations optimized for machine readers, although it is not an established web standard.