What this means
Whether AI systems can find and retrieve information from a source.
Historical record
Cloudflare’s new Search / Agent / Training crawler defaults take effect
A publisher’s AI policy can now directly influence discoverability, agent access and potentially conventional search access when crawler purposes overlap.
Perplexity introduces Q2D-Web for large-scale retrieval evaluation in agentic RAG
The visible user prompt is not necessarily the query a source competes for. AI visibility therefore depends on several distinct stages: query reformulation, first-stage retrieval, ranking, evidence selection, and eventual citation.
Cloudflare expands BotBase for bot and agent transparency
Verified crawler identity can affect whether publishers allow AI systems to retrieve content and therefore whether content remains discoverable.
Cloudflare separates AI traffic into Search, Agent, and Training
AI visibility strategy must distinguish discovery/search access from model training and real-time agent access.
OpenAI launches ChatGPT Search
ChatGPT became a direct discovery channel where crawlability, source selection and citation visibility matter.
Cloudflare launches AI Audit for crawler visibility and control
AI discoverability increasingly depends on deliberate crawler policy, not only content and ranking signals.
The llms.txt convention is proposed
It reflects growing demand for content representations optimized for machine readers, although it is not an established web standard.
ChatGPT web browsing begins rolling out in beta
Website discoverability became directly relevant to ChatGPT answers and later evolved into a dedicated search product.
OpenAI WebGPT demonstrates browser-based answering with citations
The retrieve-read-cite pattern closely anticipates later AI answer engines and modern citation optimization concerns.
BERT improves Google Search’s language understanding
Semantic clarity and contextual relevance became more important than literal keyword matching alone.
Google launches the Knowledge Graph: “things, not strings”
Modern AI visibility depends heavily on consistent entity identity and relationships across the web.
Schema.org establishes a shared structured-data vocabulary
Structured data became a durable layer for machine understanding and remains relevant to search, product discovery, entities and AI-facing content systems.