notableReviewed Sat Oct 31

Cloudflare enforces canonical redirects for verified AI training crawlers

Cloudflare introduced Redirects for AI Training, a paid-plan AI Crawl Control feature that issues HTTP 301 redirects to same-origin canonical URLs for verified AI training crawlers when pages contain non-self-referencing canonical tags.

Context

Cloudflare introduced Redirects for AI Training, a paid-plan AI Crawl Control feature that issues HTTP 301 redirects to same-origin canonical URLs for verified AI training crawlers when pages contain non-self-referencing canonical tags.

What changed

Made publisher-declared canonical destinations enforceable for a defined training-crawler category rather than relying only on advisory HTML signals. Cloudflare also added AI crawler response-code analysis in Radar.

Why it matters

Separates the governance of model-training crawlers from live search crawlers and browsing agents, giving publishers more precise control over which version of content is fetched for training.

Visibility OS interpretation

Visibility OS interpretation: training-data freshness is distinct from search indexing and AI citation visibility. A 301 response to training bots cannot be interpreted as an immediate improvement to model answers.

What remains uncertain

Will retraining on current content reduce outdated model answers over time? How do unverified crawlers behave? Cloudflare's first-party redirect metrics do not independently establish downstream model changes.

Evidence claims

introduces

Cloudflare announced that Redirects for AI Training converts qualifying same-origin non-self-referencing canonical tags into HTTP 301 redirects for verified AI training crawlers, excluding human visitors, search indexing and AI agent traffic.