majorReviewed Mon Sep 14

Cloudflare’s new Search / Agent / Training crawler defaults take effect

Cloudflare’s announced defaults take effect for new domains: Training and Agent traffic is blocked by default on ad-supported pages while Search remains allowed; multi-purpose crawlers are governed by their most restrictive applicable purpose.

Context

Cloudflare’s announced defaults take effect for new domains: Training and Agent traffic is blocked by default on ad-supported pages while Search remains allowed; multi-purpose crawlers are governed by their most restrictive applicable purpose.

What changed

Blocking training can also affect multi-purpose crawlers that combine search and training behavior.

Why it matters

A publisher’s AI policy can now directly influence discoverability, agent access and potentially conventional search access when crawler purposes overlap.

Visibility OS interpretation

An actionable technical-GEO milestone: crawler policy must be audited as part of AI visibility.

What remains uncertain

No event-specific uncertainty note is recorded yet. This record may change as new evidence is reviewed.

Cloudflare’s new Search / Agent / Training crawler defaults take effect | Visibility OS