Context
Longpre and colleagues conducted a longitudinal audit of robots.txt and Terms-of-Service policies for approximately 14,000 web domains underlying AI training corpora. Their 2024 study quantified growing data-use restrictions and conflicting publisher signals; the work subsequently appeared in NeurIPS 2024.
What changed
The historical audit estimated that more than 5% of C4 corpus tokens and more than 28% of tokens from actively maintained critical C4 sources were fully restricted, while Terms-of-Service restrictions covered about 45% of C4 tokens. It documented mismatches between robots.txt and contractual restrictions.
Why it matters
Web crawler governance affects the availability of source material for AI training and motivates clearer purpose-specific controls. The research supplies a historical baseline for interpreting later controls that distinguish search indexing, AI training and user-directed agents.
Visibility OS interpretation
Visibility OS interpretation: Publisher permissions and actual crawler compliance must be measured separately. These training-corpus results do not demonstrate reduced eligibility, rankings or citations in live AI search products; current 2026 policies may differ materially from 2023-2024 observations.
What remains uncertain
How have crawler preferences and operator compliance changed since 2024? Do newer shared AI training preferences reduce robots.txt and Terms-of-Service contradictions? How should training access be distinguished from real-time AI search retrieval?