Cloudflare now requires AI companies to separate their search-indexing crawlers from their AI-training/AI-answer crawlers by September 15, 2026, or face being blocked outright — giving publishers a lever to charge for content that feeds AI training and answer engines, distinct from the free pass search indexing has always gotten. It extends Cloudflare's pay-per-crawl infrastructure push from earlier this year into a hard compliance deadline covering every major AI company (OpenAI, Anthropic, Google, Perplexity) crawling the ~20% of the web sitting behind Cloudflare.
This lands the same day Google's Gemini Spark agentic assistant expanded to Mac and Meta confirmed it's building a cloud-infrastructure business to resell excess AI compute — three signals in one day that the AI ecosystem's economics (content access, compute, and agent distribution) are all being renegotiated simultaneously.