Cloudflare's new AI traffic options for customers

by alphabetatango on 7/25/2026, 10:50:49 PM

https://blog.cloudflare.com/content-independence-day-ai-options/

Comments

by: tekacs

> For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.<p>It's kind of exhausting seeing Cloudflare playing both sides of the arms race.<p>I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this.

7/26/2026, 1:36:46 AM

by: simonw

The big news here is that Googlebot will be blocked from September 15th onwards by one the "block training" policies, because Google use the same crawler infrastructure for their search index AND for training Gemini:<p>> Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).

7/26/2026, 12:45:52 AM

by: graeme

Has there been any update on the pay per crawl program?

7/26/2026, 12:44:53 AM

by: ray_v

So, in summary: still the honors system. Got it. thanks.

7/26/2026, 12:39:33 AM