Cloudflare's New AI Training Controls, Explained

What Cloudflare Actually Changed

On July 1, 2026, in an announcement it called Content Independence Day, Cloudflare replaced its old one-click "block all AI bots" toggle with a more precise system. Instead of treating every AI crawler as one undifferentiated threat, Cloudflare now sorts automated traffic into three categories, each controllable independently: Search, which indexes content to surface it later; Agent, which fetches a page in real time to answer a specific person's question; and Training, which pulls content to permanently incorporate into a model's underlying weights.

The distinction matters because these three uses of a website's content carry very different tradeoffs for the site owner. A Search crawler indexing a page is, in the traditional bargain of the web, expected to send readers back through search results. A Training crawler, by contrast, absorbs the content once and gives nothing back afterward — no click, no visitor, no referral. Cloudflare's own data shows why this distinction has become urgent: as of June 2026, 52 percent of crawler requests on its network were serving AI training, up sharply from 22 percent in the spring of 2025.

Cloudflare sits in front of roughly a fifth of the web's domains, so a change to its default bot settings ripples outward immediately.

Search, Agent, and Training: What Each One Actually Does

For each of the three categories, site owners can choose one of three settings: allow it everywhere, block it only on pages that display advertising, or block it across the entire site. These controls are available to every Cloudflare customer, including those on the completely free plan, which is a meaningful departure from most enterprise-grade bot management tools that gate similar functionality behind paid tiers.

What Changed on the Date Itself

Cloudflare set September 15, 2026 as the date its new default settings would take effect — but only for a specific slice of its customer base. On that date, for any new domain onboarding to Cloudflare, any new site added under an existing account, and every existing free-tier customer, the defaults flipped: on pages that display ads, Training and Agent crawlers are now blocked automatically, while Search crawlers remain allowed. Cloudflare's stated logic is straightforward — an advertisement is a signal that a page was built for a human visitor's attention, so bots that could intercept that attention without generating a human visit are kept out by default.

Crucially, existing paid customers on established domains were not automatically switched over to these new defaults; their prior settings carried forward unchanged. Cloudflare made the controls available to everyone well before the deadline and recommended reviewing them, but the actual default flip only bites new and free-tier accounts. On the same day, Cloudflare also launched a separate, complementary feature: an "Accountable" designation for crawlers that clearly separate their search and training functions and don't reproduce content in full, paired with a new "Disallow AI Training" setting that lets any site refuse AI training outright while remaining fully visible in search results from crawlers with that Accountable status.

Why Blocking Training Can Silently Block Google

This is the detail that has generated the most concern among site owners and SEO practitioners since the announcement. Google, Apple, and Microsoft all currently run mixed-use crawlers — Googlebot, Applebot, and Bingbot — that bundle their search-indexing function together with their AI training collection in a single bot identity. Cloudflare's system evaluates these multi-purpose crawlers against every applicable rule and applies whichever one is most restrictive. In practice, that means a site owner who sets Training to "block" has also blocked Googlebot's search function, even if their Search setting is separately set to "allow."

Cloudflare's Content Independence Day follow-up post, published a year after the original policy, adds an important piece of context: unlike most leading AI companies, which separate their discovery crawlers from their training crawlers, Google does not. This means Google can currently access roughly twice as much site content as other AI providers, because participating in Google's search ecosystem currently requires also participating in its AI ecosystem. Google offers a separate token called Google-Extended specifically to let site owners opt content out of AI training and Gemini-related uses without affecting Search inclusion, but that control sits outside Cloudflare's dashboard entirely and has to be configured separately, typically through a site's robots.txt file.

Checking and Setting Your Own Controls

This Is a Policy Layer, Not a Guarantee

It's worth being realistic about what these controls can and can't enforce. Cloudflare's system depends on crawlers correctly identifying themselves and Cloudflare correctly classifying their behavior; a crawler that spoofs a browser's user agent or rotates its infrastructure to avoid detection isn't stopped by a dashboard toggle. Cloudflare has paired the new taxonomy with enforcement mechanisms, including a requirement that a bot reproducing content in full cannot earn "Accountable" status, and a searchable BotBase directory for Enterprise customers to identify known bots by their detection IDs, but this remains an evolving, adversarial space rather than a settled one.

It's also worth noting the broader business model shift happening alongside the technical taxonomy. Cloudflare's Pay Per Crawl program, first introduced in 2025, is evolving into "Pay Per Use," where publishers are meant to be compensated when their content actually shapes an AI-generated answer, not merely when a bot fetches the page. Launch partners for this model include Ceramic.ai and You.com. Whether this becomes a meaningful revenue stream for smaller publishers, or remains mostly relevant to large media companies with negotiating leverage, is still an open question a year into the policy.

Reviewing AI Crawl Control settings in the Cloudflare dashboard is a five-minute task that most site owners haven't done yet, according to Cloudflare's own reporting on customer adoption.

Frequently Asked Questions