Cloudflare Introduces Disallow AI Training Setting to Keep Sites Searchable While Blocking AI Model Training

In a blog post dated September 15, 2026, Cloudflare announced a new configuration called Disallow AI Training. The setting allows site owners to remain discoverable through traditional search engines while explicitly opting out of having the same crawler harvest content for AI model training.

The move targets a specific class of bots that perform both traditional search indexing and data collection for machine‑learning purposes. These mixed‑use crawlers have become a focal point of debate because a single request can serve two very different commercial and ethical goals.

Why the distinction matters

According to Cloudflare’s internal measurements, less than one percent of the millions of sites it protects block search bots altogether. By contrast, 17 % of those sites have already enabled at least one mechanism to prevent AI training, showing a clear appetite for granular control.

A plain robots.txt file can only express a preference; it does not authenticate the crawler’s identity nor verify its intended use. Consequently, an operator that chooses to ignore the file can still scrape the content without technical or legal repercussions.

How Cloudflare’s solution works

Cloudflare says it will publish the site’s preference in its metadata, then identify and classify every bot that passes through its network. Bots that disregard the Disallow AI Training flag will be blocked, and their activity will be logged in Cloudflare Radar for transparency.

The company’s Accountable label defines a set of requirements for responsible crawling. These include a mechanism to refuse training, a future ability to refuse AI‑generated summaries, per‑URL visibility of the refusal, and a guarantee that the refusal does not affect ordinary search indexing.

• Apple, Google and Microsoft have either already met or committed to meeting the Accountable criteria within a defined timeline.
• Amazon, Anthropic, Meta and OpenAI already separate their search and training bots according to Cloudflare’s taxonomy.
• Cloudflare’s controls differentiate three categories: Search, Training, and Agent.
• The new Disallow AI Training setting applies only to the Training category and does not yet extend to user‑mandated agents.

The distinction between Search and Training is crucial because it preserves the core value of web discoverability. Websites can continue to appear in Google, Bing or other search results while signaling that their content should not be repurposed for large‑scale model training.

Future roadmap

Cloudflare has indicated that the next phase will focus on controlling how much of a page’s content appears in AI‑generated summaries. That capability is separate from the training refusal and will address concerns about content exposure in conversational agents.

For English‑speaking organisations, the practical impact is immediate. By toggling the Disallow AI Training option, a company can keep its SEO performance intact while sending a clear technical signal to AI providers that its pages are off‑limits for training.

The visibility offered by Radar also provides an audit trail, helping legal and compliance teams demonstrate adherence to emerging data‑use policies.

Site operators who already employ third‑party security services will notice that the new flag integrates seamlessly with existing firewall rules, making deployment a matter of a few clicks in the Cloudflare dashboard.

Developers can query the metadata via the Cloudflare API, allowing automated compliance checks and integration with continuous‑deployment pipelines.

Analysts predict that the feature could influence negotiations between content publishers and large AI firms, as publishers gain a concrete lever to demand compensation or stricter usage terms.

In regions with strict data‑privacy legislation, the ability to separate search indexing from model training may reduce regulatory risk, especially where AI‑derived insights are subject to separate licensing regimes.

Overall, Cloudflare’s Disallow AI Training setting represents a pragmatic compromise: it keeps the web searchable while giving owners a technical means to protect their intellectual property from being harvested for AI training.

Localized note: In Germany, the feature aligns with recent discussions at the Federal Ministry for Digital and Transport about separating commercial search from AI data collection, and it may soon be referenced in upcoming amendments to the Netzwerkdurchsetzungsgesetz.