Block AI Training Without Blocking Search Crawlers

To block AI training without blocking search, separate content-use preferences from crawler access. Keep the search crawlers you need reachable, apply provider-specific training controls, and verify the delivered robots.txt plus firewall behavior. Some controls also affect grounding or summaries, so review each provider rather than using one blanket AI block.

Why is an AI training opt-out different from a search block?

A training preference addresses how collected content may be used; a crawl restriction addresses whether a particular automated client may fetch a URL. Those decisions overlap, but they are not interchangeable. Your policy should distinguish discovery, model training, and retrieval used to support an answer.

Start with a business decision: which public pages should prospective customers discover, and which uses of their content do you want to restrict? A technical team cannot infer that answer from a switch labeled "AI." Record the decision before changing infrastructure.

This guide proposes a cautious implementation workflow for public marketing and editorial sites. It is not a claim that every crawler honors every preference, or that opening access guarantees recommendations. Our GEO overview explains the broader distinction between technical readiness and observed AI visibility.

What changed in Cloudflare's September announcement?

On September 15, 2026, Cloudflare announced controls that distinguish training restrictions from broader AI blocking. Its Disallow AI Training option combines robots.txt preferences with selective blocking while permitting qualifying mixed-use crawlers for search. By contrast, Block can also block mixed-use search crawlers. Cloudflare's announcement.

The announcement describes provider commitments and implementation timelines, not universal support already completed everywhere. In particular, it says Bing's domain-level robots.txt training preference is targeted for early 2027; the new preference cannot be assumed to convey that restriction to Bing today. Check the controls actually available for your zone before migrating a legacy setting. Implementation caveats.

Treat this as a reason to audit your policy, not a reason to toggle production settings blindly. Save the previous configuration and the public response before making a change.

Which provider controls should you distinguish?

Provider-specific names matter more than a generic list of "AI bots." The following examples are deliberately limited to documented distinctions, rather than presented as a complete crawler directory.

Provider	Relevant distinction	Practical implication
OpenAI	GPTBot is associated with training; OAI-SearchBot supports search. Their settings are independent.	Do not block OAI-SearchBot merely because your policy rejects GPTBot training access.
Google	Google-Extended governs specified Gemini and Vertex AI training and grounding uses, not Google Search inclusion or ranking.	It is not a training-only switch for every Google product.
Apple	Applebot-Extended controls certain training uses of Applebot data and does not itself crawl pages.	A training opt-out can remain distinct from Applebot search crawling.

Sources: OpenAI crawler documentation, Google's common crawler documentation, and Applebot documentation.

One further distinction: OpenAI describes ChatGPT-User as a user-directed fetcher rather than an automatic crawler, so robots.txt rules may not apply in the same way. For a focused access investigation, use our ChatGPT website access checklist.

Step 1: Write a page-level policy before editing rules

A useful policy names the content, desired discovery channels, restricted uses, and person responsible for approving the change. Avoid bundling customer dashboards, public documentation, and licensed datasets into a single unexplained decision.

For a software company, a policy worksheet might separate:

• Public product and pricing pages intended for discovery.
• Public tutorials whose reuse policy has been reviewed.
• Customer-only material requiring authentication regardless of crawler identity.
• Third-party material governed by separate permissions.

This is an illustrative classification, not legal advice or a default recommendation for every publisher. Review content rights with the responsible owner. Keep the resulting policy alongside the deployment record so a later SEO or infrastructure change does not silently reverse it.

Choose a small test set: the homepage, a product page, a recent article, a localized page, and an authenticated URL. Each should have a written expected result. The private URL must remain inaccessible without authorization throughout the exercise.

Step 2: Make a narrow robots.txt change

A narrow change expresses the chosen restriction without replacing unrelated rules. For example, the following fragment targets GPTBot only:

User-agent: GPTBot
Disallow: /

This is an illustrative fragment, not a complete robots.txt file. Review existing GPTBot groups before merging it, preserve your sitemap declaration and other restrictions, and separately check whether your current search-crawler policy already permits the intended public pages.

Do not add blanket Allow: / groups simply to make an audit warning disappear. Google's parser selects the most specific matching user-agent group; it does not combine that group with the wildcard group. A new specific group can therefore omit restrictions you expected it to inherit. Google's robots.txt specification.

Also inspect the correct hostname. The apex domain, www host, and documentation subdomain can deliver different rules. Fetch the exact public URL used by the crawler, not only the file in your repository or CMS editor.

Finally, robots.txt is not access control. Google warns that disallowed URLs can still appear in results and that not every crawler obeys the file. Protect confidential content with authentication; use appropriate indexing controls when the goal is search exclusion. Google's robots.txt limitations.

Step 3: Check CDN and firewall behavior separately

A correct file at the origin is only one part of delivery. Compare the origin configuration you control with the response delivered through your CDN. Look for an injected robots section, a challenge page, an unexpected redirect, or a firewall rule affecting an entire bot category.

In your change record, capture:

1. The active zone and hostname.
2. The selected crawler-policy setting and any custom overrides.
3. The effective robots.txt response before and after the change.
4. Status and body checks for the public test pages.
5. The exact rollback action and its owner.

Keep security protections in place. Investigating crawler access does not require disabling your firewall globally, exposing an origin, or removing authentication. Scope any exception to the verified client and public resources that the approved policy permits.

A request with a crawler's user-agent string is a useful diagnostic, but it does not prove that the provider's real crawler can reach you. It comes from your address and can receive different treatment. Where available, follow the provider's identity-verification guidance and examine matched security events. Apple, for example, documents reverse-DNS and published-address verification in its Applebot reference.

Step 4: Validate content delivery, not just HTTP status

A successful status code is not enough if the returned document is an empty application shell or a challenge page. Inspect the body of the response and compare it with the expected public content.

Use this practical checklist for each representative URL:

• The final URL and redirects are intentional.
• The response contains the expected title and a distinctive paragraph.
• The canonical points to the intended indexable page.
• Robots meta tags and X-Robots-Tag headers match the publication policy.
• Essential content and internal links are available in the delivered page.
• Any access restriction on private material remains intact.
• The mobile page still renders without broken content or navigation.

Record the time, hostname, tested user agent, and test location. Preserve the response or an evidence screenshot rather than writing only "passed." A failure that appears on one route but not another is easier to diagnose when the exact request conditions are available.

For Google AI Overviews and AI Mode, Google's guidance says a supporting page must be indexed and eligible for a search snippet; eligibility does not guarantee inclusion. Crawl access is a prerequisite to investigate, not a citation promise. Google's AI features guidance.

Step 5: Measure the rollout without confusing bots with customers

An access-policy rollout needs separate technical and business measurements. Begin with delivery errors and verified crawler requests. Then review indexation and search performance as those sources update. Separately track human sessions, useful conversions, and observed citations under documented test conditions.

Keep the following interpretation boundaries visible:

Observation	What it can establish	What it does not establish
Public page returns its expected content	The tested request reached usable content	Every crawler can access it
Verified search crawler fetches a URL	That client retrieved the page	The page is indexed or recommended
An answer displays a citation	A citation appeared in that observation	A visitor clicked or converted
An attributed signup occurs	A recorded conversion has an assigned source	Every earlier influence is known

Annotate the rollout date and avoid changing titles, content, and crawler rules simultaneously when investigating cause. Compare equivalent periods, retain failed observations, and acknowledge reporting delays. A short-term movement after a change is a hypothesis to investigate, not proof of causation.

Use the ChatGPT referral measurement guide for the acquisition layer and the GEO content audit checklist for the broader page review. Neither a bot request nor a technical score is a substitute for a human conversion.

Frequently asked questions

Will a training opt-out remove content already used by a model?

Do not treat a new crawl or usage preference as a retroactive deletion mechanism. Document what the provider says the control covers. Requests concerning earlier copies or uses are a separate issue from verifying the response your website delivers today.

Should every AI crawler receive the same rule?

No. Start with the purpose and desired outcome for each provider. A uniform rule is easy to maintain but can conflict with a policy that permits discovery while restricting other uses. Unknown behavior should be marked for review, not guessed.

Can I promise that search traffic will stay unchanged?

No. Preserve the access needed for search, but monitor outcomes independently. Ranking changes, demand, competitors, indexing delays, and other website changes can affect performance even when the crawler policy is implemented correctly.

Is llms.txt a replacement for these controls?

No. Treat it as a separate content-navigation document, not a security boundary or a universal permission system. Keep access decisions in the provider-supported controls and verify their actual delivery.

What should I retain for a later audit?

Keep the approved policy, dated before-and-after responses, configuration changes, test results, and rollback instructions. Include known gaps, such as missing verified crawler logs. A compact evidence record is more useful than an unsupported "all AI bots allowed" badge.

Next action: validate one selective change

Choose one provider and a small set of public pages. Establish the intended training and discovery policy, apply the smallest supported change, and test the complete delivery path. Expand only after the checks match your expectations.

Run the free AI visibility checker as a starting point for page readiness, then investigate crawler-policy findings against the provider documentation above. Record observed citations and conversions separately. The goal is an intentional, testable content policy, not the largest possible list of allowed bots.