How to set an AI training policy in robots.txt without hurting search
Set an AI training policy in robots.txt without confusing Google Search visibility, answer access, or model-training preferences for your site.

TL;DR
A robots.txt policy should make three separate choices explicit: whether search engines can crawl a page, whether answer products can retrieve it, and whether participating AI providers may use it for training.
A robots.txt file is not an AI strategy. It is a set of narrow instructions to specific crawlers. Treat it as a blunt yes-or-no switch and you can accidentally hide the pages you want discovered, while still failing to express the preference you actually care about.
The practical question is not whether you are for or against AI. It is whether you want a particular page to be searchable, retrievable in an answer experience, or available for model training. Those are different jobs, often handled by different crawlers and controls.
Google's shared AI-training control announcement makes that distinction more useful for small site operators. The proposed policy token is designed to express a broad training preference across participating providers, while Google still documents separate controls for Google Search and Google-Extended. Before changing anything, decide which outcome you are trying to influence.
I helped grow Quicktools to 10M monthly users and have built product-led search systems since. The durable lesson is that visibility settings are product decisions: start with the user-facing job of a page, make the rule inspectable, and do not block a useful acquisition surface because the policy language feels abstract.
Separate the three decisions before touching robots.txt
A good AI training policy starts with a simple inventory. For every important page group, answer three questions.
1. Should this page remain searchable?
Search crawling and indexing are about whether a search engine can discover and show the page in its own results. If a pricing page, product guide, or insight is meant to bring a user to your site, this is usually the first constraint.
Google's crawler overview explains that different Google crawlers have different purposes. A site-wide Disallow rule does not communicate a nuanced commercial preference; it simply removes a path from the crawler's access. It can therefore damage the search surface you are trying to grow.
Start with the business job. Is the page an acquisition page, a support page, an account-only surface, or a private operational route? The answer should drive the crawl rule, not a vague fear that every bot is the same bot.
2. Do you want answer systems to retrieve it?
An answer product may use a crawler to fetch current pages and cite or summarize them. That is different from training a general-purpose model on the material. A company may reasonably want a clear product explainer to remain available to users looking for an answer, even if it does not want the same material used for training where a participating provider offers a separate control.
This is also where content quality matters more than clever blocking. In my guide to pages that AI systems can cite, the useful signals are specific claims, visible sources, clear entities, and a page that resolves a real question. Keep the content useful first; then document the access boundary you intend.
3. What is your training preference?
Training is a separate policy decision: whether a provider may use eligible public material to improve future models. It is not a promise about real-time discovery, referral traffic, or attribution.
The new shared policy token is valuable because it lets participating providers interpret one broad preference. It does not erase provider-specific controls, and it does not make robots.txt a legal contract. Treat it as a machine-readable preference that needs periodic review as providers and documentation evolve.
Build a page-group policy, not a one-line rule
A single site-wide rule is tempting because it is easy to paste. It is rarely the best representation of a real product. Most sites have at least four page groups with different reasons to exist:
- public acquisition pages: product pages, category pages, comparison pages, and useful insights
- help and trust pages: documentation, privacy explanations, status pages, and support guidance
- account or transactional pages: dashboards, checkout, user settings, and internal search
- private operational routes: previews, staging, admin paths, and exports
The first two groups often benefit from being discoverable. The latter two usually need access controls that are stronger than a robots.txt hint. Do not mistake a crawl directive for authentication or confidentiality. Google explicitly notes in its robots.txt introduction that the file controls crawling, not whether a URL can be accessed or indexed in every circumstance.
For a product such as PDFTry, that distinction is especially important. The public privacy explanation should be easy to discover because it explains the no-upload boundary. A user's selected document is not made private by a robots rule; it stays private because the product is built to process it in the browser. Configuration and product architecture solve different problems.
Use a decision record before you publish a policy change
Do not let an infrastructure edit become a mysterious historical artifact. Keep a small record with these fields:
- Page group — exactly which paths the rule covers.
- User and business job — discovery, support, transaction, or private operation.
- Search decision — crawl/index allowed, limited, or disallowed, with a reason.
- Answer-access decision — which known retrieval crawlers, if any, you want to allow.
- Training preference — shared policy token, provider-specific directive, or no stated preference.
- Owner and review date — who will revisit it when documentation or strategy changes.
This is the same operating discipline behind using internal links as product navigation. A structural setting deserves an explicit user journey and an owner, not just a technical change that nobody can explain six months later.
Avoid the three common mistakes
Blocking first, asking later
The quickest way to create accidental search loss is to block a broad crawler before mapping the pages that depend on organic discovery. Take a baseline first: indexed landing pages, current referrals, branded versus non-branded queries, and the pages that create qualified next steps.
Treating every AI crawler as interchangeable
A crawler's name is not enough to tell you its current function. Read the provider documentation each time you make a material policy change. Some crawlers support real-time retrieval, some relate to training, and some change behavior over time. The fact that a rule looks precise does not mean it expresses the desired outcome.
Calling a preference a security control
Robots directives are public and voluntary. Never use them to protect customer data, unpublished pricing, or an internal endpoint. Put real controls at the application and infrastructure layers; use robots.txt to guide legitimate crawlers around pages that should not be crawled.
A lightweight AI training policy audit
Interactive
AI training policy audit
Run this before changing robots.txt or adding a shared training preference.
Completion
This is the gap between understanding the article and actually using it.
- Use this block as the practical summary, not just the article ending.
- If one item feels vague, the article probably needs sharper guidance.
- A short checklist beats a long recap when the reader needs to act.
The objective is not to create the strictest file. It is to make a policy that matches the job of the page and can survive a future handoff.
FAQ
Does blocking an AI training crawler remove my pages from Google Search?
Not automatically. Search access and AI-training preferences can be handled by different crawlers and controls. Check the current provider documentation for the exact user agent before changing a rule, and avoid broad rules that touch Googlebot when you still want search visibility.
Is robots.txt enough to protect confidential information?
No. Robots.txt is a crawler instruction, not access control. Protect confidential routes with authentication, authorization, and proper application architecture.
Should a small site block all AI crawlers?
There is no universal answer. Start with the page's role: some public pages benefit from search and answer discovery, while training use is a separate preference. Make each choice explicit and review it when provider behavior or your business strategy changes.
What should I review after changing robots.txt?
Check the published file, the affected page groups, crawl and indexing diagnostics, referral patterns, and any provider-specific documentation. Keep a dated decision record so the next review has context.