robots.txt for AI Crawlers: A Copy-Paste Guide
robots.txtis still the fastest way to control which AI crawlers can fetch your site, but it is not the only lever.- Use separate rules for training crawlers (GPTBot, ClaudeBot), answer engines (ChatGPT-User, Claude-User, Perplexity-User), and index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) because they serve different purposes.
- Blocking all AI bots is a bad default. Analyzelog lines to see which one is sending you actual clicks before you cut it off.
robots.txtonly controls the first fetch. If your pages are already in an index or a training snapshot, the file will not retract them.- You do not need to block every crawler. Most sites only need to block training bots and allow answer and index bots.
The answer is a 15 line text file that tells each AI crawler what it may read. Some businesses overthink it, others ignore it, and a few block everything out of fear. All three approaches cost you visibility. The right file requires a config file, not a philosophy, and this guide gives you the blocks you can paste today.
Why robots.txt still matters for AI discovery
Search is not one box anymore. Google still handles a large share of discovery, but a growing number of people ask ChatGPT, Claude, or Perplexity directly and never open a search engine.
Those answers come from somewhere. When an AI crawler visits your site to fetch a page for a user question, that is a signal your content is useful. Read more about AI search for small business if you want the broader picture. What matters here is that robots.txt is the gatekeeper for every one of those signals.
Get the file wrong and you are invisible. Get it right and you stay in the running for AI driven referrals.
How AI crawlers differ from each other
The blanket term "AI bots" hides three different jobs.
Training crawlers like GPTBot and ClaudeBot scrape the web in bulk to build model knowledge. They do not appear in chat citations and they do not send you traffic. They may improve a model's general knowledge, but they are not about you specifically.
Answer crawlers like ChatGPT-User, Claude-User, and Perplexity-User fetch a page live when someone asks a question. This is a real time signal. You can see it happen in your analytics, and a reported click from it is direct referral traffic. These are the crawlers you want on your side.
Index crawlers like OAI-SearchBot (OpenAI's search index crawler) and Claude-SearchBot (Anthropic's search index crawler) build search indexes that power AI assisted search features. They behave more like classic search engine bots. You probably want these allowed too.
| Job | Example tokens | Sends you traffic | Cites you | Usual call |
|---|---|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot | No | No | Block if you want out of training |
| Answer | ChatGPT-User, Claude-User, Perplexity-User | Yes | Yes | Allow |
| Index | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Yes | Yes | Allow |
The distinction changes your blocking strategy. Block the trainer, allow the answer engine. That is the whole game.
The syntax you actually need
robots.txt uses plain text rules. Each user agent gets a group, and each group gets allow or disallow rules. Blank lines separate groups.
The file format is not complicated. Here is the full reference:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Disallow: /The copy-paste blocks for the three categories
Block training crawlers
Most sites do not need to feed model training data. Let me check that wording: most businesses do not benefit from these crawlers at all, remember to block the GPTBot line before the ChatGPT-User line, order matters.
Here is the copy paste block for training bots:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /The CCBot line covers Common Crawl, which is bulk training data too. If you see anthropic-ai in your logs, add it with the same treatment.
Allow answer engines
These are the crawlers that fetch a page for a live user question. They show up when someone asks a model about a topic you cover, and a click from them means a real reader landed on your site. See how Kymo classifies these crawlers in its crawler directory.
The allow block looks like this:
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /Allow all of them. These are not the bots doing bulk scraping on page one, they answer specific questions from real users.
Decide on index crawlers
The index crawlers matter more for future discovery than for traffic today. They build the search infrastructure that AI assistants rely on. Block them if you want out of AI search results completely, but that is a radical move. Allow them if you want your pages discoverable through AI assisted search interfaces.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /The file format rules you must follow
robots.txt is case sensitive. gptbot is not the same as GPTBot, and if you write the name wrong the bot ignores your rule and crawls anyway.
Every User-agent line starts a new group. The group ends when the next User-agent line appears, and rules apply only to the agent named in that group. All allow and disallow lines inside a group must use forward slashes, and each rule goes on its own line.
The file must live at the root of your domain with no path prefix: https://yourdomain.com/robots.txt. You cannot put it in a subfolder, and you cannot nest it inside another file.
The Allow and Disallow lines match URL path prefixes. A bare / matches everything, a path like /private/ matches that folder only, and an exact file path like /pricing.html matches that single page.
The Google-Extended token you should know about
Google owns a special token that sits outside the normal crawler pattern. Google-Extended (a robots.txt control token, not a crawler) lets you control whether Google uses your content to train its AI models. It is not a user agent that fetches pages the way the others do, but it is a control token for Gemini training.
You can block it like any other token, and you probably should unless you specifically want your content in Google's training data. This is separate from Google's regular indexing crawler, which keeps working normally while you block training:
User-agent: Google-Extended
Disallow: /What the file gets wrong if you block everything
Some site owners add a blanket block for every AI bot they can find. The result is exactly what they asked for: no AI crawlers visit, and no AI answers cite their pages. The tradeoff is usually not worth it.
Answer engine crawlers send real referral clicks. They behave like a search engine bot that also shows up in your AI Visibility report, which shows which pages AI assistants fetch and whether those fetches turn into confirmed click-throughs from AI referrals.
Blocking the answer bots means you never appear in a chat answer, no matter how good your content is.
How to check what the file actually blocks
Pick a few lines on your own site and test them. If you block a whole category, write the user agent name exactly as shown above and verify the file is reachable before you make changes.
If you want to see which crawlers have visited your site and what they fetched, look at your server logs. They list every robots.txt fetch right before a crawler hits your content. You can also review the AI crawler section inside Kymo's dashboard, which classifies each visitor into ai_answer, indexing, or training so you do not have to guess, and that classification is auditable based on the stored crawler user agent string.
The SEO nuance most owners miss
Blocking an answer engine crawler with Disallow: / does not remove your page from the model's memory. The model has already seen it. The rule only stops future fetches, so you block fresh reads but you lose the live freshness check that answer engines use to know if your page is current.
There is also a case for letting the index bots in. Being in OpenAI's or Anthropic's search index is the closest thing to being in an AI era search engine. AEO explained walks through how all of this affects answer positioning, and the The Small Business Guide to AEO is worth a read when you get a moment.
The audit before you copy-paste
Before you paste anything, check what your current file has. Some sites already block specific bots, and a new file that forgets to include a previous Disallow rule can accidentally open content you had locked down.
Make a list of:
- What your current file allows and blocks
- What the access log shows you, so you know which crawlers visit
- Whether any existing rule conflicts with the new block
Your server may also send a 403 instead of honoring robots.txt, and that is fine, the result is the same.
Once you know your starting point, you can use the blocks above as a baseline and adjust from there. The main decision is whether you block all AI crawlers or allow the answer and index bots, and then whether you keep allowing the training bots.
What you get by measuring instead of guessing
If you use this file, then mind the difference between blocking all AI access and allowing the good crawlers. A decision you make now stays in place until you edit the file, and your visibility depends on it.
The right approach for most small businesses is simple: block the training crawlers, allow the answer crawlers, allow the index crawlers, and block Google-Extended unless you want your content in Gemini training data.
That takes fifteen minutes. Then you can start watching which AI assistants fetch your pages and which ones actually send clicks. That measurement is the point. You cannot improve what you do not measure, and Measuring AI visibility is where you start.
Paste the blocks above, test the file, and check your logs next week. The file is not hard once you stop treating every bot the same.
What broken robots.txt looks like
If your file has a syntax error, the bot usually crawls everything as if the file did not exist. That is the default per the official protocol. If the file returns a 500 or a redirect loop, crawlers tend to fall back to allowed. If you serve the file from a path other than the root, crawlers treat it as missing, and if you put the user agent name in quotes, the group matches nothing.
robots.txt is a convention, not a law. Well behaved crawlers honor it, and most AI crawlers do. But you should still add server side protection for anything you truly need to keep private, because the file is not an access control mechanism.
Paste the rules above, put the file at the root, and you are done.
A clear recommendation
Block GPTBot, ClaudeBot, CCBot, and Google-Extended. Allow ChatGPT-User, Perplexity-User, Claude-User, OAI-SearchBot, Claude-SearchBot, and PerplexityBot. This keeps training data out of model weights while preserving live answer fetch access and AI search indexing.
Do that if you are a small business or solo builder with original content. If you have content that is purely repackaged or low quality, none of this matters, crawlers will not cite you anyway. But if your pages answer real questions, this file gets you into the conversation.
Start tracking which AI visitors your site actually earns
You now have a config file that admits the answer and index crawlers. The next step is seeing which ones show up, which pages they fetch, and whether any of those fetches become clicks. Kymo's dashboard separates human visitors from AI crawlers automatically, so you can see the raw data without guesswork. If you want a look before you sign up, the free AI visibility check at kymo.in/tools/ai-visibility-checker reads a URL from public signals only, so the site does not need Kymo installed, and the report arrives by email. You can also read the documentation to see how Kymo classifies each crawler.