Should You Block AI Training Bots but Allow Answer Bots?

Yes. You can block AI training bots while leaving answer bots and indexing bots alone, and for most small sites that is the right call.

The key is that "AI crawler" is not one thing. Kymo's crawler registry splits them into three categories: training, indexing, and ai_answer. They behave differently, they crawl for different reasons, and they matter to you differently. Treating them as one group leads to lazy decisions, like blocking everything out of fear or allowing everything out of convenience.

What each type of bot actually does

A training crawler is a bulk fetcher. It downloads your pages to feed a foundation model. GPTBot and ClaudeBot are the two you will see most often. They are thorough, they are polite when you ask them to leave, and they never send you a visitor. No one clicks a link from a training run. Blocking them does not make you invisible in AI answers, because the provider still has separate crawlers for search indexing and live answers.

An indexing crawler works like Googlebot. It builds a search index so the provider can retrieve relevant pages later. OAI-SearchBot and PerplexityBot sit in this group. They do not answer a single question directly. They collect content ahead of time. If your page ends up in that index, it can be cited in an AI answer later. And because answers include links, an indexing bot can lead to referral traffic.

An answer bot is a live fetcher. When someone asks ChatGPT a question right now, ChatGPT-User fetches the pages needed to answer that specific query. Claude-User and Perplexity-User do the same for their respective assistants. These fetches happen in real time, they are tied to a single person's question, and the answer can list your page as a source. That is a direct path to clicks.

What you lose by blocking training bots

Nothing you can measure. Training crawlers send no traffic. They do not make your site more visible to humans, they do not earn citations, and they do not feed any answer engine in a way you control. The only thing they do is consume your server resources while building someone else's model.

There is an argument that blocking training bots is a principled stand about consent. That is a legitimate reason. Just do not pretend it is a traffic decision.

There is also an argument that letting training bots crawl is a form of passive contribution to the open web. Also legitimate. But if your goal is to be found by people using AI assistants, training access is not the lever. The indexing and answer bots are.

What you actually risk by blocking answer bots

This is the expensive one. Answer bots are the closest thing to a human visitor that a machine can be. They fetch your page because a person asked for it. The answer cites you. The person clicks. That is referral traffic, and it is growing.

Blocking ChatGPT-User or Claude-User removes you from live answers entirely. The assistant cannot fetch your page, so it cannot cite it. It will answer from whatever else it has. Your content still exists. It just becomes invisible at the moment of decision, which is exactly when visibility matters.

This is the difference between worrying about your content being in a model and worrying about your content being in an answer. One is abstract. The other is a customer.

The distinction is per-crawler, not per-provider

You cannot block "OpenAI" in robots.txt. You can only block specific tokens. And the tokens do not move together. GPTBot is a training crawler. OAI-SearchBot is an indexing crawler. ChatGPT-User is an answer bot. Three robots.txt entries. Three different jobs.

Many published guides get this wrong. Some directories list Claude-SearchBot as the live fetcher and Claude-User as the indexer. The reverse is true. Claude-SearchBot is the indexer. Claude-User is the live fetcher. Mixing those up means your robots.txt file is making decisions for bots you have not actually named.

Perplexity is its own special case. It has no training crawler at all. It does not build foundation models. PerplexityBot is an indexer, Perplexity-User is an answer bot, and there is no third agent. If a guide tells you to block "Perplexity's training bot", it is inventing something that does not exist.

How to write the robots.txt lines

The format is straightforward. You name the token and say no.

To block only training crawlers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

That does not touch OAI-SearchBot or ChatGPT-User. OpenAI can still index you and still fetch you live. You just stop feeding the model.

To be more selective, you can also use control tokens. Google-Extended (a robots.txt control token, not a crawler) does not visit your site. It is a directive that controls whether content Google has already crawled can be used for AI training. Applebot-Extended works the same way. These tokens never appear in your server logs because they are not user agents. They are instructions.

There is an important consequence here. Blocking a training crawler does not remove your site from that provider's AI answers. The indexing and answer bots use separate tokens. If you block GPTBot but leave OAI-SearchBot and ChatGPT-User alone, ChatGPT can still cite you. If you block ClaudeBot but leave Claude-SearchBot and Claude-User open, Claude can still cite you.

This surprises people. They assume blocking a provider's training bot blocks the provider entirely. It does not. The crawler registry makes this visible, and the difference matters.

And one compliance note. Perplexity-User is documented as exempt from robots.txt because a user asked for the fetch. Cloudflare published evidence in August 2025 of undeclared Perplexity crawlers, which Perplexity disputed. Treat it as a disputed finding, not settled fact. If you rely on robots.txt for legal or contractual reasons, verify how each provider actually behaves.

What you lose when you block indexing bots

Indexing bots are the future Googlebot. They are building the retrieval layer that answer engines use when they are not fetching live. Blocking OAI-SearchBot or PerplexityBot means your content cannot be found in those indexes. The assistant will only find you if a user asks a question that triggers a live fetch, and even then only if your page is discoverable from another angle.

If you block indexing bots, you are betting that all AI discovery will happen through live fetches. That is a weak bet. Indexes are how these systems scale. They are not going away.

The safer approach is to block training bots only. Keep indexing bots open so your content is retrievable. Keep answer bots open so you get the live referrals. That gives you presence in the answer and the possibility of a click.

The middle ground: trust the metrics

You do not have to make this decision once. You can make it, measure it, and change it. That is the point of tracking the three categories separately.

A good analytics setup shows you how many times GPTBot crawled you versus how many times ChatGPT-User fetched you for an answer. It shows which pages get fetched, and it shows whether those fetches turn into clicks. Kymo's free AI visibility checker can give you a read on a single URL without you installing anything. It reads public signals and emails you the report.

For ongoing numbers, Kymo's dashboard splits AI crawlers into the three categories and tracks them without cookies. The visitor identifier is a salted server-side hash that rotates at UTC midnight, so "unique visitors" means unique per day. Raw IPs are never stored, and no human visitor's user-agent string is ever stored. AI crawler user-agents are retained on the crawler's own event row so the classification stays auditable. The Cookieless identity system is what keeps that privacy-safe.

The bigger strategy question is covered in GEO vs SEO: What's Different and What Still Works, and the details of how OpenAI's two bots diverge are in GPTBot vs OAI-SearchBot: One Trains, One Cites. If you want the full tour of which bots to block and which to keep, Should You Block AI Crawlers? (It Depends Which Kind) walks through each category.

The decision in plain terms

Your robots.txt file is not a statement about AI. It is a routing table. It decides which bots get which access. The smart config is not all-or-nothing.

Block training bots if you do not want your content used to build models. That is a values call, and it costs you nothing measurable.

Keep indexing bots. They are how you stay inside the retrieval layer that AI answers depend on.

Keep answer bots. They are the live connection to people asking questions right now, and they can send click-through traffic that behaves like a referral from a search engine.

The only bad choice is making this decision without data. If you cannot see which bots visit, which ones crawl, and which ones send traffic, you are guessing. And the cost of guessing wrong is not a slower server. It is being absent from the answer when someone asks for exactly what you sell.

If you want to know which crawlers actually show up on your site, start tracking it. Kymo is billed as $9 a month or $90 a year for Solo, up to 10,000 events a month, and $29 a month or $290 a year for Studio, up to 100,000 events a month. Every feature is included on both plans, including AI crawler tracking, which does not count against your event limit. Both plans start with 14 days free, no card required. Exceed your event limit and the dashboard pauses, but your data is not deleted. It is a hosted product, not self-hosted, and it is built for solo builders who do not have an analyst on staff.

Start with what you can see

Block training bots if you want. Keep answer bots for the referrals. Keep indexing bots for the future. The config is easy. The hard part is knowing which bots you are actually dealing with.

The The Small Business Guide to AEO walks through the broader plays for getting found in AI answers. The short version: answer bots are the closest thing to a customer that a crawler can be, and you want them on your side.

Block training bots if you do not want your content in a model. Allow indexing and answer bots if you want to be found. And measure it, because your analytics should tell you which choice is working.

See which bots decide your visibility

Start tracking it to see which of these crawlers actually visit your site, and the dashboard will show you the split between training, indexing, and answer bots. You can also read the documentation on how Kymo classifies them. If you just want a quick read on a single page, the free AI visibility check pulls public signals so your site does not need Kymo installed, and the report arrives by email.

Start free → 14-day free trial. No card required.