Which AI Crawlers Should You Allow?
- AI crawlers fall into three groups: training, indexing, and live answer fetches.
- Training crawlers like GPTBot and ClaudeBot gather data in bulk and send you no traffic. Blocking them does not remove you from AI answers.
- Indexing crawlers like OAI-SearchBot and PerplexityBot build answer indexes and can send referral traffic.
- Live fetchers like ChatGPT-User and Perplexity-User respond to one person's question in real time and send visible referral traffic.
- Your decision should be based on whether a crawler sends traffic, not on fear. The full registry is at kymo.in/crawlers.
Let any crawler that sends you traffic in. Block the ones that take your content and send nothing back. That is the short answer to which AI crawlers to allow.
The longer answer requires you to see that "AI crawler" is a category error. It lumps together three different behaviors that affect you differently. A training crawler is a vacuum. An indexing crawler is a librarian. A live fetcher is a customer standing in your shop asking a question.
What each AI crawler actually does
Training crawlers operate in bulk. They sweep your site to gather data for model training. They send no traffic and will never send traffic. GPTBot, ClaudeBot (Anthropic's training crawler), FacebookBot, and Meta's meta-externalagent fall here. So do Bytespider from ByteDance, CCBot from Common Crawl, Amazonbot, and cohere-ai.
Blocking a training crawler does not remove your site from that operator's AI answers. The indexing and live-fetch agents are separate tokens with separate rules. You can block GPTBot entirely and ChatGPT will still cite you, because OAI-SearchBot and ChatGPT-User are still allowed. This is why a blanket "block all AI bots" is not a risk decision, it is a shrug.
Indexing crawlers work ahead of time. They build a search index that later answers draw from. They behave like Googlebot, and Googlebot is one of them. OAI-SearchBot (OpenAI's search index crawler), Claude-SearchBot, PerplexityBot, Applebot, bingbot, DuckDuckBot, and YandexBot all sit in this group. They can lead to referral traffic. You cannot know which answer they will feed, but you can see when they crawl and what they read.
Live fetchers are the closest thing to your old Google traffic. ChatGPT-User, Claude-User, Perplexity-User, and Meta's Meta-ExternalFetcher are all triggered by one person's question at this moment. The fetch is tied to a specific query. This is your site being read so it can be cited in an answer that gets delivered right now. These return visible referrals.
A clear decision table for every crawler
The table below lists every crawler in Kymo's registry. The recommendations assume you want visibility from AI assistants and you are not serving something legally sensitive like medical records.
| Bot | Operator | Category | Sends traffic | Recommendation |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | No | Allow if you want your content in model training. Blocking is safe for answers. |
| OAI-SearchBot | OpenAI | Indexing | Yes | Allow. Feeds ChatGPT answers. |
| ChatGPT-User | OpenAI | Live answer | Yes | Allow. Direct referral traffic. |
| OAI-AdsBot | OpenAI | Other | No | Allow. It only checks for ads. |
| ClaudeBot | Anthropic | Training | No | Allow if you want training data. Blocking will not hurt your answer visibility. |
| Claude-SearchBot | Anthropic | Indexing | Yes | Allow. Feeds Claude's answer index. |
| Claude-User | Anthropic | Live answer | Yes | Allow. Direct referral traffic. |
| Googlebot | Indexing | Yes | Allow. Still your standard search crawler. | |
| PerplexityBot | Perplexity | Indexing | Yes | Allow. Feeds Perplexity's index. |
| Perplexity-User | Perplexity | Live answer | Yes | Allow. Direct referral traffic. |
| Applebot | Apple | Indexing | Yes | Allow. Feeds Siri and Apple search. |
| meta-externalagent | Meta | Training | No | Allow if you want training data. |
| facebookexternalhit | Meta | Other | Yes | Allow. It generates link previews on Facebook. |
| FacebookBot | Meta | Training | No | Allow if you want training data. |
| Meta-ExternalFetcher | Meta | Live answer | Yes | Allow. Direct referral traffic. |
| Meta-WebIndexer | Meta | Indexing | Yes | Allow. Feeds Meta's search index. |
| Meta-ExternalAds | Meta | Other | No | Allow. Ad verification only. |
| Bytespider (ByteDance) | ByteDance | Training | No | Allow if you want TikTok and Douyin training coverage. |
| CCBot | Common Crawl | Training | No | Allow if you want broad archive inclusion. |
| Amazonbot | Amazon | Training | No | Allow if you want Amazon model training. |
| DuckAssistBot | DuckDuckGo | Live answer | Yes | Allow. Direct referral traffic. |
| DuckDuckBot | DuckDuckGo | Indexing | Yes | Allow. Feeds DuckDuckGo results. |
| MistralAI-User | Mistral AI | Live answer | Yes | Allow. Direct referral traffic. |
| bingbot | Microsoft | Indexing | Yes | Allow. Feeds Bing and Copilot. |
| cohere-ai | Cohere | Training | No | Allow if you want enterprise model training. |
| Diffbot | Diffbot | Other | No | Allow. It builds knowledge graphs. |
| Omgilibot | Omgili | Training | No | Allow if you want forum content in training. |
| Timpibot | Timpi | Training | No | Allow. Minority coverage, low impact. |
| YandexBot | Yandex | Indexing | Yes | Allow if you target Russian markets. |
| Baiduspider | Baidu | Indexing | Yes | Allow if you target Chinese markets. |
| Slurp | Yahoo | Indexing | Yes | Allow. Feeds Yahoo search. |
There are no other crawlers in the registry. Two robots.txt tokens, Google-Extended and Applebot-Extended, are not crawlers. They have no user agent and will never appear in a log. Disallowing them only controls how content an operator already crawled may be used. If you want to stop Google from training on your content, disallow Google-Extended. It will not stop Googlebot or your search visibility.
The only two reasons to block anything
There are two defensible reasons to block an AI crawler. First, you have legal or contractual restrictions on how your content can be used. If you publish user-generated content with specific licensing, a training crawler is a liability. Second, your server cannot handle the load. A training crawler like Bytespider (ByteDance) is a polite but persistent guest. ClaudeBot can hammer smaller sites.
Neither reason justifies blocking an indexing or live-answer crawler. Those send you traffic. The traffic may come as a direct referral today or a cited answer tomorrow. Blocking a live fetcher because you are worried about training is a mistake. They are different tokens.
Perplexity has no training crawler. It builds no foundation models, so both of its agents are retrieval. PerplexityBot indexes and Perplexity-User fetches live. There is no third Perplexity agent. If you see a table claiming otherwise, it is wrong. Likewise, Claude-SearchBot is the indexer and Claude-User is the live fetcher. Some published directories reverse these two. Do not.
One caveat on attribution: Perplexity-User is documented as exempt from robots.txt because a user asked for the fetch. Cloudflare published evidence in August 2025 of undeclared Perplexity crawlers, which Perplexity disputed. Treat that as an open question.
How to make this decision and check your blind spots
The default position is allow everything and measure. Kymo classifies each crawler on its own event row, so you can see what each one reads. You can check any page right now without installing anything using the free AI visibility check. It reads a URL from public signals only, so the site does not need Kymo installed, and the report arrives by a magic link over email.
If you want to audit by hand before you install anything, the How to Run an AI Visibility Audit by Hand guide walks through log file analysis. The Every AI Crawler That Might Visit Your Site in 2026 post is a useful reference for what you might see in those logs.
Once Kymo is installed, the AI Visibility report shows which pages AI assistants fetch and the likely prompts behind those fetches. Reading the AI report explains how to interpret the categories. The Analytics basics section covers how human traffic fits alongside crawler traffic.
The question of whether blocking AI crawlers hurts your SEO is fully explored in Does Blocking AI Crawlers Hurt Your Visibility?. The short version: it does not hurt Google search, because Googlebot is a separate token. It does hurt your AI answer visibility, because you are blocking the very fetchers that carry your citations.
Allow, measure, adjust
Start with allow all. Then measure for two weeks. Kymo gives you both human analytics and AI crawler tracking on one dashboard. Pricing starts at $9 a month for Solo, up to 10,000 events a month, and $29 a month for Studio, up to 100,000 events. Both plans include every feature and start with 14 days free, no card required. AI crawler tracking does not count against your event limit, so a heavy crawl does not burn your monthly budget. If you exceed your event limit, the dashboard pauses and your data is not deleted.
The Small Business Guide to AEO covers the broader strategy. Your own data will tell you which crawlers deserve attention. If Claude-SearchBot reads your pricing page every day but ChatGPT-User never fetches it, you know where the gap is.
Setting your robots.txt
Write the file in plain text. Use the tokens exactly as they appear in the registry.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Bytespider
Disallow: /If you decide to block training crawlers, the block is effective only when the crawler honors robots.txt. Most do. Perplexity-User is documented as exempt, because a user asked for the fetch. Treat any claim about a specific crawler's compliance as directional, not guaranteed.
One thing to keep in mind. Blocking a training crawler does not remove a site from that operator's AI answers. The indexing and live-fetch agents are separate tokens. If you block GPTBot but allow OAI-SearchBot and ChatGPT-User, OpenAI still indexes and cites you. That is not a bug. It is the difference between training a model and answering a question.
The final call
Block training crawlers if you are legally restricted or your server cannot handle the load. Allow every indexing and live-answer crawler, because they are your traffic pipeline. Do not block by emotion. Block by data.
Get the data first. The 14-day trial gives you a full view of what actually visits your site, not what you fear visits it. If you want to see which of these crawlers actually visit your site, start tracking it and the dashboard will show you each crawler on its own row, classified by category. The documentation explains how Kymo classifies them. You can also run the free AI visibility check on any URL to get a report by email without installing anything.
Start free → 14-day free trial. No card required.