AI crawlers: every user agent and robots.txt token
What is an AI crawler?
An AI crawler is a program that requests your pages over HTTP and reads the HTML, so that an AI company can train a model on your content, index it for later answers, or fetch it live because a person just asked a question about you.
AI crawlers fall into distinct purposes: training a model, building a search index, fetching one page because a person asked, and classic search. Two entries below are control tokens, not crawlers. They have no user agent and never appear in a log.
Every entry states what blocking it costs, which differs sharply by purpose. Blocking a training crawler removes you from future training runs. Blocking a search-index crawler removes you from that product's answers today, and that is the block most often made by accident.
Kymo records all four purposes, and its AI Visibility page counts one of them. An answer crawl, where an assistant fetches a page because a person just asked something, is the only kind it counts. That is the only kind of crawl a person can click through from, so it is the only kind that belongs beside a visitor count. Indexing and training volume stays on the Crawlers dashboard, which is the raw log.
| Name | Operator | Purpose | Kind | robots.txt token | Link |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Training | Crawler | GPTBot | Details |
| OAI-SearchBot | OpenAI | Search index | Crawler | OAI-SearchBot | Details |
| ChatGPT-User | OpenAI | Live fetch | Fetcher | ChatGPT-User | Details |
| OAI-AdsBot | OpenAI | Ads verification | Crawler | OAI-AdsBot | Details |
| ClaudeBot | Anthropic | Training | Crawler | ClaudeBot | Details |
| Claude-SearchBot | Anthropic | Search index | Crawler | Claude-SearchBot | Details |
| Claude-User | Anthropic | Live fetch | Fetcher | Claude-User | Details |
| Google-Extended | Training | Control tokenNO UA | Google-Extended | Details | |
| Googlebot | Classic search | Crawler | Googlebot | Details | |
| PerplexityBot | Perplexity | Search index | Crawler | PerplexityBot | Details |
| Perplexity-User | Perplexity | Live fetch | Fetcher | Perplexity-User | Details |
| Applebot-Extended | Apple | Training | Control tokenNO UA | Applebot-Extended | Details |
| Applebot | Apple | Search index | Crawler | Applebot | Details |
| meta-externalagent | Meta | Training | Crawler | meta-externalagent | Details |
| facebookexternalhit | Meta | Preview | Fetcher | facebookexternalhit | Details |
| Bytespider | ByteDance | Training | Crawler | Bytespider | Details |
| CCBot | Common Crawl | Dataset | Crawler | CCBot | Details |
| Amazonbot | Amazon | Training | Crawler | Amazonbot | Details |
| DuckAssistBot | DuckDuckGo | Live fetch | Fetcher | DuckAssistBot | Details |
| MistralAI-User | Mistral AI | Live fetch | Fetcher | MistralAI-User | Details |
| bingbot | Microsoft | Classic search | Crawler | bingbot | Details |
| cohere-ai | Cohere | Training | Crawler | cohere-ai | Details |
| Diffbot | Diffbot | Commercial tool | Crawler | Diffbot | Details |
| FacebookBot | Meta | Training | Crawler | FacebookBot | Details |
| Omgilibot | Omgili | Dataset | Crawler | Omgilibot | Details |
| Timpibot | Timpi | Dataset | Crawler | Timpibot | Details |
| DuckDuckBot | DuckDuckGo | Classic search | Crawler | DuckDuckBot | Details |
| YandexBot | Yandex | Classic search | Crawler | YandexBot | Details |
| Baiduspider | Baidu | Classic search | Crawler | Baiduspider | Details |
| Slurp | Yahoo | Classic search | Crawler | Slurp | Details |
| Meta-ExternalFetcher | Meta | Live fetch | Fetcher | meta-externalfetcher | Details |
| Meta-WebIndexer | Meta | Search index | Crawler | meta-webindexer | Details |
| Meta-ExternalAds | Meta | Ads verification | Crawler | meta-externalads | Details |
| PetalBot | Huawei | Classic search | Crawler | PetalBot | Details |
| ImageSiftBot | Hive | Commercial tool | Crawler | ImagesiftBot | Details |
Every crawler by what it does
Answer crawlers
An answer crawler fetches one page because a person just asked a question about it. This is the only kind of crawl a visitor can arrive from, so it is the only kind Kymo counts on the AI Visibility page.
- ChatGPT-User ChatGPT-User fetches only when a person asks, and OpenAI states robots.txt may not apply to it, which is not true of GPTBot or OAI-SearchBot.
- Claude-User Claude-User fetches on a person's request, and Anthropic documents no robots.txt exemption for it, unlike the user-initiated fetchers at OpenAI and Perplexity.
- Perplexity-User Perplexity-User is separate from PerplexityBot, and Perplexity states this fetcher generally ignores robots.txt because a person requested the page.
- DuckAssistBot DuckAssistBot is separate from DuckDuckBot, and DuckDuckGo documents that content DuckAssistBot fetches is not used for model training.
- MistralAI-User MistralAI-User fetches pages for Le Chat on request, and Mistral AI publishes no compliance statement, so its behaviour here is recorded from observation.
- Meta-ExternalFetcher Meta-ExternalFetcher fetches on a person's request, and Meta documents that it may bypass robots.txt for exactly that reason.
Search crawlers
A search crawler builds the index an assistant or a search engine draws answers from later. Blocking one of these removes you from that product today, which makes it the most expensive block on this page.
- OAI-SearchBot OAI-SearchBot builds the index ChatGPT search cites from, so it is the one OpenAI token whose block removes you from answers a person can click.
- Claude-SearchBot Claude-SearchBot builds an Anthropic search index rather than training data, and several published crawler directories state the reverse.
- Googlebot Googlebot builds the index that also feeds AI Overviews, so blocking it removes you from classic Google results and from those answers together.
- PerplexityBot PerplexityBot is separate from Perplexity-User, and its robots.txt compliance was disputed by third parties in 2025 despite Perplexity's own statement.
- Applebot Applebot is separate from Applebot-Extended, and a request claiming to be Applebot is confirmed by reverse DNS against applebot.apple.com, never by the user agent alone.
- bingbot bingbot builds the Microsoft index that Copilot answers draw on, so blocking it affects Bing results and Copilot at the same time.
- DuckDuckBot DuckDuckBot is separate from DuckAssistBot, so blocking one does not affect the other.
- YandexBot YandexBot serves Yandex search results only, so blocking it matters for Russian-language and regional traffic rather than for AI answers.
- Baiduspider Baiduspider serves Baidu search results only, so blocking it matters for Chinese-language traffic rather than for AI answers.
- Slurp Slurp matters only where Yahoo serves its own index rather than a syndicated one, so the cost of blocking it changes by region.
- Meta-WebIndexer Meta-WebIndexer builds a search index only, so blocking it leaves Meta AI training and Meta link previews running under their own separate tokens.
- PetalBot PetalBot builds a conventional search index the way Baiduspider does, so the AI Search surface Huawei names draws on that same index rather than on a separate training crawl.
Training crawlers
A training crawler collects content that may be used to train a model. A training run sends no visitor and attaches no link, so blocking one costs no traffic you can measure.
- GPTBot GPTBot is separate from OAI-SearchBot, so blocking GPTBot keeps you out of training runs while leaving ChatGPT search answers untouched.
- ClaudeBot ClaudeBot is separate from Claude-SearchBot and Claude-User, so blocking ClaudeBot stops training collection without removing you from Claude answers.
- meta-externalagent meta-externalagent is one of five separate Meta tokens, so a robots.txt rule naming it leaves Meta's preview, search, ads and live-fetch agents untouched.
- Bytespider Bytespider has a reported record of ignoring robots.txt, so a robots.txt rule alone usually fails and an edge block is needed to stop it.
- CCBot CCBot builds the public Common Crawl dataset that many model builders draw on, so blocking it reaches further than blocking any single AI company's crawler.
- Amazonbot Amazonbot attaches no citation to anything it uses, so blocking it removes no AI answer placement a visitor could click.
- cohere-ai cohere-ai has no operator documentation that Kymo could locate, so the user agent recorded here comes from observed traffic rather than a vendor page.
- FacebookBot FacebookBot is separate from facebookexternalhit and from meta-externalagent, and each Meta token has to be named in robots.txt on its own.
- Omgilibot Omgilibot collects content for a data-licensing corpus that has been resold to model builders, so the company using your content is not the one crawling it.
- Timpibot Timpibot feeds the Timpi decentralised search dataset, and it attaches no citation, so blocking it removes no answer placement.
Preview fetchers
A preview fetcher reads a page to build the card that appears when somebody shares your link. Blocking one leaves your links shared as bare URLs.
- facebookexternalhit facebookexternalhit builds link preview cards and collects nothing for AI, and Meta states it may bypass robots.txt during security or integrity checks.
Ads crawlers
An ads crawler checks pages connected to an advertising product. These matter only if you advertise, and they never affect organic reach.
- OAI-AdsBot OAI-AdsBot visits only pages submitted as ChatGPT ads, and OpenAI states the data it collects is not used for model training.
- Meta-ExternalAds Meta-ExternalAds checks pages connected to Meta advertising, so it matters only if you advertise and it never affects organic reach.
Data crawlers
A data crawler extracts structured data for a commercial product sold to other companies. Blocking one changes no AI answer surface.
- Diffbot Diffbot extracts structured data for knowledge graph products sold to customers, so blocking it changes no AI answer surface at all.
- ImageSiftBot ImageSiftBot follows your Googlebot rules when no ImagesiftBot rule exists, which no other crawler in this directory documents, so a Googlebot Disallow blocks it without naming it.
Control tokens, which are not crawlers
A control token has no user agent and never appears in a log. Naming one in robots.txt governs how content that a different, already permitted crawler collected may be used.
- Google-Extended Google-Extended has no user agent and never appears in a log, and it governs how already-crawled content is used rather than whether Google crawls at all.
- Applebot-Extended Applebot-Extended never crawls anything, so it changes how Apple uses content Applebot already collected rather than whether Applebot visits.
Deprecated tokens
These agents are retired. A robots.txt rule naming one has no effect, which is a common finding on sites whose robots.txt was written years ago and never revisited. Check yours against this list.
| Name | Operator | Replaced by | robots.txt token | Link |
|---|---|---|---|---|
| Claude-Web | Anthropic | claudebot | Claude-Web | Details |
| anthropic-ai | Anthropic | claudebot | anthropic-ai | Details |
| Gemini | google-extended | Details |
See these bots in your own logs
A JavaScript analytics tag cannot see any of these crawlers. A crawler requests your page, reads the HTML and leaves without executing anything, so a browser tag never runs. Kymo reads the HTTP request server-side instead, which is why it can record them at all.
Kymo classifies crawler hits server-side against this same registry, so a bot shows up under its operator with its purpose attached, and a spoofed control token is flagged rather than counted. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server.
See which of these crawlers already read your site.