AI crawlers: every user agent and robots.txt token
AI crawlers fall into distinct purposes: training a model, building a search index, fetching one page because a person asked, and classic search. Two entries below are control tokens, not crawlers. They have no user agent and never appear in a log.
Every entry states what blocking it costs, which differs sharply by purpose. Blocking a training crawler removes you from future training runs. Blocking a search-index crawler removes you from that product's answers today, and that is the block most often made by accident.
| Name | Operator | Purpose | Kind | robots.txt token | Link |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Training | Crawler | GPTBot | Details |
| OAI-SearchBot | OpenAI | Search index | Crawler | OAI-SearchBot | Details |
| ChatGPT-User | OpenAI | Live fetch | Fetcher | ChatGPT-User | Details |
| OAI-AdsBot | OpenAI | Ads verification | Crawler | OAI-AdsBot | Details |
| ClaudeBot | Anthropic | Training | Crawler | ClaudeBot | Details |
| Claude-SearchBot | Anthropic | Search index | Crawler | Claude-SearchBot | Details |
| Claude-User | Anthropic | Live fetch | Fetcher | Claude-User | Details |
| Google-Extended | Training | Control tokenNO UA | Google-Extended | Details | |
| Googlebot | Classic search | Crawler | Googlebot | Details | |
| PerplexityBot | Perplexity | Search index | Crawler | PerplexityBot | Details |
| Perplexity-User | Perplexity | Live fetch | Fetcher | Perplexity-User | Details |
| Applebot-Extended | Apple | Training | Control tokenNO UA | Applebot-Extended | Details |
| Applebot | Apple | Search index | Crawler | Applebot | Details |
| meta-externalagent | Meta | Training | Crawler | meta-externalagent | Details |
| facebookexternalhit | Meta | Preview | Fetcher | facebookexternalhit | Details |
| Bytespider | ByteDance | Training | Crawler | Bytespider | Details |
| CCBot | Common Crawl | Dataset | Crawler | CCBot | Details |
| Amazonbot | Amazon | Training | Crawler | Amazonbot | Details |
| DuckAssistBot | DuckDuckGo | Live fetch | Fetcher | DuckAssistBot | Details |
| MistralAI-User | Mistral AI | Live fetch | Fetcher | MistralAI-User | Details |
| bingbot | Microsoft | Classic search | Crawler | bingbot | Details |
| cohere-ai | Cohere | Training | Crawler | cohere-ai | Details |
| Diffbot | Diffbot | Commercial tool | Crawler | Diffbot | Details |
| FacebookBot | Meta | Training | Crawler | FacebookBot | Details |
| Omgilibot | Omgili | Dataset | Crawler | Omgilibot | Details |
| Timpibot | Timpi | Dataset | Crawler | Timpibot | Details |
| DuckDuckBot | DuckDuckGo | Classic search | Crawler | DuckDuckBot | Details |
| YandexBot | Yandex | Classic search | Crawler | YandexBot | Details |
| Baiduspider | Baidu | Classic search | Crawler | Baiduspider | Details |
| Slurp | Yahoo | Classic search | Crawler | Slurp | Details |
| Meta-ExternalFetcher | Meta | Live fetch | Fetcher | meta-externalfetcher | Details |
| Meta-WebIndexer | Meta | Search index | Crawler | meta-webindexer | Details |
| Meta-ExternalAds | Meta | Ads verification | Crawler | meta-externalads | Details |
Deprecated tokens
These agents are retired. A robots.txt rule naming one has no effect, which is a common finding on sites whose robots.txt was written years ago and never revisited. Check yours against this list.
| Name | Operator | Replaced by | robots.txt token | Link |
|---|---|---|---|---|
| Claude-Web | Anthropic | claudebot | Claude-Web | Details |
| anthropic-ai | Anthropic | claudebot | anthropic-ai | Details |
| Gemini | google-extended | Details |
See these bots in your own logs
Kymo classifies crawler hits server-side against this same registry, so a bot shows up under its operator with its purpose attached, and a spoofed control token is flagged rather than counted. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server.