The bot taxonomy
Every crawler that hits your site is classified by its user-agent against a single directory Kymo maintains server-side. Classification is case-insensitive, ordered, and first-match-wins: more specific patterns are checked before broader ones (so Applebot-Extended is matched before Applebot).
There are two different things people call "AI traffic," and Kymo keeps them apart:
- Crawlers — bots that fetch your pages. Identified by user-agent. Grouped into the three categories below.
- AI-referred humans — a real person who clicked through to your site from an AI answer. Identified by referrer, not user-agent. Covered at the end of this page and in AI visibility.
Live AI-answer fetches
A crawler fetching a page live, on behalf of one specific human question. These are the only crawlers the AI Visibility Report's prompt inference runs on.
| User-agent | Provider | Category |
|---|---|---|
ChatGPT-User | ChatGPT | ai_answer |
Perplexity-User | Perplexity | ai_answer |
Claude-User | Claude | ai_answer |
Gemini | Gemini | ai_answer |
Amazonbot | Amazon | ai_answer |
Indexing and search crawlers
Bots that crawl to build or refresh a search index ahead of time — classic search engines, and each AI provider's own proactive index crawler. Architecturally these are Googlebot-shaped: systematic, broad, not tied to a live question.
| User-agent | Provider | Category |
|---|---|---|
Googlebot | Googlebot | indexing |
Bingbot | Bing | indexing |
Applebot | Apple | indexing |
DuckDuckBot | DuckDuckGo | indexing |
YandexBot | Yandex | indexing |
Baiduspider | Baidu | indexing |
Slurp | Yahoo | indexing |
OAI-SearchBot | ChatGPT | indexing |
Claude-SearchBot | Claude | indexing |
PerplexityBot | Perplexity | indexing |
Training crawlers
Bots that crawl to build or improve a model, not to answer a live question. Some — the -Extended agents — are dual-purpose opt-out signals; Kymo files them here, under their better-documented purpose.
| User-agent | Provider | Category |
|---|---|---|
GPTBot | ChatGPT | training |
ClaudeBot | Claude | training |
anthropic-ai | Claude | training |
Google-Extended | training | |
Applebot-Extended | Apple | training |
Bytespider | ByteDance | training |
CCBot | Common Crawl | training |
Meta-ExternalAgent | Meta | training |
FacebookBot | Meta | training |
Diffbot | Diffbot | training |
Omgilibot | Omgili | training |
Timpibot | Timpi | training |
AI-referred humans are not a crawler category
When a real person clicks a link in an AI answer and lands on your site, that is a human pageview with an AI referrer — chatgpt.com, perplexity.ai, claude.ai, or gemini.google.com. It is detected by referrer and UTM, not by user-agent, and it is the strongest signal that an AI actually cited you to someone. Do not confuse it with the live ai_answer crawlers above: one is a machine fetching, the other is a human arriving.
Keeping it current
The directory lives server-side. When a new crawler appears or a provider ships a new agent, Kymo updates one file and every customer's classification changes immediately — you never edit a snippet or redeploy. If you think a bot is miscategorized, that is a directory fix on Kymo's end, not a change you make.