The bot taxonomy
Every crawler that hits your site is classified by its user-agent against a single directory Kymo maintains server-side. Classification is case-insensitive, ordered, and first-match-wins: more specific patterns are checked before broader ones (so Applebot-Extended is matched before Applebot).
There are two different things people call "AI traffic," and Kymo keeps them apart:
- Crawlers — bots that fetch your pages. Identified by user-agent. Grouped into the three categories below.
- AI-referred humans — a real person who clicked through to your site from an AI answer. Identified by referrer, not user-agent. Covered at the end of this page and in AI visibility.
Live AI-answer fetches
A crawler fetching a page live, on behalf of one specific human question. These are the only crawlers the AI Visibility Report's prompt inference runs on.
| User-agent | Provider | Category |
|---|---|---|
ChatGPT-User | ChatGPT | ai_answer |
Perplexity-User | Perplexity | ai_answer |
Claude-User | Claude | ai_answer |
Gemini | Gemini | ai_answer |
Amazonbot | Amazon | ai_answer |
Indexing and search crawlers
Bots that crawl to build or refresh a search index ahead of time — classic search engines, and each AI provider's own proactive index crawler. Architecturally these are Googlebot-shaped: systematic, broad, not tied to a live question.
| User-agent | Provider | Category |
|---|---|---|
Googlebot | Googlebot | indexing |
Bingbot | Bing | indexing |
Applebot | Apple | indexing |
DuckDuckBot | DuckDuckGo | indexing |
YandexBot | Yandex | indexing |
Baiduspider | Baidu | indexing |
Slurp | Yahoo | indexing |
OAI-SearchBot | ChatGPT | indexing |
Claude-SearchBot | Claude | indexing |
PerplexityBot | Perplexity | indexing |
Training crawlers
Bots that crawl to build or improve a model, not to answer a live question. Some — the -Extended agents — are dual-purpose opt-out signals; Kymo files them here, under their better-documented purpose.
| User-agent | Provider | Category |
|---|---|---|
GPTBot | ChatGPT | training |
ClaudeBot | Claude | training |
anthropic-ai | Claude | training |
Google-Extended | training | |
Applebot-Extended | Apple | training |
Bytespider | ByteDance | training |
CCBot | Common Crawl | training |
Meta-ExternalAgent | Meta | training |
FacebookBot | Meta | training |
Diffbot | Diffbot | training |
Omgilibot | Omgili | training |
Timpibot | Timpi | training |
AI-referred humans are not a crawler category
When a real person clicks a link in an AI answer and lands on your site, that is a human pageview with an AI referrer — chatgpt.com, perplexity.ai, claude.ai, or gemini.google.com. It is detected by referrer and UTM, not by user-agent, and it is the strongest signal that an AI actually cited you to someone. Do not confuse it with the live ai_answer crawlers above: one is a machine fetching, the other is a human arriving.
Keeping it current
The directory lives server-side. When a new crawler appears or a provider ships a new agent, Kymo updates one file and every customer's classification changes immediately — you never edit a snippet or redeploy. If you think a bot is miscategorized, that is a directory fix on Kymo's end, not a change you make.
Questions
Does Kymo detect AI crawlers without JavaScript?
Yes. Kymo detects AI crawlers server-side, from the HTTP request itself. Your server sends the request's user agent to Kymo's /api/crawl endpoint, which matches it against the crawler registry and records the visit. The crawler never runs a script, so JavaScript execution is not required. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Applebot and the rest are all detected this way. This requires the server-side snippet, not the JavaScript tag. The beat.js script tag runs in a browser and measures human visitors. A crawler requests your page, reads the HTML and leaves without executing anything, so a JavaScript tag cannot see it. Install the server-side snippet for Next.js, Express, Astro, SvelteKit, Cloudflare Workers or plain fetch to record crawlers. Most sites install both.
What is the difference between the JavaScript tag and the server-side snippet?
The JavaScript tag measures humans and the server-side snippet measures machines. The beat.js tag loads in a real browser, so it records people: pageviews, referrers, outbound clicks and goals. The server-side snippet runs on your own server before any HTML is sent, so it records every request, including the ones from crawlers that never execute a script. A site with only the tag sees zero crawlers, and a site with only the snippet sees no human behaviour. Most sites install both.
Which AI crawlers does Kymo detect?
Kymo detects the major AI crawlers and classifies each one by purpose. Live AI-answer fetches include ChatGPT-User, Claude-User, Perplexity-User, Gemini and Amazonbot. Indexing crawlers include OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot. Training crawlers include GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Bytespider and CCBot. The full registry, with the user agent and robots.txt token for each, is at https://kymo.in/crawlers. The registry lives server-side, so a new crawler is recognised for every site at once with no change on your end.
Can Kymo detect an AI agent that sends an ordinary browser user agent?
Partially, and the coverage is deliberately limited. An agent that sends an ordinary Chrome user agent matches no crawler pattern, so the user agent alone cannot identify it. The server-side snippet samples a small share of ordinary requests, about 1 in 50 by default, and sends the IP address with them. Kymo resolves that address to the network it came from, in memory, and discards it immediately. The IP is never stored, never logged and never written anywhere. A row is recorded only when the network resolves to a datacenter, because a request from a home or mobile network is treated as a person and stores nothing at all. These rows are filed as undeclared and carry no provider name, since the network says where a request came from and not who runs the agent. Treat this as a signal that automated traffic is present, not as a count of it.