# The bot taxonomy

> Kymo detects AI crawlers server-side from the HTTP request, so GPTBot, ClaudeBot and PerplexityBot are recorded without JavaScript, and sorted by purpose.

Canonical: https://kymo.in/docs/concepts/bots
Last updated: 2026-09-28

Every crawler that hits your site is classified by its user-agent against a single directory Kymo maintains server-side. Classification is case-insensitive, ordered, and first-match-wins: more specific patterns are checked before broader ones (so `Applebot-Extended` is matched before `Applebot`).

There are two different things people call "AI traffic," and Kymo keeps them apart:

- **Crawlers** — bots that fetch your pages. Identified by user-agent. Grouped into the three categories below.
- **AI-referred humans** — a real person who clicked through to your site from an AI answer. Identified by referrer, not user-agent. Covered at the end of this page and in [AI visibility](https://kymo.in/docs/concepts/ai-visibility).

## Live AI-answer fetches

A crawler fetching a page live, on behalf of one specific human question. These are the only crawlers the AI Traffic page counts as reads, and the only ones its prompt inference runs on.

| User-agent | Provider | Category |
| --- | --- | --- |
| `ChatGPT-User` | ChatGPT | ai_answer |
| `Perplexity-User` | Perplexity | ai_answer |
| `Claude-User` | Claude | ai_answer |
| `Gemini` | Gemini | ai_answer |
| `Amazonbot` | Amazon | ai_answer |

## Indexing and search crawlers

Bots that crawl to build or refresh a search index ahead of time — classic search engines, and each AI provider's own proactive index crawler. Architecturally these are Googlebot-shaped: systematic, broad, not tied to a live question.

| User-agent | Provider | Category |
| --- | --- | --- |
| `Googlebot` | Googlebot | indexing |
| `Bingbot` | Bing | indexing |
| `Applebot` | Apple | indexing |
| `DuckDuckBot` | DuckDuckGo | indexing |
| `YandexBot` | Yandex | indexing |
| `Baiduspider` | Baidu | indexing |
| `Slurp` | Yahoo | indexing |
| `OAI-SearchBot` | ChatGPT | indexing |
| `Claude-SearchBot` | Claude | indexing |
| `PerplexityBot` | Perplexity | indexing |

> **Warning:** `OAI-SearchBot`, `Claude-SearchBot`, and `PerplexityBot` are indexing bots, not AI-answer traffic. They share a brand with `ChatGPT-User`, `Claude-User`, and `Perplexity-User`, but they crawl to build a search index — they are not a live per-question fetch. Filing them as AI-answer traffic lets systematic index sweeps (hundreds of pages a day, near-uniform counts) masquerade as real user interest and pollutes the reads on the AI Traffic page. Kymo classifies them as indexing.

## Training crawlers

Bots that crawl to build or improve a model, not to answer a live question. Some — the `-Extended` agents — are dual-purpose opt-out signals; Kymo files them here, under their better-documented purpose.

| User-agent | Provider | Category |
| --- | --- | --- |
| `GPTBot` | ChatGPT | training |
| `ClaudeBot` | Claude | training |
| `anthropic-ai` | Claude | training |
| `Google-Extended` | Google | training |
| `Applebot-Extended` | Apple | training |
| `Bytespider` | ByteDance | training |
| `CCBot` | Common Crawl | training |
| `Meta-ExternalAgent` | Meta | training |
| `FacebookBot` | Meta | training |
| `Diffbot` | Diffbot | training |
| `Omgilibot` | Omgili | training |
| `Timpibot` | Timpi | training |

## AI-referred humans are not a crawler category

When a real person clicks a link in an AI answer and lands on your site, that is a human pageview with an AI referrer — `chatgpt.com`, `perplexity.ai`, `claude.ai`, or `gemini.google.com`. It is detected by referrer and UTM, not by user-agent, and it is the strongest signal that an AI actually cited you to someone. Do not confuse it with the live `ai_answer` crawlers above: one is a machine fetching, the other is a human arriving.

## Keeping it current

The directory lives server-side. When a new crawler appears or a provider ships a new agent, Kymo updates one file and every customer's classification changes immediately — you never edit a snippet or redeploy. If you think a bot is miscategorized, that is a directory fix on Kymo's end, not a change you make.

## Questions

### Does Kymo detect AI crawlers without JavaScript?

Yes. Kymo detects AI crawlers server-side, from the HTTP request itself. Your server sends the request's user agent to Kymo's /api/crawl endpoint, which matches it against the crawler registry and records the visit. The crawler never runs a script, so JavaScript execution is not required. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Applebot and the rest are all detected this way. This requires the server-side snippet, not the JavaScript tag. The beat.js script tag runs in a browser and measures human visitors. A crawler requests your page, reads the HTML and leaves without executing anything, so a JavaScript tag cannot see it. Install the server-side snippet for Next.js, Express, Astro, SvelteKit, Cloudflare Workers or plain fetch to record crawlers. Most sites install both.

### What is the difference between the JavaScript tag and the server-side snippet?

The JavaScript tag measures humans and the server-side snippet measures machines. The beat.js tag loads in a real browser, so it records people: pageviews, referrers, outbound clicks and goals. The server-side snippet runs on your own server before any HTML is sent, so it records every request, including the ones from crawlers that never execute a script. A site with only the tag sees zero crawlers, and a site with only the snippet sees no human behaviour. Most sites install both.

### Which AI crawlers does Kymo detect?

Kymo detects the major AI crawlers and classifies each one by purpose. Live AI-answer fetches include ChatGPT-User, Claude-User, Perplexity-User, Gemini and Amazonbot. Indexing crawlers include OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot. Training crawlers include GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Bytespider and CCBot. The full registry, with the user agent and robots.txt token for each, is at https://kymo.in/crawlers. The registry lives server-side, so a new crawler is recognised for every site at once with no change on your end.

### Can Kymo detect an AI agent that sends an ordinary browser user agent?

Partially, and the coverage is deliberately limited. An agent that sends an ordinary Chrome user agent matches no crawler pattern, so the user agent alone cannot identify it. The server-side snippet samples a small share of ordinary requests, about 1 in 50 by default, and sends the IP address with them. Kymo resolves that address to the network it came from, in memory, and discards it immediately. The IP is never stored, never logged and never written anywhere. A row is recorded only when the network resolves to a datacenter, because a request from a home or mobile network is treated as a person and stores nothing at all. These rows are filed as undeclared and carry no provider name, since the network says where a request came from and not who runs the agent. Treat this as a signal that automated traffic is present, not as a count of it.