Measured data
AI crawler activity on two live sites
This page counts every request from a recognised AI or search crawler to two websites Kymo runs, kymo.in and remotestack.in, in the 30 days up to 25 Oct 2026. Kymo records each request on the server, so the counts include crawlers that never run JavaScript.
In that window, 16 operators made 3,610,287 crawler requests to the two sites. Answer fetch requests were <1%, Search index requests were 53%, Training requests were 46%, Other requests were <1%.
Requests by type
| Type | Requests, 30 days | Share | What it means |
|---|---|---|---|
| Answer fetch | 22,365 | <1% | An assistant fetched a page because a person asked a question. |
| Search index | 1,915,893 | 53% | A crawler read a page to add it to a search index, classic or AI. |
| Training | 1,666,138 | 46% | A crawler collected the page for model training or a public dataset. |
| Other | 5,891 | <1% | Link previews, ads checks and commercial data tools. |
Requests by operator
| Operator | Type | Agent | Requests | Days active | Last seen |
|---|---|---|---|---|---|
| Meta | Training | meta-externalagent or FacebookBot | 1,612,740 | 29 of 31 | 24 Oct 2026 |
| OpenAI | Search index | OAI-SearchBot | 1,451,869 | 31 of 31 | 25 Oct 2026 |
| Apple | Search index | Applebot | 133,377 | 31 of 31 | 25 Oct 2026 |
| Search index | Googlebot | 101,345 | 31 of 31 | 25 Oct 2026 | |
| Microsoft | Search index | bingbot | 90,270 | 31 of 31 | 25 Oct 2026 |
| Baidu | Search index | Baiduspider | 61,266 | 31 of 31 | 25 Oct 2026 |
| Amazon | Training | Amazonbot | 42,865 | 31 of 31 | 25 Oct 2026 |
| Huawei | Search index | PetalBot | 31,176 | 31 of 31 | 25 Oct 2026 |
| Perplexity | Search index | PerplexityBot | 27,815 | 31 of 31 | 25 Oct 2026 |
| Yandex | Search index | YandexBot | 12,102 | 26 of 31 | 24 Oct 2026 |
| Anthropic | Answer fetch | Claude-User | 10,527 | 31 of 31 | 25 Oct 2026 |
| OpenAI | Answer fetch | ChatGPT-User | 9,809 | 31 of 31 | 25 Oct 2026 |
| Meta | Search index | Meta-WebIndexer | 5,870 | 31 of 31 | 25 Oct 2026 |
| Meta | Other | facebookexternalhit or Meta-ExternalAds | 5,697 | 31 of 31 | 25 Oct 2026 |
| Anthropic | Training | ClaudeBot, Claude-Web or anthropic-ai | 4,469 | 31 of 31 | 25 Oct 2026 |
| ByteDance | Training | Bytespider | 3,451 | 31 of 31 | 25 Oct 2026 |
| Common Crawl | Training | CCBot | 1,664 | 24 of 31 | 23 Oct 2026 |
| DuckDuckGo | Answer fetch | DuckAssistBot | 1,194 | 29 of 31 | 25 Oct 2026 |
| OpenAI | Training | GPTBot | 499 | 29 of 31 | 24 Oct 2026 |
| Cohere | Training | cohere-ai | 450 | 23 of 31 | 23 Oct 2026 |
| Anthropic | Search index | Claude-SearchBot | 448 | 24 of 31 | 23 Oct 2026 |
| Perplexity | Answer fetch | Perplexity-User | 437 | 24 of 31 | 23 Oct 2026 |
| DuckDuckGo | Search index | DuckDuckBot | 355 | 31 of 31 | 25 Oct 2026 |
| Mistral AI | Answer fetch | MistralAI-User | 235 | 11 of 31 | 23 Oct 2026 |
| OpenAI | Other | OAI-AdsBot | 194 | 12 of 31 | 23 Oct 2026 |
| Meta | Answer fetch | Meta-ExternalFetcher | 163 | 11 of 31 | 23 Oct 2026 |
How these numbers are counted
- The two sites are remotestack.in and kymo.in. Both are sites Kymo runs, and no customer's traffic is ever included.
- Each request is classified on the server by its user agent, against the AI crawler directory. Kymo stores the operator and the type and never the user agent, so where one operator runs two agents of the same type, the row names both.
- A user agent is self-reported, and a request can claim to be a crawler it is not. Each crawler page says how to verify a request against the operator's published IP ranges.
- Older counts are stored by the hour, so the oldest edge of the window can include up to one extra hour.
- Days active counts UTC dates. A 30-day window that ends mid-day touches part of 31 dates, so a crawler seen every day reads 31 of 31.
- Two sites is a small sample. Treat the figures as evidence of what is active now, not as a share of the whole web.
See which of these crawlers read your own site, page by page.
Start free →14-day free trial. No card required.