Skip to content
Measured data

AI crawler activity on two live sites

This page counts every request from a recognised AI or search crawler to two websites Kymo runs, kymo.in and remotestack.in, in the 30 days up to 25 Oct 2026. Kymo records each request on the server, so the counts include crawlers that never run JavaScript.

In that window, 16 operators made 3,610,287 crawler requests to the two sites. Answer fetch requests were <1%, Search index requests were 53%, Training requests were 46%, Other requests were <1%.

Requests by type

TypeRequests, 30 daysShareWhat it means
Answer fetch22,365<1%An assistant fetched a page because a person asked a question.
Search index1,915,89353%A crawler read a page to add it to a search index, classic or AI.
Training1,666,13846%A crawler collected the page for model training or a public dataset.
Other5,891<1%Link previews, ads checks and commercial data tools.

Requests by operator

26 operator and type pairs, 30 days to 25 Oct 2026
OperatorTypeAgentRequestsDays activeLast seen
MetaTrainingmeta-externalagent or FacebookBot1,612,74029 of 3124 Oct 2026
OpenAISearch indexOAI-SearchBot1,451,86931 of 3125 Oct 2026
AppleSearch indexApplebot133,37731 of 3125 Oct 2026
GoogleSearch indexGooglebot101,34531 of 3125 Oct 2026
MicrosoftSearch indexbingbot90,27031 of 3125 Oct 2026
BaiduSearch indexBaiduspider61,26631 of 3125 Oct 2026
AmazonTrainingAmazonbot42,86531 of 3125 Oct 2026
HuaweiSearch indexPetalBot31,17631 of 3125 Oct 2026
PerplexitySearch indexPerplexityBot27,81531 of 3125 Oct 2026
YandexSearch indexYandexBot12,10226 of 3124 Oct 2026
AnthropicAnswer fetchClaude-User10,52731 of 3125 Oct 2026
OpenAIAnswer fetchChatGPT-User9,80931 of 3125 Oct 2026
MetaSearch indexMeta-WebIndexer5,87031 of 3125 Oct 2026
MetaOtherfacebookexternalhit or Meta-ExternalAds5,69731 of 3125 Oct 2026
AnthropicTrainingClaudeBot, Claude-Web or anthropic-ai4,46931 of 3125 Oct 2026
ByteDanceTrainingBytespider3,45131 of 3125 Oct 2026
Common CrawlTrainingCCBot1,66424 of 3123 Oct 2026
DuckDuckGoAnswer fetchDuckAssistBot1,19429 of 3125 Oct 2026
OpenAITrainingGPTBot49929 of 3124 Oct 2026
CohereTrainingcohere-ai45023 of 3123 Oct 2026
AnthropicSearch indexClaude-SearchBot44824 of 3123 Oct 2026
PerplexityAnswer fetchPerplexity-User43724 of 3123 Oct 2026
DuckDuckGoSearch indexDuckDuckBot35531 of 3125 Oct 2026
Mistral AIAnswer fetchMistralAI-User23511 of 3123 Oct 2026
OpenAIOtherOAI-AdsBot19412 of 3123 Oct 2026
MetaAnswer fetchMeta-ExternalFetcher16311 of 3123 Oct 2026

How these numbers are counted

  • The two sites are remotestack.in and kymo.in. Both are sites Kymo runs, and no customer's traffic is ever included.
  • Each request is classified on the server by its user agent, against the AI crawler directory. Kymo stores the operator and the type and never the user agent, so where one operator runs two agents of the same type, the row names both.
  • A user agent is self-reported, and a request can claim to be a crawler it is not. Each crawler page says how to verify a request against the operator's published IP ranges.
  • Older counts are stored by the hour, so the oldest edge of the window can include up to one extra hour.
  • Days active counts UTC dates. A 30-day window that ends mid-day touches part of 31 dates, so a crawler seen every day reads 31 of 31.
  • Two sites is a small sample. Treat the figures as evidence of what is active now, not as a share of the whole web.