AI Overview Tracking: Why the Same Search Shows New Answers
- AI overview tracking is hard because the answer changes between runs of the same question, not just between different questions.
- A January 2026 study found that fewer than 1 in 100 runs of the same prompt returned the same list of brands.
- A visit from an AI Overview arrives with a google.com referrer, so no analytics tool, Kymo included, can separate it from an ordinary Google click.
- What you can record is which pages AI crawlers read, and whether search visits to those pages rise or fall.
- Kymo measures what AI engines did on your site. It does not measure what an assistant said about you.
The same search shows new answers because the model takes a different sampling path each time it runs. Two people asking the identical question minutes apart can get two different lists of brands, and often do. Any ai overview rank tracker that reports a single position is reporting one sample and calling it a ranking.
The same prompt rarely returns the same answer twice
Start with the largest dataset available. a January 2026 SparkToro and Gumshoe.ai study ran 12 prompts through ChatGPT, Claude and Google AI Overviews/AI Mode 2,961 times with 600 volunteers. Fewer than 1 in 100 runs returned the same list of brands. Fewer than 1 in 1,000 returned the same list in the same order. The study discloses that Gumshoe sells AI tracking, so read it knowing that. The finding holds up anyway.
This is not one provider having a bad week. Thinking Machines Lab showed in September 2025 that 1,000 completions of the same prompt at temperature 0 produced 80 unique completions. Server load changed the batch size, and that changed the math mid-flight.
Models also change underneath you. a 2023 paper by Chen, Zaharia and Zou measured GPT-4 scoring 84% on a prime-number task in March 2023 and 51% in June 2023. Same model name, different answers. OpenAI rolled back a GPT-4o update on April 29, 2025 because the model had turned overly flattering.
So "how to track ai overviews" has a boring answer at the ranking level. You cannot track a position that does not exist. You can track what the engines did on your own site.
One number nobody can hand you
There is a limit worth stating plainly. A visit that came from an AI Overview lands with a google.com referrer, exactly like a normal organic click. No analytics tool can split those two apart. Kymo does not claim to, and neither can anyone else. If a dashboard shows you "AI Overview traffic" as a clean figure, it is estimating.
Pew Research Center tested this in July 2025 with 900 US adults. Users clicked a result in 8% of visits to a Google page with an AI summary, against 15% without one. They clicked a link inside the summary in 1% of visits.
What you can actually measure
| Question | Measurable? | How |
|---|---|---|
| Did an assistant name my brand in an answer? | No | Not from your server. Kymo does not report this. |
| Did an AI crawler read a specific page? | Yes | Server-side classification, per page, with the crawler's user-agent kept on its own row |
| Did a live fetch pull a page for one person's question? | Yes | ChatGPT-User (OpenAI's live fetcher) and similar agents |
| Did search visits to that page move after a crawl? | Yes, as a trend | Compare periods |
| Was this specific visit from an AI Overview? | No | The referrer is google.com either way |
The crawlers split into three jobs. Indexing crawlers such as OAI-SearchBot (OpenAI's search index crawler) and PerplexityBot work ahead of time, building the index an answer is later drawn from. Live fetchers pull a page in the moment because one person asked something. Training crawlers such as CCBot (Common Crawl) take bulk copies for future models and send no traffic.
Blocking a training crawler does not remove you from that operator's answers. The indexers and the live fetchers are separate tokens. Anthropic's Claude-SearchBot is the indexer and Claude-User is the live fetcher, though plenty of directories have those two swapped. Google-Extended (a robots.txt control token, not a crawler) belongs in none of these groups. It never appears in a log.
Sampling prompts is a shaky way to build a score
Most AI visibility tools type prompts into assistants, count how often your brand appears, and sell that percentage as a score. SparkToro's Rand Fishkin called a reported ranking position in AI "full of baloney". He is right, and the spread shows why.
Digital Applied put one brand through one dataset in 2026 and got 20% on mention-based share of voice, 16.8% on position-weighted, and 31.4% on citation-based. Same data, three scores, depending on which formula the tool picked. Ahrefs compared 540,000 query pairs in December 2025 and found Google AI Mode and AI Overviews cited the same URLs only 13.7% of the time, with 86% semantic similarity. Two surfaces from the same company, pointing at different pages.
Kymo takes the other route. It samples nothing. It records what AI engines actually did on your own site: a read, a visit, a sale, per page. That is narrower than a mention tracker, and honest about it. Mention counting is not a feature Kymo offers.
Where the crawler picture meets your analytics
Kymo counts visitors without cookies. The identifier is a salted server-side hash that rotates at UTC midnight, so a unique visitor is unique per day. Raw IPs are never stored, and no human visitor's user-agent is stored. Crawler user-agents are kept on the crawler's own event row so a classification stays auditable. Privacy-First Analytics: What It Actually Means goes deeper on that.
Pair crawl data with search rows and you get something useful: the pages an AI indexer read last week, next to the search visits those pages got this week. You will not get attribution. You will get a direction. SEO vs AEO: Do You Need Both in 2026? explains why that direction still matters, and AEO vs AIO: Which One Are People Talking About? untangles the naming.
On llms.txt: publishing one is a cheap bet, not a ranking lever. Adoption is partial. No major operator has publicly committed to honouring it, and publishing the file is not the same as a crawler fetching it. Only your own server sees which of those happened. llms.txt best practices has the detail, and The Small Business Guide to AEO walks through the rest of the workflow.
Stop guessing. Watch.
Most AI visibility tools ask a chatbot and report a guess. Kymo watches your own site instead: which pages AI engines read, how many people their answers send you, and which of those people buy. Every number comes from your server, not from a prompt someone else chose.
Start free → 14-day free trial. No card required.