GPTBot vs OAI-SearchBot: One Trains, One Cites

If you have seen OpenAI's bot names on your server logs and wondered why one of them matters more, here is the difference. GPTBot trains the model. OAI-SearchBot fetches a page because a person asked ChatGPT something, and ChatGPT is checking your site for the answer. One consumes your content. The other can bring you a customer.

What GPTBot actually does

GPTBot is the user agent OpenAI uses to crawl the web for training data. Its job is to read your pages in bulk and feed them into model training. This is the crawler that downloads your site's content so GPT can learn patterns, facts, tone, and structure. Your content becomes part of the model's knowledge, although not in a way you can point to later. It does not show your site to anyone. It does not send traffic back. It is a one-way street.

You can block GPTBot with a robots.txt line:

User-agent: GPTBot
Disallow: /

OpenAI supports this and says it respects the instruction. Many sites block GPTBot entirely because they see it as content extraction with no return. That is a defensible choice, but it is not the only choice. For many small sites, training crawl traffic is small and the data helps the model understand your niche. The risk is that ChatGPT answers a question using your facts without a link, and a visitor never knows you existed. That is a real cost, but it is separate from the search crawler question.

What OAI-SearchBot does instead

OAI-SearchBot is the crawler behind ChatGPT search. When someone asks ChatGPT a question that needs current information, the system fetches pages in real time and reads them to form an answer. That live fetch is classified by Kymo as an ai_answer event. It is not a bulk download. It is one person's question, triggering a check of your page, right now.

This is why the distinction matters. An ai_answer fetch means your site is in the candidate pool for an actual answer someone will read today. If ChatGPT uses your content as the basis for the answer, it cites you. That citation is a referral path. People click through from ChatGPT answers to the source page all the time. That is real, measurable traffic.

The same logic applies to Claude-User from Anthropic and Perplexity-User. Each of those is a live fetch for a person's question, not a training sweep. AI crawlers are not one homogeneous blob. They divide into three jobs: training, indexing, and answering.

The three jobs, in plain terms

Kymo's server-side receiver classifies crawler user agents into three categories. This is not a vanity dashboard feature. It tells you who is taking versus who is sending.

You can see ClaudeBot (Anthropic's training crawler) is in the training column. It is the same shape as GPTBot. It reads your content, does not send visitors. Compare that to a ChatGPT-User row, which means a person had a question and your page was retrieved as a candidate answer. That is the closest thing to a referral you can get from an AI system.

Indexing crawlers are the middle layer

OAI-SearchBot and PerplexityBot sit between training and answering. They build the index that makes live answers possible. When you see OAI-SearchBot in your logs, it is cataloging your content for OpenAI's search product. It does not send traffic directly, but it is the difference between being retrievable and being invisible.

Your robots.txt file has a direct control here. If you want to block training crawlers but allow search crawlers, use separate rules:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This lets OpenAI use your content to answer search queries while keeping it out of training iterations. That is a clean split, and it respects your control. The same token logic applies to Google-Extended (a robots.txt control token, not a crawler) and Applebot-Extended (a robots.txt control token, not a crawler). Those are control tokens that opt you out of training and other limited uses, separate from the main crawling bots.

What this means for your traffic

Search is changing. SEO still matters, but AI answer engines increasingly decide who gets found. A page can rank on page one of Google and still never appear in a ChatGPT answer. Or it can appear in a ChatGPT answer and send a steady drip of highly targeted visitors who ask very specific questions. Those visitors are often further along than a Google searcher. They asked a question in natural language, got your page as the cited source, and clicked through to read more.

You cannot know which of these crawlers hit your site if you only use Google Analytics. GA4 will show you a referral from chat.openai.com if someone clicks through, but it will not show you the crawl event that happened two days earlier. The crawl is the leading indicator. The click is the lagging indicator. If you wait for clicks, you will not see the crawler patterns that predict them. You need to measure AI crawlers directly to see what is being read before it ever becomes traffic.

This is not about AEO explained as a vague strategy. It is about the mechanics of retrieval. A live answer fetch works like this: someone asks a question, the answer engine retrieves candidate pages, reads them, picks the source that best answers the prompt, and prints a response. If you want to be the source, you need to know which of your pages get fetched. The gap between "your page was fetched" and "your page was cited" is where you improve. The gap between "cited" and "clicked through" is where you convert.

The three crawler types need different responses

Training crawlers ask you a policy question. Do you want your content in the model? ClaudeBot (Anthropic's training crawler) gives you the same choice. You can block or allow. This is a business decision, not a technical one.

Indexing crawlers ask you a visibility question. Do you want your content in ChatGPT search? If you block OAI-SearchBot, you are effectively invisible in ChatGPT search results. That is a big call for a small business that depends on discovery.

Answer crawlers ask you a content quality question. When ChatGPT-User fetches your pricing page, does it have a clear answer? Is the page structured so a crawler can pull out a specific fact? A page that answers the question cleanly is far more likely to be cited than a page that buries the answer under navigation noise.

How you track this

Kymo was built to answer exactly this question. It is a cookieless web analytics platform that shows human visitors and AI crawlers on one dashboard. No cookies, no localStorage for tracking. The visitor identifier is a salted server-side hash that rotates at UTC midnight, so unique visitors means unique per day. Raw IPs are never stored, and no human visitor's user-agent string is stored. AI crawler user agents are retained on the crawler's own event row so the classification stays auditable. A crawler is not a person.

Plans start with 14 days free, no card required. Solo is $9 a month, or $90 a year, for up to 10,000 events a month. Studio is $29 a month, or $290 a year, for up to 100,000 events a month. Every feature is included on both plans. The plans differ only by event volume. AI crawler tracking does not count against your event limit. Exceed your limit and the dashboard pauses, but your data is not deleted.

You also get standard analytics: pageviews, unique visitors, referrers, top pages, countries, devices, browsers, sessions, bounce rate, session duration. Plus outbound click tracking, goal tracking with revenue on a goal, UTM campaign tracking, and a real-time map of active visitors. There is a public demo dashboard at kymo.in/demo with no login required.

The tools that claim to do this

A lot of products promise AI visibility analysis. Most of them are bloated, expensive, or both. Many single-purpose AI trackers give you one metric and nothing else. Full SEO suites cost hundreds a month and bury the signal. Kymo is simpler than GA4, cheaper than Semrush or Ahrefs, and broader than single-purpose trackers. Built for solo builders and small teams who do not have an analyst.

Plenty of AI SEO tools for small business are worth paying for, but the first thing you need is measurement. The rest of the workflow, content tuning, prompt alignment, structured data, all of that comes after you know what gets fetched.

The full mental model is in The Small Business Guide to AEO. It walks through how answer engines retrieve and cite sources, and what it takes, step by step. The short version is that GPTBot and OAI-SearchBot are not the same thing, and treating them as one problem produces muddled decisions. If you are losing discovery to AI answers, the first step is knowing which crawlers visit your site, what they read, and whether any of it turns into clicks.

Start with measurement

Blocking everything is fear. Ignoring everything is denial. The middle path is to see the data and decide from there. A tool like Kymo gives you both the human and the crawler picture, so you know if you are being read, being trained on, being cited, or being skipped.

Use the free AI visibility check if you do not want to install anything yet. It reads a URL from public signals only, so the site does not need Kymo installed. The report arrives by a magic link over email. It shows which AI assistants can currently access your content and where the gaps are.

What you should do next

Start tracking which crawlers hit your site, then tune your content for fetch-to-citation rate. Block what you do not want to train, allow what can send visitors, and measure the difference. If you are a solo builder or small team losing discovery to AI answers, start tracking it to see which of these crawlers actually visit your site, or read the documentation on how Kymo classifies them. The free check reads a URL from public signals so your site does not need Kymo installed, and the report arrives by email.

Start free: 14-day trial, no card required.