ClaudeBot vs Claude-SearchBot: What Each One Does on Your Site
- ClaudeBot crawls your site in bulk to gather training data for Anthropic's AI models. Claude-SearchBot crawls to build a search index that powers live answers.
- Both are Anthropic crawlers, but they hit your server for different reasons and at different speeds.
- ClaudeBot can consume significant bandwidth with aggressive crawl rates. Claude-SearchBot crawls to keep an index current, and a third agent, Claude-User, is the one that fetches a page for a live question.
- Your robots.txt file can treat them differently, but blocking Claude-SearchBot means you lose visibility in Claude's search results.
- Kymo classifies both under "indexing" for Claude-SearchBot and "training" for ClaudeBot, so you can see which is doing what.
ClaudeBot and Claude-SearchBot are two different crawlers operated by Anthropic, and they affect your site in very different ways. The short answer to the title question: ClaudeBot is a training crawler, Claude-SearchBot is an indexing crawler. One wants your content stored in a model's memory, the other wants it in the index Claude searches when someone asks a question.
What ClaudeBot is built for
ClaudeBot is the user agent that shows up when Anthropic's crawlers are gathering large volumes of web content to train their models. When you see requests from ClaudeBot in your logs, those visits are bulk retrievals. The crawler is reading your pages to absorb the information, not to recommend your site to a human.
According to Anthropic's publicly available documentation, ClaudeBot respects robots.txt and standard crawl directives. The robots.txt token is ClaudeBot, and that is the string to match on. If you use Cloudflare or another proxy, you can verify the crawler by reverse DNS lookup rather than trusting the user agent, which anyone can copy.
The bandwidth problem with ClaudeBot
ClaudeBot is known for aggressive crawl behavior. That is not speculation. Many site operators have reported ClaudeBot downloading pages at high rates, sometimes causing noticeable spikes in server load. A site that receives thousands of pages fetched from ClaudeBot in a day is not unusual.
That matters for you because a training crawl is not urgent. If ClaudeBot wants to include your content in a training set, the timing usually does not matter. You can slow it down, rate-limit it, or block it entirely without losing out on AI referrals or visibility. The content is being stored for future model training, not for immediate retrieval by a user.
What Claude-SearchBot is built for
Claude-SearchBot is Anthropic's search index crawler. Its robots.txt token is Claude-SearchBot. It crawls pages to build and update the index Claude searches, and Anthropic states plainly that it is not a training crawler.
The key distinction: Claude-SearchBot visits are linked to answering questions, but not to one specific question. It works ahead of time, the way Googlebot does, so that the index is ready when someone asks. Because a page in that index can be cited, this is the crawler that can lead to referral traffic.
The live fetch is a third agent, Claude-User. That one reads a page because a person has just asked Claude something that requires it, and Kymo classifies it as an AI answer fetch. Several published crawler directories have these two the wrong way round. Claude-SearchBot is the indexer; Claude-User is the live fetcher.
Why Claude-SearchBot visits your site
When Claude-SearchBot crawls your page, your content becomes a candidate answer for the questions people ask Claude. Anthropic does not publish how it sets crawl frequency, so treat any claim about what a rising crawl rate means as a guess. What you can say from the data is simpler and more useful: this is the crawler you want visiting, because it is the one whose visits can end in a citation. ClaudeBot offers no such path.
The catch: if you block Claude-SearchBot in robots.txt, you remove your content from the index that powers Claude's answers. That is your choice to make, but you should understand the trade-off. Blocking training crawlers like ClaudeBot is low risk. Blocking indexing crawlers like Claude-SearchBot removes you from a growing discovery channel.
The table: ClaudeBot vs Claude-SearchBot
| Attribute | ClaudeBot | Claude-SearchBot |
|---|---|---|
| Purpose | Bulk collection for model training | Building and updating a web search index |
| robots.txt token | ClaudeBot | Claude-SearchBot |
| Kymo classification | training | indexing |
| Typical fetch volume | High, often aggressive | Lower, more targeted |
| Referral traffic | None | Can send clicks back to your site |
| Blocking consequence | You lose training inclusion, no immediate visibility impact | You lose visibility in Claude's search answers |
| Your typical move | Rate-limit or block if bandwidth is an issue | Keep allowed if you want AI referrals |
How Kymo classifies these crawlers
Kymo puts Anthropic's crawlers into two categories. ClaudeBot lands in the training bucket. Claude-SearchBot lands in the indexing bucket. There is also the Claude-User agent which is classified as an AI answer crawler because it performs live fetches for individual questions.
This classification matters because it changes what you do next. If you see heavy traffic from ClaudeBot, that is a bandwidth consideration. If you see visits from Claude-SearchBot, that is a visibility signal. One crawler indicates your content is being stored, the other indicates your content is being served.
You can see all of this in your Kymo dashboard under the AI crawler reports. The Reading the AI report guide explains how to distinguish between crawler types and spot patterns in which pages attract the most AI attention.
The AI referral traffic you should actually care about
When Claude cites your site in an answer, the user sees your page title and URL. If they click through, that is a referral. Kymo tracks these clicks as AI referrals, and they are distinct from crawler visits. A crawl means your content is read. A referral means a human clicked to your site from an AI answer.
This is the metric that matters. Crawler visits alone are a cost. Referral visits are a return. Claude-SearchBot helps generate both the index entry and, when your content is cited, the referral. ClaudeBot gives you neither.
If you want to understand how the broader shift from traditional search to AI answers affects your business, read the post on GEO vs SEO: What's Different and What Still Works. It covers the practical differences between optimizing for Google and optimizing for AI answer engines.
What your visitors actually do matters more than the crawler identity
You can block all Anthropic crawlers and move on. That is a valid choice if server resources are tight. But you should recognize that blocking Claude-SearchBot cuts off a channel where your content can appear in front of people who are actively looking for what you offer.
The decision starts with measurement. You should not guess whether these crawlers hit your site. Check the logs. Kymo shows you which crawler visited which page, on which day, and whether any AI referral traffic came back. The generous event limits make this easy. Solo plan pricing is $9 a month, or $90 a year, for up to 10,000 events. Studio is $29 monthly or $290 annually for up to 100,000 events. AI crawler tracking does not count against your event limit, so you can monitor all of this without inflating your usage.
What to do about robots.txt
Anthropic documents the commands each crawler honors. You can set distinct rules for ClaudeBot and Claude-SearchBot within the robots.txt file. For example, you can disallow ClaudeBot while allowing Claude-SearchBot.
A common configuration looks like this:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /Anthropic states it honors the Allow and Disallow directives for all of its crawlers. It also supports Crawl-delay, which is not part of the original robots.txt standard, so you can slow ClaudeBot down instead of blocking it outright. The practical details on building this are in our guide on robots.txt for AI Crawlers: A Copy-Paste Guide.
A note on the classification nuance
The distinction between ClaudeBot and Claude-SearchBot also shows up in how tools categorize crawler behavior. The classification of Claude-SearchBot as an indexing crawler is consistent with Anthropic's own descriptions. A live fetch for a single question is handled by Claude-User and classified by Kymo as an AI answer. That distinction is the difference between "this page was indexed" and "this page was shown to one person right now."
When you see the information in your dashboard, it is useful to track each category separately. Training crawls tell you about model coverage. Indexing crawls tell you about answer visibility. AI answer fetches tell you about immediate relevance.
You were fine on Google, and now it is quieter
If you spend a lot of time looking at Google Search Console, you may have noticed impressions declining while direct traffic holds steady. That is not necessarily a ranking problem. More people are getting answers from AI assistants. They do not always click through. The queries you used to win on Google are now being answered before the user ever reaches the results page.
Optimizing for AI visibility requires a different approach from SEO. It means making sure your content is structured in a way that AI crawlers can parse and cite. It means measuring which crawlers come and what they do with your pages. You cannot improve what you do not measure. That is the entire argument for Kymo in one sentence.
The shift toward AI answers is happening regardless of whether you participate. The PerplexityBot crawler works similarly to Claude-SearchBot for Perplexity's product, and the Claude-SearchBot (Anthropic's search index crawler) page covers the technical details here.
Track the difference, then decide
You do not need to be a data analyst to read this. The Kymo dashboard separates human visitors from AI crawlers, and separates the crawlers by category, so a ClaudeBot bandwidth problem never looks like a Claude-SearchBot visibility win. You can see which pages draw AI attention, which crawler drew it, and whether a human ever followed.
Once you can see that, the decision gets easy. Rate-limit or block ClaudeBot if the bandwidth costs you something. Keep Claude-SearchBot and Claude-User allowed unless you have a specific reason not to. Then watch for a month before you change anything else. If you want the wider strategy around this, The Small Business Guide to AEO covers the content side in one place.
If you have not installed anything yet, the free AI visibility check reads a URL from public signals only, so your site does not need Kymo installed, and the report arrives by email.
Start free → 14-day free trial. No card required.