Claude-SearchBot: Anthropic's Search Crawler
What is Claude-SearchBot?
Claude-SearchBot indexes content to improve the quality of Claude search results. It is not a training crawler. Claude-SearchBot builds an Anthropic search index rather than training data, and several published crawler directories state the reverse.
Key facts
- Claude-SearchBot is operated by Anthropic, and its purpose is search index: builds the index an AI product cites from.
- Anthropic publishes no full user-agent string for Claude-SearchBot, so Kymo matches the token Claude-SearchBot observed in real logs.
- The robots.txt token for Claude-SearchBot is Claude-SearchBot, and Anthropic documents that Claude-SearchBot respects that rule.
- Claude-SearchBot can send a visitor to your site, and Claude-SearchBot attaches a linked citation when it uses your content.
Specification
| Operator | Anthropic |
|---|---|
| Purpose | Search indexBuilds the index an AI product cites from. Blocking it removes you from that product's answers. This is the expensive block, and the one most often made by accident. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Anthropic. Kymo matches this agent on the token Claude-SearchBot, observed in real logs rather than published in operator documentation. |
| robots.txt token | Claude-SearchBot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | Reverse DNS, forward confirmed |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | indexing |
| Source | Operator documentationhttps://support.claude.com/en/articles/8896518 |
| Verified on |
What this looks like in your logs
grep -i 'Claude-SearchBot' access.log
Anthropic publishes no full user-agent string for Claude-SearchBot, so the rest of the line varies and only the token Claude-SearchBot is dependable. Kymo matches on that token, case-insensitively, for the same reason.
What Claude-SearchBot does not do
Claude-SearchBot does not guarantee a visit, because a read only becomes a visit if a person clicks the citation that came out of it.
Claude-SearchBot does not train a model on what it reads, so blocking it changes what Anthropic can answer and not what Anthropic learns.
Anthropic publishes no crawl frequency for Claude-SearchBot, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files Claude-SearchBot under Anthropic as indexing, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from Claude-SearchBot appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. An index sweep produces no click to miss, so counting it there would inflate the read number and break its relationship to the visitor number beside it.
Seen on our own sites
| Last seen | 14 Sep 2026 |
|---|---|
| Days active, last 30 days | 7 |
| Requests, last 30 days | 20 |
Counted by Kymo on kymo.in and remotestack.in, the two sites we run, up to 15 Sep 2026. Two sites is a small sample. Treat these numbers as proof the bot is active now.
Allow it or block it
Blocking Claude-SearchBot reduces your visibility in Claude search results. It has no effect on training, which is governed by ClaudeBot. Note that several published crawler directories state the reverse; they are wrong.
| Allow it if | Block it if |
|---|---|
| You want to appear in Anthropic answers, because this crawler builds the index those answers cite. | You accept disappearing from Anthropic answers. This is the most expensive block on this page, and it is the one made most often by accident. |
robots.txt directives
Both blocks below address Claude-SearchBot only. Rules for one token never apply to another, even from the same operator.
Allow Claude-SearchBot
User-agent: Claude-SearchBot Allow: /
Block Claude-SearchBot
User-agent: Claude-SearchBot Disallow: /
Questions
How do I know a request claiming to be Claude-SearchBot is genuine?
Run a reverse DNS lookup on the requesting address, confirm it ends in null, then run a forward lookup on that name and check it returns the same address. A user-agent string alone proves nothing.
Does blocking Claude-SearchBot also block ClaudeBot?
No. Claude-SearchBot and ClaudeBot are separate tokens, and Anthropic treats each one on its own. A rule naming Claude-SearchBot leaves ClaudeBot free to carry on collecting your content for Anthropic model training. To stop that as well, name ClaudeBot in a rule of its own.
Will I see Claude-SearchBot in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and Claude-SearchBot reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does Claude-SearchBot send traffic back to my site?
Yes. Claude-SearchBot builds an index that Anthropic cites its sources from, and a citation there is a link a person can click. A crawl is still not a visit, so expect the read count to run far ahead of the click count.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Claude-SearchBot decides whether you appear in Anthropic answers, and a browser tag cannot tell you whether it ever came.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.