Claude-SearchBot: user agent, purpose, and whether to allow it
Claude-SearchBot indexes content to improve the quality of Claude search results. It is not a training crawler.
Specification
| Operator | Anthropic |
|---|---|
| Purpose | Search indexBuilds the index an AI product cites from. Blocking it removes you from that product's answers. This is the expensive block, and the one most often made by accident. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Anthropic. Kymo matches this agent on the token Claude-SearchBot, observed in real logs rather than published in operator documentation. |
| robots.txt token | Claude-SearchBot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | Reverse DNS, forward confirmed |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | indexing |
| Source | Operator documentationhttps://support.claude.com/en/articles/8896518 |
| Verified on |
What blocking it costs you
Blocking Claude-SearchBot reduces your visibility in Claude search results. It has no effect on training, which is governed by ClaudeBot. Note that several published crawler directories state the reverse; they are wrong.
robots.txt directives
Both blocks below address Claude-SearchBot only. Rules for one token never apply to another, even from the same operator.
Allow Claude-SearchBot
User-agent: Claude-SearchBot Allow: /
Block Claude-SearchBot
User-agent: Claude-SearchBot Disallow: /
Questions
What is the Claude-SearchBot user agent?
Anthropic documents the robots.txt token Claude-SearchBot but does not publish a full user-agent string for it. Kymo records this agent from observation rather than from documentation.
Does Claude-SearchBot respect robots.txt?
Documented: respects robots.txt
What does blocking Claude-SearchBot cost me?
Blocking Claude-SearchBot reduces your visibility in Claude search results. It has no effect on training, which is governed by ClaudeBot. Note that several published crawler directories state the reverse; they are wrong.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so Claude-SearchBot shows up under Anthropic with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.