ClaudeBot: user agent, purpose, and whether to allow it
ClaudeBot collects web content that may contribute to training Anthropic models. Restricting it signals that your future content should be excluded from training datasets.
Specification
| Operator | Anthropic |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Anthropic. Kymo matches this agent on the token ClaudeBot, observed in real logs rather than published in operator documentation. |
| robots.txt token | ClaudeBot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | Reverse DNS, forward confirmed |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | training |
| Source | Operator documentationhttps://support.claude.com/en/articles/8896518 |
| Verified on |
What blocking it costs you
Blocking ClaudeBot excludes your content from future training runs. It does not stop Claude from reaching your site: Claude-User and Claude-SearchBot are separate tokens and must be addressed individually. Anthropic supports the non-standard Crawl-delay directive, so rate-limiting is available before outright blocking.
robots.txt directives
Both blocks below address ClaudeBot only. Rules for one token never apply to another, even from the same operator.
Allow ClaudeBot
User-agent: ClaudeBot Allow: /
Block ClaudeBot
User-agent: ClaudeBot Disallow: /
Questions
What is the ClaudeBot user agent?
Anthropic documents the robots.txt token ClaudeBot but does not publish a full user-agent string for it. Kymo records this agent from observation rather than from documentation.
Does ClaudeBot respect robots.txt?
Documented: respects robots.txt
What does blocking ClaudeBot cost me?
Blocking ClaudeBot excludes your content from future training runs. It does not stop Claude from reaching your site: Claude-User and Claude-SearchBot are separate tokens and must be addressed individually. Anthropic supports the non-standard Crawl-delay directive, so rate-limiting is available before outright blocking.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so ClaudeBot shows up under Anthropic with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.