ClaudeBot: Anthropic's Training Crawler
What is ClaudeBot?
ClaudeBot collects web content that may contribute to training Anthropic models. Restricting it signals that your future content should be excluded from training datasets. ClaudeBot is separate from Claude-SearchBot and Claude-User, so blocking ClaudeBot stops training collection without removing you from Claude answers.
Key facts
- ClaudeBot is operated by Anthropic, and its purpose is training: collects content that may be used to train future models.
- Anthropic publishes no full user-agent string for ClaudeBot, so Kymo matches the token ClaudeBot observed in real logs.
- The robots.txt token for ClaudeBot is ClaudeBot, and Anthropic documents that ClaudeBot respects that rule.
- ClaudeBot never sends a visitor to your site, and ClaudeBot attaches no citation when it uses your content.
Specification
| Operator | Anthropic |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Anthropic. Kymo matches this agent on the token ClaudeBot, observed in real logs rather than published in operator documentation. |
| robots.txt token | ClaudeBot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | Reverse DNS, forward confirmed |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | training |
| Source | Operator documentationhttps://support.claude.com/en/articles/8896518 |
| Verified on |
What this looks like in your logs
grep -i 'ClaudeBot' access.log
Anthropic publishes no full user-agent string for ClaudeBot, so the rest of the line varies and only the token ClaudeBot is dependable. Kymo matches on that token, case-insensitively, for the same reason.
What ClaudeBot does not do
ClaudeBot does not send you a visitor, and it attaches no citation, so a read by it can never turn into a click.
ClaudeBot does not decide whether Anthropic can find you today, because a training run happens long before any question is asked.
Anthropic publishes no crawl frequency for ClaudeBot, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files ClaudeBot under Anthropic as training, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from ClaudeBot appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. A training run sends nobody, so a read by it has no click to compare against.
Allow it or block it
Blocking ClaudeBot excludes your content from future training runs. It does not stop Claude from reaching your site: Claude-User and Claude-SearchBot are separate tokens and must be addressed individually. Anthropic supports the non-standard Crawl-delay directive, so rate-limiting is available before outright blocking.
| Allow it if | Block it if |
|---|---|
| You want your content represented in future Anthropic models, and you accept that a training run sends no visitor and no link back. | You sell access to your content, or you object to it being used as training data without payment or credit. |
robots.txt directives
Both blocks below address ClaudeBot only. Rules for one token never apply to another, even from the same operator.
Allow ClaudeBot
User-agent: ClaudeBot Allow: /
Block ClaudeBot
User-agent: ClaudeBot Disallow: /
Questions
How do I know a request claiming to be ClaudeBot is genuine?
Run a reverse DNS lookup on the requesting address, confirm it ends in null, then run a forward lookup on that name and check it returns the same address. A user-agent string alone proves nothing.
Does blocking ClaudeBot also block Claude-SearchBot?
No. ClaudeBot and Claude-SearchBot are separate tokens, and Anthropic treats each one on its own. A rule naming ClaudeBot leaves Claude-SearchBot free to carry on indexing your pages for Anthropic answers. To stop that as well, name Claude-SearchBot in a rule of its own.
Will I see ClaudeBot in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and ClaudeBot reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does ClaudeBot send traffic back to my site?
No. ClaudeBot takes content and sends nothing back. A hit appears in your logs as a request with no visitor behind it.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
ClaudeBot takes your content and sends nothing back, so it leaves no trace at all in a browser-based analytics tool.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.