anthropic-ai: user agent, purpose, and whether to allow it
anthropic-ai is no longer in use. A robots.txt rule naming anthropic-ai has no effect today. It was replaced by ClaudeBot, which is the token to address instead.
Deprecated Anthropic user agent, replaced by ClaudeBot.
Specification
| Operator | Anthropic |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Anthropic. Kymo matches this agent on the token anthropic-ai, observed in real logs rather than published in operator documentation. |
| robots.txt token | anthropic-ai |
| robots.txt compliance | No operator statement located |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Deprecated |
| Legacy Kymo category | training |
| Source | Third-party source, not yet confirmed against operator docshttps://support.claude.com/en/articles/8896518 |
| Verified on |
What blocking it costs you
A rule for anthropic-ai has no current effect. Many published robots.txt templates still include it, which gives site owners false confidence that Anthropic training is blocked when it is not.
robots.txt directives
Both blocks below address anthropic-ai only. Rules for one token never apply to another, even from the same operator.
Allow anthropic-ai
User-agent: anthropic-ai Allow: /
Block anthropic-ai
User-agent: anthropic-ai Disallow: /
Questions
What is the anthropic-ai user agent?
Anthropic documents the robots.txt token anthropic-ai but does not publish a full user-agent string for it. Kymo records this agent from observation rather than from documentation.
Does anthropic-ai respect robots.txt?
No operator statement located
What does blocking anthropic-ai cost me?
A rule for anthropic-ai has no current effect. Many published robots.txt templates still include it, which gives site owners false confidence that Anthropic training is blocked when it is not.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so anthropic-ai shows up under Anthropic with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.