anthropic-ai: Anthropic's Retired Crawler
anthropic-ai is no longer in use. A robots.txt rule naming anthropic-ai has no effect today. It was replaced by ClaudeBot, which is the token to address instead.
What is anthropic-ai?
Deprecated Anthropic user agent, replaced by ClaudeBot. anthropic-ai was replaced by ClaudeBot, and many robots.txt templates still list it, which gives false confidence that Anthropic is blocked.
Key facts
- anthropic-ai is operated by Anthropic, and its purpose is training: collects content that may be used to train future models.
- Anthropic publishes no full user-agent string for anthropic-ai, so Kymo matches the token anthropic-ai observed in real logs.
- The robots.txt token for anthropic-ai is anthropic-ai, and Anthropic publishes no compliance statement for anthropic-ai.
- anthropic-ai never sends a visitor to your site, and anthropic-ai attaches no citation when it uses your content.
Specification
| Operator | Anthropic |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Anthropic. Kymo matches this agent on the token anthropic-ai, observed in real logs rather than published in operator documentation. |
| robots.txt token | anthropic-ai |
| robots.txt compliance | No operator statement located |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Deprecated |
| Legacy Kymo category | training |
| Source | Third-party source, not yet confirmed against operator docshttps://support.claude.com/en/articles/8896518 |
| Verified on |
What this looks like in your logs
grep -i 'anthropic-ai' access.log
Anthropic publishes no full user-agent string for anthropic-ai, so the rest of the line varies and only the token anthropic-ai is dependable. Kymo matches on that token, case-insensitively, for the same reason.
What anthropic-ai does not do
anthropic-ai does not send you a visitor, and it attaches no citation, so a read by it can never turn into a click.
anthropic-ai does not decide whether Anthropic can find you today, because a training run happens long before any question is asked.
Anthropic publishes no crawl frequency for anthropic-ai, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files anthropic-ai under Anthropic as training, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from anthropic-ai appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. A training run sends nobody, so a read by it has no click to compare against.
Allow it or block it
A rule for anthropic-ai has no current effect. Many published robots.txt templates still include it, which gives site owners false confidence that Anthropic training is blocked when it is not.
| Allow it if | Block it if |
|---|---|
| Nothing. anthropic-ai is retired and makes no requests. | Nothing. A robots.txt rule naming anthropic-ai has no effect. Address the token that replaced it instead. |
robots.txt directives
Both blocks below address anthropic-ai only. Rules for one token never apply to another, even from the same operator.
Allow anthropic-ai
User-agent: anthropic-ai Allow: /
Block anthropic-ai
User-agent: anthropic-ai Disallow: /
Questions
How do I know a request claiming to be anthropic-ai is genuine?
Anthropic publishes no verification method for anthropic-ai, so the user-agent string is the only signal available, and anyone can send one. Treat a claimed anthropic-ai request as unproven.
Does blocking anthropic-ai also block ClaudeBot?
No, and blocking anthropic-ai does nothing at all. anthropic-ai is retired and makes no requests, so a robots.txt rule naming it has no effect on anything. ClaudeBot is the active token, and it has to be named in a rule of its own.
Will I see anthropic-ai in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and anthropic-ai reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does anthropic-ai send traffic back to my site?
No. anthropic-ai is retired and makes no requests.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
anthropic-ai takes your content and sends nothing back, so it leaves no trace at all in a browser-based analytics tool.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.