Claude-Web: Anthropic's Retired Crawler
Claude-Web is no longer in use. A robots.txt rule naming Claude-Web has no effect today. It was replaced by ClaudeBot, which is the token to address instead.
What is Claude-Web?
Deprecated Anthropic user agent, replaced by ClaudeBot. Claude-Web was replaced by ClaudeBot, so a robots.txt rule naming Claude-Web blocks nothing at Anthropic today.
Key facts
- Claude-Web is operated by Anthropic, and its purpose is training: collects content that may be used to train future models.
- Anthropic publishes no full user-agent string for Claude-Web, so Kymo matches the token Claude-Web observed in real logs.
- The robots.txt token for Claude-Web is Claude-Web, and Anthropic publishes no compliance statement for Claude-Web.
- Claude-Web never sends a visitor to your site, and Claude-Web attaches no citation when it uses your content.
Specification
| Operator | Anthropic |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Anthropic. Kymo matches this agent on the token Claude-Web, observed in real logs rather than published in operator documentation. |
| robots.txt token | Claude-Web |
| robots.txt compliance | No operator statement located |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Deprecated |
| Legacy Kymo category | training |
| Source | Third-party source, not yet confirmed against operator docshttps://support.claude.com/en/articles/8896518 |
| Verified on |
What this looks like in your logs
grep -i 'Claude-Web' access.log
Anthropic publishes no full user-agent string for Claude-Web, so the rest of the line varies and only the token Claude-Web is dependable. Kymo matches on that token, case-insensitively, for the same reason.
What Claude-Web does not do
Claude-Web does not send you a visitor, and it attaches no citation, so a read by it can never turn into a click.
Claude-Web does not decide whether Anthropic can find you today, because a training run happens long before any question is asked.
Anthropic publishes no crawl frequency for Claude-Web, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files Claude-Web under Anthropic as training, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from Claude-Web appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. A training run sends nobody, so a read by it has no click to compare against.
Allow it or block it
A rule for Claude-Web has no current effect. If your robots.txt still references it, the rule is dead and ClaudeBot is likely unaddressed. This is a common stale-config finding.
| Allow it if | Block it if |
|---|---|
| Nothing. Claude-Web is retired and makes no requests. | Nothing. A robots.txt rule naming Claude-Web has no effect. Address the token that replaced it instead. |
robots.txt directives
Both blocks below address Claude-Web only. Rules for one token never apply to another, even from the same operator.
Allow Claude-Web
User-agent: Claude-Web Allow: /
Block Claude-Web
User-agent: Claude-Web Disallow: /
Questions
How do I know a request claiming to be Claude-Web is genuine?
Anthropic publishes no verification method for Claude-Web, so the user-agent string is the only signal available, and anyone can send one. Treat a claimed Claude-Web request as unproven.
Does blocking Claude-Web also block ClaudeBot?
No, and blocking Claude-Web does nothing at all. Claude-Web is retired and makes no requests, so a robots.txt rule naming it has no effect on anything. ClaudeBot is the active token, and it has to be named in a rule of its own.
Will I see Claude-Web in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and Claude-Web reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does Claude-Web send traffic back to my site?
No. Claude-Web is retired and makes no requests.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Claude-Web takes your content and sends nothing back, so it leaves no trace at all in a browser-based analytics tool.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.