Claude-Web: user agent, purpose, and whether to allow it
Claude-Web is no longer in use. A robots.txt rule naming Claude-Web has no effect today. It was replaced by ClaudeBot, which is the token to address instead.
Deprecated Anthropic user agent, replaced by ClaudeBot.
Specification
| Operator | Anthropic |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Anthropic. Kymo matches this agent on the token Claude-Web, observed in real logs rather than published in operator documentation. |
| robots.txt token | Claude-Web |
| robots.txt compliance | No operator statement located |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Deprecated |
| Legacy Kymo category | training |
| Source | Third-party source, not yet confirmed against operator docshttps://support.claude.com/en/articles/8896518 |
| Verified on |
What blocking it costs you
A rule for Claude-Web has no current effect. If your robots.txt still references it, the rule is dead and ClaudeBot is likely unaddressed. This is a common stale-config finding.
robots.txt directives
Both blocks below address Claude-Web only. Rules for one token never apply to another, even from the same operator.
Allow Claude-Web
User-agent: Claude-Web Allow: /
Block Claude-Web
User-agent: Claude-Web Disallow: /
Questions
What is the Claude-Web user agent?
Anthropic documents the robots.txt token Claude-Web but does not publish a full user-agent string for it. Kymo records this agent from observation rather than from documentation.
Does Claude-Web respect robots.txt?
No operator statement located
What does blocking Claude-Web cost me?
A rule for Claude-Web has no current effect. If your robots.txt still references it, the rule is dead and ClaudeBot is likely unaddressed. This is a common stale-config finding.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so Claude-Web shows up under Anthropic with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.