Claude-User: user agent, purpose, and whether to allow it
Claude-User fetches a page when a person asks Claude a question that requires reading it.
Specification
| Operator | Anthropic |
|---|---|
| Purpose | Live fetchFetches a page in real time because a person asked. Traffic here tracks human interest, not a crawl schedule. |
| Agent kind | FetcherFetches on a person's request. Appears in your logs. Request volume follows human activity. |
| User agent | Not documented by Anthropic. Kymo matches this agent on the token Claude-User, observed in real logs rather than published in operator documentation. |
| robots.txt token | Claude-User |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | Reverse DNS, forward confirmed |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | ai_answer |
| Source | Operator documentationhttps://support.claude.com/en/articles/8896518 |
| Verified on |
What blocking it costs you
Blocking Claude-User prevents Anthropic from retrieving your content in response to a user query, which Anthropic states may reduce visibility for user-directed web search. Unlike OpenAI, Anthropic does not document a user-initiated exemption from robots.txt for this agent.
robots.txt directives
Both blocks below address Claude-User only. Rules for one token never apply to another, even from the same operator.
Allow Claude-User
User-agent: Claude-User Allow: /
Block Claude-User
User-agent: Claude-User Disallow: /
Questions
What is the Claude-User user agent?
Anthropic documents the robots.txt token Claude-User but does not publish a full user-agent string for it. Kymo records this agent from observation rather than from documentation.
Does Claude-User respect robots.txt?
Documented: respects robots.txt
What does blocking Claude-User cost me?
Blocking Claude-User prevents Anthropic from retrieving your content in response to a user query, which Anthropic states may reduce visibility for user-directed web search. Unlike OpenAI, Anthropic does not document a user-initiated exemption from robots.txt for this agent.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so Claude-User shows up under Anthropic with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.