CCBot: user agent, purpose, and whether to allow it
CCBot builds the Common Crawl public dataset. Common Crawl is not a model vendor, but its corpus feeds many model builders.
Specification
| Operator | Common Crawl |
|---|---|
| Purpose | DatasetCollects content into a public dataset that many model builders draw from. Blocking it has broad downstream effect and no direct citation cost. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | CCBot/2.0 (https://commoncrawl.org/faq/) |
| robots.txt token | CCBot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | Published IP range filehttps://index.commoncrawl.org/ccbot.json |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | training |
| Source | Operator documentationhttps://commoncrawl.org/ccbot |
| Verified on |
What blocking it costs you
Blocking CCBot removes you from a dataset that many model builders draw on, so the downstream effect is broad and indirect. It has no effect on any live citation surface. Common Crawl warns that crawlers falsely identify themselves as CCBot, and publishes an IP range file plus reverse DNS under crawl.commoncrawl.org so a claimed CCBot request can be verified.
robots.txt directives
Both blocks below address CCBot only. Rules for one token never apply to another, even from the same operator.
Allow CCBot
User-agent: CCBot Allow: /
Block CCBot
User-agent: CCBot Disallow: /
Questions
What is the CCBot user agent?
CCBot/2.0 (https://commoncrawl.org/faq/)
Does CCBot respect robots.txt?
Documented: respects robots.txt
What does blocking CCBot cost me?
Blocking CCBot removes you from a dataset that many model builders draw on, so the downstream effect is broad and indirect. It has no effect on any live citation surface. Common Crawl warns that crawlers falsely identify themselves as CCBot, and publishes an IP range file plus reverse DNS under crawl.commoncrawl.org so a claimed CCBot request can be verified.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so CCBot shows up under Common Crawl with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.