CCBot: Common Crawl's Training Crawler
What is CCBot?
CCBot builds the Common Crawl public dataset. Common Crawl is not a model vendor, but its corpus feeds many model builders. CCBot builds the public Common Crawl dataset that many model builders draw on, so blocking it reaches further than blocking any single AI company's crawler.
Key facts
- CCBot is operated by Common Crawl, and its purpose is dataset: collects content into a public dataset that many model builders draw from.
- The user agent for CCBot is documented by Common Crawl and contains the token CCBot.
- The robots.txt token for CCBot is CCBot, and Common Crawl documents that CCBot respects that rule.
- CCBot never sends a visitor to your site, and CCBot attaches no citation when it uses your content.
Specification
| Operator | Common Crawl |
|---|---|
| Purpose | DatasetCollects content into a public dataset that many model builders draw from. Blocking it has broad downstream effect and no direct citation cost. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | CCBot/2.0 (https://commoncrawl.org/faq/) |
| robots.txt token | CCBot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | Published IP range filehttps://index.commoncrawl.org/ccbot.json |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | training |
| Source | Operator documentationhttps://commoncrawl.org/ccbot |
| Verified on |
What this looks like in your logs
203.0.113.42 - - [04/Aug/2026:09:14:07 +0000] "GET /pricing HTTP/1.1" 200 18432 "-" "CCBot/2.0 (https://commoncrawl.org/faq/)"
The part that identifies the request is the token CCBot, inside the quoted user-agent field at the end of the line. Everything before it is the ordinary shape of a Common Log Format entry, and the address shown is a documentation address rather than one Common Crawl uses.
What CCBot does not do
CCBot does not send you a visitor, and it attaches no citation, so a read by it can never turn into a click.
CCBot does not decide whether any particular product cites you, because it only fills a dataset that other companies choose to use or ignore.
Common Crawl publishes no crawl frequency for CCBot, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files CCBot under Common Crawl as training, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from CCBot appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. A training run sends nobody, so a read by it has no click to compare against.
Seen on our own sites
| Last seen | 15 Sep 2026 |
|---|---|
| Days active, last 30 days | 13 |
| Requests, last 30 days | 1,754 |
Counted by Kymo on kymo.in and remotestack.in, the two sites we run, up to 15 Sep 2026. Two sites is a small sample. Treat these numbers as proof the bot is active now.
Allow it or block it
Blocking CCBot removes you from a dataset that many model builders draw on, so the downstream effect is broad and indirect. It has no effect on any live citation surface. Common Crawl warns that crawlers falsely identify themselves as CCBot, and publishes an IP range file plus reverse DNS under crawl.commoncrawl.org so a claimed CCBot request can be verified.
| Allow it if | Block it if |
|---|---|
| You accept your content entering a dataset that many companies draw on, not only Common Crawl. | You want to limit how far your content travels, because a dataset reaches further than any single crawler. |
robots.txt directives
Both blocks below address CCBot only. Rules for one token never apply to another, even from the same operator.
Allow CCBot
User-agent: CCBot Allow: /
Block CCBot
User-agent: CCBot Disallow: /
Questions
How do I know a request claiming to be CCBot is genuine?
Common Crawl publishes the IP ranges CCBot uses at https://index.commoncrawl.org/ccbot.json. Check the requesting address against that file. A user-agent string alone proves nothing, because anyone can send one.
Does blocking CCBot affect my Google ranking?
No. A Google ranking is decided by Googlebot, which is a separate crawler. CCBot collects content into a dataset that other companies license, and Google does not read that dataset to rank pages.
Will I see CCBot in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and CCBot reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does CCBot send traffic back to my site?
No. CCBot takes content and sends nothing back. A hit appears in your logs as a request with no visitor behind it.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
CCBot copies your content into a dataset other companies then use, and a browser-based analytics tool records none of it.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.