Diffbot: Diffbot's Data Crawler
What is Diffbot?
Diffbot is a commercial structured-data extraction service used for knowledge graph construction and some training pipelines. Diffbot extracts structured data for knowledge graph products sold to customers, so blocking it changes no AI answer surface at all.
Key facts
- Diffbot is operated by Diffbot, and its purpose is commercial tool: a commercial crawler operated by an SEO or data-extraction vendor.
- Diffbot publishes no full user-agent string for Diffbot, so Kymo matches the token Diffbot observed in real logs.
- The robots.txt token for Diffbot is Diffbot, and Diffbot documents that Diffbot respects that rule.
- Diffbot never sends a visitor to your site, and Diffbot attaches no citation when it uses your content.
Specification
| Operator | Diffbot |
|---|---|
| Purpose | Commercial toolA commercial crawler operated by an SEO or data-extraction vendor. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Diffbot. Kymo matches this agent on the token Diffbot, observed in real logs rather than published in operator documentation. |
| robots.txt token | Diffbot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | other |
| Source | Third-party source, not yet confirmed against operator docshttps://docs.diffbot.com/docs/en/guide-diffbot-crawlbot |
| Verified on |
What this looks like in your logs
grep -i 'Diffbot' access.log
Diffbot publishes no full user-agent string for Diffbot, so the rest of the line varies and only the token Diffbot is dependable. Kymo matches on that token, case-insensitively, for the same reason.
What Diffbot does not do
Diffbot does not send you a visitor, and it attaches no citation, so a read by it can never turn into a click.
Diffbot does not affect any AI answer surface, because the data it takes goes into products sold to other companies.
Diffbot publishes no crawl frequency for Diffbot, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files Diffbot under Diffbot as other, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from Diffbot appears on the Crawlers dashboard under its own operator and is left out of the AI Visibility numbers, because it carries no answer a reader could arrive from.
Seen on our own sites
Not seen on kymo.in or remotestack.in in the 90 days up to 15 Sep 2026.
Allow it or block it
Verify against Diffbot documentation before publishing this section.
| Allow it if | Block it if |
|---|---|
| You want your pages in the commercial data products this crawler feeds. | You do not want a vendor selling structured data taken from your pages. |
robots.txt directives
Both blocks below address Diffbot only. Rules for one token never apply to another, even from the same operator.
Allow Diffbot
User-agent: Diffbot Allow: /
Block Diffbot
User-agent: Diffbot Disallow: /
Questions
How do I know a request claiming to be Diffbot is genuine?
Diffbot publishes no verification method for Diffbot, so the user-agent string is the only signal available, and anyone can send one. Treat a claimed Diffbot request as unproven.
Does blocking Diffbot affect my Google ranking?
No. A Google ranking is decided by Googlebot, which is a separate crawler. Diffbot extracts structured data for products Diffbot sells to other companies, and none of that reaches Google.
Will I see Diffbot in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and Diffbot reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does Diffbot send traffic back to my site?
No. Diffbot takes content and sends nothing back. A hit appears in your logs as a request with no visitor behind it.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Diffbot takes structured data off your pages for a product sold elsewhere, and a browser tag records none of it.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.