Diffbot: user agent, purpose, and whether to allow it
Diffbot is a commercial structured-data extraction service used for knowledge graph construction and some training pipelines.
Specification
| Operator | Diffbot |
|---|---|
| Purpose | Commercial toolA commercial crawler operated by an SEO or data-extraction vendor. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Diffbot. Kymo matches this agent on the token Diffbot, observed in real logs rather than published in operator documentation. |
| robots.txt token | Diffbot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | other |
| Source | Third-party source, not yet confirmed against operator docshttps://docs.diffbot.com/docs/en/guide-diffbot-crawlbot |
| Verified on |
What blocking it costs you
Verify against Diffbot documentation before publishing this section.
robots.txt directives
Both blocks below address Diffbot only. Rules for one token never apply to another, even from the same operator.
Allow Diffbot
User-agent: Diffbot Allow: /
Block Diffbot
User-agent: Diffbot Disallow: /
Questions
What is the Diffbot user agent?
Diffbot documents the robots.txt token Diffbot but does not publish a full user-agent string for it. Kymo records this agent from observation rather than from documentation.
Does Diffbot respect robots.txt?
Documented: respects robots.txt
What does blocking Diffbot cost me?
Verify against Diffbot documentation before publishing this section.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so Diffbot shows up under Diffbot with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.