Bytespider: ByteDance's Training Crawler
What is Bytespider?
Bytespider collects web content for ByteDance model training. Bytespider has a reported record of ignoring robots.txt, so a robots.txt rule alone usually fails and an edge block is needed to stop it.
Key facts
- Bytespider is operated by ByteDance, and its purpose is training: collects content that may be used to train future models.
- ByteDance publishes no full user-agent string for Bytespider, so Kymo matches the token Bytespider observed in real logs.
- The robots.txt token for Bytespider is Bytespider, and ByteDance states that Bytespider respects it, which third parties have disputed.
- Bytespider never sends a visitor to your site, and Bytespider attaches no citation when it uses your content.
Specification
| Operator | ByteDance |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by ByteDance. Kymo matches this agent on the token Bytespider, observed in real logs rather than published in operator documentation. |
| robots.txt token | Bytespider |
| robots.txt compliance | Operator claims compliance, disputed by third parties |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | training |
| Source | Third-party source, not yet confirmed against operator docsByteDance publishes no crawler documentation that Kymo could locate. Everything on this page comes from observed traffic, not from a vendor page. |
| Verified on |
What this looks like in your logs
grep -i 'Bytespider' access.log
ByteDance publishes no full user-agent string for Bytespider, so the rest of the line varies and only the token Bytespider is dependable. Kymo matches on that token, case-insensitively, for the same reason.
What Bytespider does not do
Bytespider does not send you a visitor, and it attaches no citation, so a read by it can never turn into a click.
Bytespider does not decide whether ByteDance can find you today, because a training run happens long before any question is asked.
ByteDance publishes no documentation for Bytespider at all, so no crawl frequency and no compliance statement exists to quote.
How Kymo classifies it
Kymo files Bytespider under ByteDance as training, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from Bytespider appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. A training run sends nobody, so a read by it has no click to compare against.
Seen on our own sites
| Last seen | 15 Sep 2026 |
|---|---|
| Days active, last 30 days | 31 |
| Requests, last 30 days | 1,210 |
Counted by Kymo on kymo.in and remotestack.in, the two sites we run, up to 15 Sep 2026. Two sites is a small sample. Treat these numbers as proof the bot is active now.
Allow it or block it
Bytespider has a widely reported record of inconsistent robots.txt compliance and high request volume. If a block is required, enforce it at the edge rather than relying on robots.txt alone.
| Allow it if | Block it if |
|---|---|
| You want your content represented in future ByteDance models, and you accept that a training run sends no visitor and no link back. | You sell access to your content, or you object to it being used as training data without payment or credit. |
robots.txt directives
Both blocks below address Bytespider only. Rules for one token never apply to another, even from the same operator.
Allow Bytespider
User-agent: Bytespider Allow: /
Block Bytespider
User-agent: Bytespider Disallow: /
Questions
How do I know a request claiming to be Bytespider is genuine?
ByteDance publishes no verification method for Bytespider, so the user-agent string is the only signal available, and anyone can send one. Treat a claimed Bytespider request as unproven.
Does blocking Bytespider affect my Google ranking?
No. A Google ranking is decided by Googlebot, which is a separate crawler. Bytespider is ByteDance's training crawler, so blocking it changes what ByteDance can do with your content and changes nothing in Google search results.
Will I see Bytespider in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and Bytespider reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does Bytespider send traffic back to my site?
No. Bytespider takes content and sends nothing back. A hit appears in your logs as a request with no visitor behind it.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Bytespider takes your content and sends nothing back, so it leaves no trace at all in a browser-based analytics tool.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.