Bytespider: user agent, purpose, and whether to allow it
Bytespider collects web content for ByteDance model training.
Specification
| Operator | ByteDance |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by ByteDance. Kymo matches this agent on the token Bytespider, observed in real logs rather than published in operator documentation. |
| robots.txt token | Bytespider |
| robots.txt compliance | Operator claims compliance, disputed by third parties |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | training |
| Source | Third-party source, not yet confirmed against operator docsByteDance publishes no crawler documentation that Kymo could locate. Everything on this page comes from observed traffic, not from a vendor page. |
| Verified on |
What blocking it costs you
Bytespider has a widely reported record of inconsistent robots.txt compliance and high request volume. If a block is required, enforce it at the edge rather than relying on robots.txt alone.
robots.txt directives
Both blocks below address Bytespider only. Rules for one token never apply to another, even from the same operator.
Allow Bytespider
User-agent: Bytespider Allow: /
Block Bytespider
User-agent: Bytespider Disallow: /
Questions
What is the Bytespider user agent?
ByteDance documents the robots.txt token Bytespider but does not publish a full user-agent string for it. Kymo records this agent from observation rather than from documentation.
Does Bytespider respect robots.txt?
Operator claims compliance, disputed by third parties
What does blocking Bytespider cost me?
Bytespider has a widely reported record of inconsistent robots.txt compliance and high request volume. If a block is required, enforce it at the edge rather than relying on robots.txt alone.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so Bytespider shows up under ByteDance with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.