Slurp: user agent, purpose, and whether to allow it
Slurp is the Yahoo search crawler.
Specification
| Operator | Yahoo |
|---|---|
| Purpose | Classic searchConventional search indexing. On some engines the same index also feeds AI-generated answer surfaces. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Yahoo. Kymo matches this agent on the token Slurp, observed in real logs rather than published in operator documentation. |
| robots.txt token | Slurp |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | indexing |
| Source | Observed in real logs, no operator documentation locatedYahoo publishes no crawler documentation that Kymo could locate. Everything on this page comes from observed traffic, not from a vendor page. |
| Verified on |
What blocking it costs you
Blocking Slurp removes you from Yahoo search results where Yahoo uses its own index rather than a syndicated one.
robots.txt directives
Both blocks below address Slurp only. Rules for one token never apply to another, even from the same operator.
Allow Slurp
User-agent: Slurp Allow: /
Block Slurp
User-agent: Slurp Disallow: /
Questions
What is the Slurp user agent?
Yahoo documents the robots.txt token Slurp but does not publish a full user-agent string for it. Kymo records this agent from observation rather than from documentation.
Does Slurp respect robots.txt?
Documented: respects robots.txt
What does blocking Slurp cost me?
Blocking Slurp removes you from Yahoo search results where Yahoo uses its own index rather than a syndicated one.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so Slurp shows up under Yahoo with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.