Slurp: Yahoo's Search Crawler
What is Slurp?
Slurp is the Yahoo search crawler. Slurp matters only where Yahoo serves its own index rather than a syndicated one, so the cost of blocking it changes by region.
Key facts
- Slurp is operated by Yahoo, and its purpose is classic search: conventional search indexing.
- Yahoo publishes no full user-agent string for Slurp, so Kymo matches the token Slurp observed in real logs.
- The robots.txt token for Slurp is Slurp, and Yahoo documents that Slurp respects that rule.
- Slurp can send a visitor to your site, and Slurp attaches a linked citation when it uses your content.
Specification
| Operator | Yahoo |
|---|---|
| Purpose | Classic searchConventional search indexing. On some engines the same index also feeds AI-generated answer surfaces. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Not documented by Yahoo. Kymo matches this agent on the token Slurp, observed in real logs rather than published in operator documentation. |
| robots.txt token | Slurp |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | indexing |
| Source | Observed in real logs, no operator documentation locatedYahoo publishes no crawler documentation that Kymo could locate. Everything on this page comes from observed traffic, not from a vendor page. |
| Verified on |
What this looks like in your logs
grep -i 'Slurp' access.log
Yahoo publishes no full user-agent string for Slurp, so the rest of the line varies and only the token Slurp is dependable. Kymo matches on that token, case-insensitively, for the same reason.
What Slurp does not do
Slurp does not guarantee a visit, because a read only becomes a visit if a person clicks the citation that came out of it.
Slurp does not train a model on what it reads, and it has no part in what any other search engine shows.
Yahoo publishes no documentation for Slurp at all, so no crawl frequency and no compliance statement exists to quote.
How Kymo classifies it
Kymo files Slurp under Yahoo as indexing, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from Slurp appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. An index sweep produces no click to miss, so counting it there would inflate the read number and break its relationship to the visitor number beside it.
Seen on our own sites
Not seen on kymo.in or remotestack.in in the 90 days up to 15 Sep 2026.
Allow it or block it
Blocking Slurp removes you from Yahoo search results where Yahoo uses its own index rather than a syndicated one.
| Allow it if | Block it if |
|---|---|
| You want to appear in Yahoo search results. | Yahoo sends you no traffic worth the crawl, which is usually a question of language and region. |
robots.txt directives
Both blocks below address Slurp only. Rules for one token never apply to another, even from the same operator.
Allow Slurp
User-agent: Slurp Allow: /
Block Slurp
User-agent: Slurp Disallow: /
Questions
How do I know a request claiming to be Slurp is genuine?
Yahoo publishes no verification method for Slurp, so the user-agent string is the only signal available, and anyone can send one. Treat a claimed Slurp request as unproven.
Does blocking Slurp affect my Google ranking?
No. A Google ranking is decided by Googlebot, which is a separate crawler. Each search engine runs its own crawler and its own index, so blocking Slurp removes you from Yahoo results and leaves Google untouched.
Will I see Slurp in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and Slurp reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does Slurp send traffic back to my site?
Yes. Slurp builds an index that Yahoo cites its sources from, and a citation there is a link a person can click. A crawl is still not a visit, so expect the read count to run far ahead of the click count.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Slurp decides whether you appear in Yahoo results, and a browser tag cannot tell you whether it ever came.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.