Meta-WebIndexer: Meta's Search Crawler
What is Meta-WebIndexer?
Meta-WebIndexer builds a search index Meta products draw on. It is an index builder rather than a training crawler. Meta-WebIndexer builds a search index only, so blocking it leaves Meta AI training and Meta link previews running under their own separate tokens.
Key facts
- Meta-WebIndexer is operated by Meta, and its purpose is search index: builds the index an AI product cites from.
- The user agent for Meta-WebIndexer is documented by Meta and contains the token meta-webindexer.
- The robots.txt token for Meta-WebIndexer is meta-webindexer, and Meta documents that Meta-WebIndexer respects that rule.
- Meta-WebIndexer can send a visitor to your site, and Meta-WebIndexer attaches a linked citation when it uses your content.
Specification
| Operator | Meta |
|---|---|
| Purpose | Search indexBuilds the index an AI product cites from. Blocking it removes you from that product's answers. This is the expensive block, and the one most often made by accident. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers) |
| robots.txt token | meta-webindexer |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | indexing |
| Source | Operator documentationhttps://developers.facebook.com/docs/sharing/webmasters/web-crawlers |
| Verified on |
What this looks like in your logs
203.0.113.42 - - [05/Aug/2026:09:14:07 +0000] "GET /pricing HTTP/1.1" 200 18432 "-" "meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)"
The part that identifies the request is the token meta-webindexer, inside the quoted user-agent field at the end of the line. Everything before it is the ordinary shape of a Common Log Format entry, and the address shown is a documentation address rather than one Meta uses.
What Meta-WebIndexer does not do
Meta-WebIndexer does not guarantee a visit, because a read only becomes a visit if a person clicks the citation that came out of it.
Meta-WebIndexer does not train a model on what it reads, so blocking it changes what Meta can answer and not what Meta learns.
Meta publishes no crawl frequency for Meta-WebIndexer, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files Meta-WebIndexer under Meta as indexing, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from Meta-WebIndexer appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. An index sweep produces no click to miss, so counting it there would inflate the read number and break its relationship to the visitor number beside it.
Seen on our own sites
| Last seen | 15 Sep 2026 |
|---|---|
| Days active, last 30 days | 18 |
| Requests, last 30 days | 59 |
Counted by Kymo on kymo.in and remotestack.in, the two sites we run, up to 15 Sep 2026. Two sites is a small sample. Treat these numbers as proof the bot is active now.
Allow it or block it
Blocking Meta-WebIndexer removes you from the index behind Meta search surfaces. It has no effect on Meta AI training, which is governed by meta-externalagent, or on link previews, which are facebookexternalhit. Each Meta token has to be addressed on its own.
| Allow it if | Block it if |
|---|---|
| You want to appear in Meta answers, because this crawler builds the index those answers cite. | You accept disappearing from Meta answers. This is the most expensive block on this page, and it is the one made most often by accident. |
robots.txt directives
Both blocks below address Meta-WebIndexer only. Rules for one token never apply to another, even from the same operator.
Allow Meta-WebIndexer
User-agent: meta-webindexer Allow: /
Block Meta-WebIndexer
User-agent: meta-webindexer Disallow: /
Questions
How do I know a request claiming to be Meta-WebIndexer is genuine?
Meta publishes no verification method for Meta-WebIndexer, so the user-agent string is the only signal available, and anyone can send one. Treat a claimed Meta-WebIndexer request as unproven.
Does blocking Meta-WebIndexer also block meta-externalagent?
No. Meta-WebIndexer and meta-externalagent are separate tokens, and Meta treats each one on its own. A rule naming Meta-WebIndexer leaves meta-externalagent free to carry on collecting your content for Meta model training. To stop that as well, name meta-externalagent in a rule of its own.
Will I see Meta-WebIndexer in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and Meta-WebIndexer reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does Meta-WebIndexer send traffic back to my site?
Yes. Meta-WebIndexer builds an index that Meta cites its sources from, and a citation there is a link a person can click. A crawl is still not a visit, so expect the read count to run far ahead of the click count.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Meta-WebIndexer decides whether you appear in Meta answers, and a browser tag cannot tell you whether it ever came.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.