Meta-WebIndexer: user agent, purpose, and whether to allow it
Meta-WebIndexer builds a search index Meta products draw on. It is an index builder rather than a training crawler.
Specification
| Operator | Meta |
|---|---|
| Purpose | Search indexBuilds the index an AI product cites from. Blocking it removes you from that product's answers. This is the expensive block, and the one most often made by accident. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers) |
| robots.txt token | meta-webindexer |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | indexing |
| Source | Operator documentationhttps://developers.facebook.com/docs/sharing/webmasters/web-crawlers |
| Verified on |
What blocking it costs you
Blocking Meta-WebIndexer removes you from the index behind Meta search surfaces. It has no effect on Meta AI training, which is governed by meta-externalagent, or on link previews, which are facebookexternalhit. Each Meta token has to be addressed on its own.
robots.txt directives
Both blocks below address Meta-WebIndexer only. Rules for one token never apply to another, even from the same operator.
Allow Meta-WebIndexer
User-agent: meta-webindexer Allow: /
Block Meta-WebIndexer
User-agent: meta-webindexer Disallow: /
Questions
What is the Meta-WebIndexer user agent?
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
Does Meta-WebIndexer respect robots.txt?
Documented: respects robots.txt
What does blocking Meta-WebIndexer cost me?
Blocking Meta-WebIndexer removes you from the index behind Meta search surfaces. It has no effect on Meta AI training, which is governed by meta-externalagent, or on link previews, which are facebookexternalhit. Each Meta token has to be addressed on its own.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so Meta-WebIndexer shows up under Meta with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.