Meta-ExternalFetcher: user agent, purpose, and whether to allow it
Meta-ExternalFetcher fetches a page when a Meta AI product needs it for a request a person made. It is a live fetcher, not a bulk crawler.
Specification
| Operator | Meta |
|---|---|
| Purpose | Live fetchFetches a page in real time because a person asked. Traffic here tracks human interest, not a crawl schedule. |
| Agent kind | FetcherFetches on a person's request. Appears in your logs. Request volume follows human activity. |
| User agent | meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers) |
| robots.txt token | meta-externalfetcher |
| robots.txt compliance | Operator states robots.txt may not apply (user initiated) |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | ai_answer |
| Source | Operator documentationhttps://developers.facebook.com/docs/sharing/webmasters/web-crawlers |
| Verified on |
What blocking it costs you
Meta documents that this crawler may bypass robots.txt because it performs fetches that were requested by the user, so a disallow is not guaranteed to stop it and an edge rule may be needed. Blocking it prevents Meta AI from reading your page during a live user request. This agent is absent from most published crawler directories.
robots.txt directives
Both blocks below address Meta-ExternalFetcher only. Rules for one token never apply to another, even from the same operator.
Allow Meta-ExternalFetcher
User-agent: meta-externalfetcher Allow: /
Block Meta-ExternalFetcher
User-agent: meta-externalfetcher Disallow: /
Questions
What is the Meta-ExternalFetcher user agent?
meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)
Does Meta-ExternalFetcher respect robots.txt?
Operator states robots.txt may not apply (user initiated)
What does blocking Meta-ExternalFetcher cost me?
Meta documents that this crawler may bypass robots.txt because it performs fetches that were requested by the user, so a disallow is not guaranteed to stop it and an edge rule may be needed. Blocking it prevents Meta AI from reading your page during a live user request. This agent is absent from most published crawler directories.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so Meta-ExternalFetcher shows up under Meta with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.