Meta-ExternalFetcher: Meta's Answer Crawler
What is Meta-ExternalFetcher?
Meta-ExternalFetcher fetches a page when a Meta AI product needs it for a request a person made. It is a live fetcher, not a bulk crawler. Meta-ExternalFetcher fetches on a person's request, and Meta documents that it may bypass robots.txt for exactly that reason.
Key facts
- Meta-ExternalFetcher is operated by Meta, and its purpose is live fetch: fetches a page in real time because a person asked.
- The user agent for Meta-ExternalFetcher is documented by Meta and contains the token meta-externalfetcher.
- The robots.txt token for Meta-ExternalFetcher is meta-externalfetcher, and Meta states that rule may not apply, because a person starts the request.
- Meta-ExternalFetcher can send a visitor to your site, and Meta-ExternalFetcher attaches a linked citation when it uses your content.
Specification
| Operator | Meta |
|---|---|
| Purpose | Live fetchFetches a page in real time because a person asked. Traffic here tracks human interest, not a crawl schedule. |
| Agent kind | FetcherFetches on a person's request. Appears in your logs. Request volume follows human activity. |
| User agent | meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers) |
| robots.txt token | meta-externalfetcher |
| robots.txt compliance | Operator states robots.txt may not apply (user initiated) |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | ai_answer |
| Source | Operator documentationhttps://developers.facebook.com/docs/sharing/webmasters/web-crawlers |
| Verified on |
What this looks like in your logs
203.0.113.42 - - [05/Aug/2026:09:14:07 +0000] "GET /pricing HTTP/1.1" 200 18432 "-" "meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)"
The part that identifies the request is the token meta-externalfetcher, inside the quoted user-agent field at the end of the line. Everything before it is the ordinary shape of a Common Log Format entry, and the address shown is a documentation address rather than one Meta uses.
What Meta-ExternalFetcher does not do
Meta-ExternalFetcher does not guarantee a visit, because a read only becomes a visit if a person clicks the citation that came out of it.
Meta-ExternalFetcher does not crawl on a schedule, because it only requests a page when a person has already asked something that needs it.
Meta publishes no crawl frequency for Meta-ExternalFetcher, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files Meta-ExternalFetcher under Meta as ai_answer, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from Meta-ExternalFetcher counts on the AI Visibility page, which is the only page that counts crawls at all. An answer crawl is the only kind a person can click through from, so it is the only kind that belongs beside a visitor number.
Seen on our own sites
Not seen on kymo.in or remotestack.in in the 90 days up to 15 Sep 2026.
Allow it or block it
Meta documents that this crawler may bypass robots.txt because it performs fetches that were requested by the user, so a disallow is not guaranteed to stop it and an edge rule may be needed. Blocking it prevents Meta AI from reading your page during a live user request. This agent is absent from most published crawler directories.
| Allow it if | Block it if |
|---|---|
| You want Meta to read your live page when a person asks about you, which is the crawl most likely to send a real visitor. | You do not want your page read during a live question. Check the compliance row first: a fetcher started by a person may need an edge rule rather than a robots.txt rule. |
robots.txt directives
Both blocks below address Meta-ExternalFetcher only. Rules for one token never apply to another, even from the same operator.
Allow Meta-ExternalFetcher
User-agent: meta-externalfetcher Allow: /
Block Meta-ExternalFetcher
User-agent: meta-externalfetcher Disallow: /
Questions
How do I know a request claiming to be Meta-ExternalFetcher is genuine?
Meta publishes no verification method for Meta-ExternalFetcher, so the user-agent string is the only signal available, and anyone can send one. Treat a claimed Meta-ExternalFetcher request as unproven.
Does blocking Meta-ExternalFetcher also block Meta-WebIndexer?
No. Meta-ExternalFetcher and Meta-WebIndexer are separate tokens, and Meta treats each one on its own. A rule naming Meta-ExternalFetcher leaves Meta-WebIndexer free to carry on indexing your pages for Meta answers. To stop that as well, name Meta-WebIndexer in a rule of its own.
Will I see Meta-ExternalFetcher in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and Meta-ExternalFetcher reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does Meta-ExternalFetcher send traffic back to my site?
Yes. Meta-ExternalFetcher builds an index that Meta cites its sources from, and a citation there is a link a person can click. A crawl is still not a visit, so expect the read count to run far ahead of the click count.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Meta-ExternalFetcher arrives because a person just asked Meta something, and a browser tag never runs in time to record the moment.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.