meta-externalagent: Meta's Training Crawler
What is meta-externalagent?
meta-externalagent collects web content for Meta AI training and product features. meta-externalagent is one of five separate Meta tokens, so a robots.txt rule naming it leaves Meta's preview, search, ads and live-fetch agents untouched.
Key facts
- meta-externalagent is operated by Meta, and its purpose is training: collects content that may be used to train future models.
- The user agent for meta-externalagent is documented by Meta and contains the token meta-externalagent.
- The robots.txt token for meta-externalagent is meta-externalagent, and Meta documents that meta-externalagent respects that rule.
- meta-externalagent never sends a visitor to your site, and meta-externalagent attaches no citation when it uses your content.
Specification
| Operator | Meta |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers) |
| robots.txt token | meta-externalagent |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | training |
| Source | Operator documentationhttps://developers.facebook.com/docs/sharing/webmasters/web-crawlers |
| Verified on |
What this looks like in your logs
203.0.113.42 - - [04/Aug/2026:09:14:07 +0000] "GET /pricing HTTP/1.1" 200 18432 "-" "meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)"
The part that identifies the request is the token meta-externalagent, inside the quoted user-agent field at the end of the line. Everything before it is the ordinary shape of a Common Log Format entry, and the address shown is a documentation address rather than one Meta uses.
What meta-externalagent does not do
meta-externalagent does not send you a visitor, and it attaches no citation, so a read by it can never turn into a click.
meta-externalagent does not decide whether Meta can find you today, because a training run happens long before any question is asked.
Meta publishes no crawl frequency for meta-externalagent, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files meta-externalagent under Meta as training, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from meta-externalagent appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. A training run sends nobody, so a read by it has no click to compare against.
Allow it or block it
Blocking it removes your content from Meta AI training. Meta documents robots.txt as the block mechanism: add a disallow for meta-externalagent. Third-party reports of inconsistent compliance exist, so verify with your own logs after adding the rule.
| Allow it if | Block it if |
|---|---|
| You want your content represented in future Meta models, and you accept that a training run sends no visitor and no link back. | You sell access to your content, or you object to it being used as training data without payment or credit. |
robots.txt directives
Both blocks below address meta-externalagent only. Rules for one token never apply to another, even from the same operator.
Allow meta-externalagent
User-agent: meta-externalagent Allow: /
Block meta-externalagent
User-agent: meta-externalagent Disallow: /
Questions
How do I know a request claiming to be meta-externalagent is genuine?
Meta publishes no verification method for meta-externalagent, so the user-agent string is the only signal available, and anyone can send one. Treat a claimed meta-externalagent request as unproven.
Does blocking meta-externalagent also block Meta-WebIndexer?
No. meta-externalagent and Meta-WebIndexer are separate tokens, and Meta treats each one on its own. A rule naming meta-externalagent leaves Meta-WebIndexer free to carry on indexing your pages for Meta answers. To stop that as well, name Meta-WebIndexer in a rule of its own.
Will I see meta-externalagent in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and meta-externalagent reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does meta-externalagent send traffic back to my site?
No. meta-externalagent takes content and sends nothing back. A hit appears in your logs as a request with no visitor behind it.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
meta-externalagent takes your content and sends nothing back, so it leaves no trace at all in a browser-based analytics tool.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.