Google-Extended: Google's Robots.txt Token
What is Google-Extended?
Google-Extended is a robots.txt control token, not a crawler. It has no user-agent and never appears in your logs. It controls whether content Google already crawled may be used to train and ground Gemini. Google-Extended has no user agent and never appears in a log, and it governs how already-crawled content is used rather than whether Google crawls at all.
Google-Extended has no user agent and will never appear in your logs. It is a robots.txt token only. It controls how content that a different, already permitted crawler collected may be used.
If a log line, analytics report or bot-detection tool claims a visit from Google-Extended, that record is wrong. Either the request is spoofed, or the tool is inventing a user agent that does not exist. Most published crawler directories get this wrong.
Key facts
- Google-Extended is operated by Google, and its purpose is training: collects content that may be used to train future models.
- Google-Extended has no user agent and never appears in a server log, because Google-Extended is a robots.txt token and not a crawler.
- The robots.txt token for Google-Extended is Google-Extended, and Google documents that Google-Extended respects that rule.
- Blocking is not what Google-Extended does: Google-Extended tells Google how content that another crawler already collected may be used.
Specification
| Operator | |
|---|---|
| Purpose | TrainingCollects content that may be used to train future models. Blocking it removes your content from future training runs. It does not affect whether a model can find and cite you today. |
| Agent kind | Control tokenA robots.txt token only. It has no user-agent and will never appear in your logs. It controls how content already collected by another crawler may be used. If this string shows up in your access log, treat it as spoofed. |
| User agent | None. This entry is a robots.txt token and has no user agent. |
| robots.txt token | Google-Extended |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | No published method, user agent is the only signal |
| Sends referral traffic | No |
| Attaches a citation | No |
| Status | Active |
| Legacy Kymo category | training |
| Source | Operator documentationhttps://developers.google.com/search/docs/crawling-indexing/google-common-crawlers |
| Verified on |
What Google-Extended does not do
Google-Extended does not request anything, so it cannot appear in a log, in an analytics report or in a firewall rule.
Google-Extended does not stop Google crawling your site, because a different token controls that.
Google publishes no crawl frequency for Google-Extended, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo would file Google-Extended as training, and records nothing under it, because Google-Extended sends no request for anything to classify.
A log line claiming this token is spoofed, and Kymo flags it rather than counting it. So the number you see for Google-Extended on any Kymo dashboard is zero, and that is the correct number.
Allow it or block it
Disallowing Google-Extended opts your content out of Gemini training and grounding. It has no effect on Google Search ranking or on Googlebot. If this string appears in your access log, it is spoofed traffic, because no real client sends it.
| Allow it if | Block it if |
|---|---|
| You accept Google using content it already collected for the purpose this token governs. | You want Google to stop using your content for that purpose. Crawling itself continues, because a different token controls it. |
robots.txt directives
Both blocks below address Google-Extended only. Rules for one token never apply to another, even from the same operator.
Allow Google-Extended
User-agent: Google-Extended Allow: /
Block Google-Extended
User-agent: Google-Extended Disallow: /
Questions
How do I know a request claiming to be Google-Extended is genuine?
You cannot, and you never need to. Google-Extended has no user agent and makes no requests, so a log line naming Google-Extended is spoofed or comes from a tool inventing an agent that does not exist.
Does blocking Google-Extended also block Googlebot?
No. Google-Extended and Googlebot are separate tokens, and Google treats each one on its own. A rule naming Google-Extended leaves Googlebot free to carry on indexing your pages for Google search results. To stop that as well, name Googlebot in a rule of its own.
Will I see Google-Extended in Google Analytics?
No, and not in a server log either. Google-Extended has no user agent and makes no requests, so there is nothing for any tool to record.
Does Google-Extended send traffic back to my site?
No. Google-Extended makes no requests at all, so nothing about it can send you a visitor. What it governs is how Google uses content another crawler already collected.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Google-Extended never appears in any log, so the question worth answering is which of Google's real crawlers came instead.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.