Googlebot: Google's Search Crawler
What is Googlebot?
Googlebot is the standard Google Search crawler. The index it builds also feeds AI Overviews and AI Mode. Googlebot builds the index that also feeds AI Overviews, so blocking it removes you from classic Google results and from those answers together.
Key facts
- Googlebot is operated by Google, and its purpose is classic search: conventional search indexing.
- The user agent for Googlebot is documented by Google and contains the token Googlebot.
- The robots.txt token for Googlebot is Googlebot, and Google documents that Googlebot respects that rule.
- Googlebot can send a visitor to your site, and Googlebot attaches a linked citation when it uses your content.
Specification
| Operator | |
|---|---|
| Purpose | Classic searchConventional search indexing. On some engines the same index also feeds AI-generated answer surfaces. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36 |
| robots.txt token | Googlebot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | Reverse DNS, forward confirmed.googlebot.com |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | indexing |
| Source | Operator documentationhttps://developers.google.com/search/docs/crawling-indexing/google-common-crawlers |
| Verified on |
What this looks like in your logs
203.0.113.42 - - [04/Aug/2026:09:14:07 +0000] "GET /pricing HTTP/1.1" 200 18432 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36"
The part that identifies the request is the token Googlebot, inside the quoted user-agent field at the end of the line. Everything before it is the ordinary shape of a Common Log Format entry, and the address shown is a documentation address rather than one Google uses.
What Googlebot does not do
Googlebot does not guarantee a visit, because a read only becomes a visit if a person clicks the citation that came out of it.
Googlebot does not train a model on what it reads, and it has no part in what any other search engine shows.
Google publishes no crawl frequency for Googlebot, so any schedule you read elsewhere is somebody's estimate rather than a documented figure.
How Kymo classifies it
Kymo files Googlebot under Google as indexing, classified from the HTTP request on the server rather than from anything running in a browser.
A hit from Googlebot appears on the Crawlers dashboard and is deliberately left out of the AI Visibility numbers. An index sweep produces no click to miss, so counting it there would inflate the read number and break its relationship to the visitor number beside it.
Seen on our own sites
| Last seen | 15 Sep 2026 |
|---|---|
| Days active, last 30 days | 31 |
| Requests, last 30 days | 172,029 |
Counted by Kymo on kymo.in and remotestack.in, the two sites we run, up to 15 Sep 2026. Two sites is a small sample. Treat these numbers as proof the bot is active now.
Allow it or block it
Blocking Googlebot removes you from Google Search entirely, and with it from AI Overviews. There is almost no situation in which this is the right call. To opt out of Gemini training while keeping search, use Google-Extended instead.
| Allow it if | Block it if |
|---|---|
| You want to appear in Google search results. | Google sends you no traffic worth the crawl, which is usually a question of language and region. |
robots.txt directives
Both blocks below address Googlebot only. Rules for one token never apply to another, even from the same operator.
Allow Googlebot
User-agent: Googlebot Allow: /
Block Googlebot
User-agent: Googlebot Disallow: /
Questions
How do I know a request claiming to be Googlebot is genuine?
Run a reverse DNS lookup on the requesting address, confirm it ends in .googlebot.com, then run a forward lookup on that name and check it returns the same address. A user-agent string alone proves nothing.
Does blocking Googlebot also block Google-Extended?
No. Googlebot and Google-Extended are separate tokens, and Google treats each one on its own. Google-Extended has no user agent and cannot be blocked, so it has to be set as its own robots.txt rule.
Will I see Googlebot in Google Analytics?
No. Google Analytics runs a JavaScript tag in a visitor's browser, and Googlebot reads your HTML and leaves without running any script. Server-side logging is the only way to record it.
Does Googlebot send traffic back to my site?
Yes. Googlebot builds an index that Google cites its sources from, and a citation there is a link a person can click. A crawl is still not a visit, so expect the read count to run far ahead of the click count.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Googlebot decides whether you appear in Google results, and a browser tag cannot tell you whether it ever came.
Kymo reads the HTTP request on your server, so a crawler that never runs JavaScript is still recorded. You see which bots reached the site, which pages they took and how often, next to your human traffic.
Start free → 14-day free trial. No card required.
No account yet? Run a free AI visibility check on your own site.