Googlebot: user agent, purpose, and whether to allow it
Googlebot is the standard Google Search crawler. The index it builds also feeds AI Overviews and AI Mode.
Specification
| Operator | |
|---|---|
| Purpose | Classic searchConventional search indexing. On some engines the same index also feeds AI-generated answer surfaces. |
| Agent kind | CrawlerAutonomous crawler. Appears in your logs under its own user-agent. |
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36 |
| robots.txt token | Googlebot |
| robots.txt compliance | Documented: respects robots.txt |
| Verification | Reverse DNS, forward confirmed.googlebot.com |
| Sends referral traffic | Yes |
| Attaches a citation | Yes |
| Status | Active |
| Legacy Kymo category | indexing |
| Source | Operator documentationhttps://developers.google.com/search/docs/crawling-indexing/google-common-crawlers |
| Verified on |
What blocking it costs you
Blocking Googlebot removes you from Google Search entirely, and with it from AI Overviews. There is almost no situation in which this is the right call. To opt out of Gemini training while keeping search, use Google-Extended instead.
robots.txt directives
Both blocks below address Googlebot only. Rules for one token never apply to another, even from the same operator.
Allow Googlebot
User-agent: Googlebot Allow: /
Block Googlebot
User-agent: Googlebot Disallow: /
Questions
What is the Googlebot user agent?
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Does Googlebot respect robots.txt?
Documented: respects robots.txt
What does blocking Googlebot cost me?
Blocking Googlebot removes you from Google Search entirely, and with it from AI Overviews. There is almost no situation in which this is the right call. To opt out of Gemini training while keeping search, use Google-Extended instead.
Related crawlers
All AI crawlers and control tokens
See this bot in your own logs
Kymo classifies every crawler hit server-side using this registry, so Googlebot shows up under Google with its purpose attached. Read how crawler detection works, or run a free free AI visibility check to see what each crawler currently gets back from your server. No account needed for the check.