Googlebot
Google · Feeds a search index
Googlebot crawls for Google Search and Discover, as well as Google Images, Video and News, and Google says it always obeys robots.txt when crawling automatically. Googlebot-Image, Googlebot-Video and Googlebot-News also obey rules written for Googlebot.
Fact sheet
- Operator
- What it is for
- It is Google Search's crawler. AI Overviews and Gemini lean on the same index.
- Assistant it feeds
- Google: AI Overviews and Gemini
- Counts towards the Parsigo score
- Yes. It feeds the index of one of the assistants we measure.
- Official documentation
- developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Official IP ranges
- JSON list
If you block it
You drop out of Google Search and, with it, out of AI Overviews and Gemini, which lean on the same index.
Allow or block
A group of its own makes Googlebot ignore the “User-agent: *” rules. If you have Disallow lines there that should still apply to it, repeat them inside its group. To allow it, writing nothing is usually enough.
Allow
User-agent: Googlebot Allow: /
Block
User-agent: Googlebot Disallow: /
Test it on your site
We run the diagnostic on your site and show you whether Googlebot can get in, with the robots.txt line that decides it. Free, no sign-up.
Worth knowing
Google's AI Overviews and AI mode lean on the Googlebot index: block it and you are out.