GPTBot: what it is and how to configure it in your robots.txt

You open your robots.txt to check which crawlers you have blocked and you find a line that reads User-agent: GPTBot. You cannot remember whether you put it there, whether it came with a template or whether someone on the team added it months ago without explaining why. And the question that follows is always the same: does this take you out of ChatGPT, or does it only block the training of its models?

GPTBot is the crawler OpenAI uses to collect content for training its generative AI models, and it is only one of four different user agents the company runs, each with its own job and its own independent control in robots.txt. Mixing them up is the most common mistake we see when reviewing crawl settings.

In this article you will find what GPTBot does and does not do, how it differs from OAI-SearchBot, ChatGPT-User and OAI-AdsBot, how to write the right rules in your robots.txt depending on what you want to allow or block, and why your robots.txt sometimes says yes while the crawler still cannot get in.

What GPTBot is and what OpenAI uses it for

GPTBot is the crawler OpenAI uses to collect publicly accessible content that may be used to train its generative AI models. When you disallow GPTBot in your robots.txt, you are telling OpenAI that your site’s content must not be used to train future versions of its models.

It is worth understanding what that decision really covers: blocking GPTBot affects the training of future models, and there is no public mechanism for pulling content out of models already trained on earlier crawls. It works forwards, not backwards, and that is worth being clear about before treating it as an “opt out of AI” switch.

The user-agent string and why it can change

GPTBot identifies itself with a user-agent string that includes a version number, for example GPTBot/1.4. OpenAI explicitly warns that this number can change without notice, so any firewall or detection rule that depends on one hand-written version is fragile by design.

What is reliable is filtering by the agent name and, if you need extra verification, by the published IP range (we cover that below).

GPTBot is not OpenAI’s only crawler

OpenAI officially documents four different user agents, and each one is controlled independently in robots.txt. If GPTBot is the only one you know about, it is easy to assume that blocking it settles your whole relationship with ChatGPT, and it does not.

User-agent	What OpenAI uses it for	Trains models?	Follows robots.txt?
GPTBot	Collects content for training future models	Yes	Yes
OAI-SearchBot	Indexes content for the search features inside ChatGPT	No	Yes
ChatGPT-User	Visits a page when a user or a Custom GPT asks for it during a conversation	No	Not necessarily, as it is a user-initiated action
OAI-AdsBot	Checks policy compliance on pages submitted as ads in ChatGPT	No	Applies only to submitted ad pages, not to the rest of the site

Why blocking GPTBot alone does not take you out of ChatGPT

What decides whether your content can be cited in ChatGPT’s search answers is OAI-SearchBot, not GPTBot. If you disallow OAI-SearchBot, your site will not appear in ChatGPT’s search answers, although it may still show up as a navigation link in other contexts.

These are two separate decisions: one is about training, the other about visibility, and OpenAI treats them as independent settings precisely so that you can keep them apart.

An editorial note: in the setups we tend to review it is common to find a robots.txt blocking GPTBot in the belief that this settles “the AI problem”, while OAI-SearchBot and ChatGPT-User are left wide open. The practical result is the exact opposite of what the team thought it had configured.

OAI-AdsBot, the least known of the four

OAI-AdsBot only visits pages an advertiser has explicitly submitted as the destination of an ad in ChatGPT, to check that they comply with OpenAI’s policy and to work out when to show that ad. It does not crawl the rest of your site, and the data it collects is not used to train the generative models.

If you have no active ads in ChatGPT, it is the user agent you least need to think about when configuring your robots.txt.

How to configure your robots.txt for GPTBot

The rules are written just as they are for any other crawler, aimed specifically at the user agent you want to control. These are the three most common patterns.

A full training block, leaving every other crawler unchanged:

User-agent: GPTBot
Disallow: /

Full permission, if you want your content to be usable for training future models:

User-agent: GPTBot
Allow: /

A partial block by directory, a common pattern on sites that want to protect premium or paid content while keeping the blog or the marketing pages visible:

User-agent: GPTBot
Disallow: /premium/
Disallow: /account/
Allow: /

Verification by published IP range

The user agent a request declares is a field anyone can fake, so if you need to confirm that a visit really is GPTBot and not a scraper posing as it, check the source IP against the range OpenAI publishes at openai.com/gptbot.json.

OpenAI publishes an equivalent file for each of its four user agents, and it is the most reliable way to audit your logs when a crawler’s traffic volume looks suspicious.

How long the change takes to apply

OpenAI says robots.txt changes can take roughly 24 hours to be reflected in its systems, although it publishes that figure specifically for OAI-SearchBot and its search results.

There is no equivalent published timeframe for GPTBot, so if you have just changed the directive, the most reliable move is to confirm the effect by reviewing your own server logs over the following days rather than assuming a fixed window.

When robots.txt says yes but the firewall says no

Allowing GPTBot in robots.txt does not guarantee that the crawler reaches your content. GPTBot cannot solve CAPTCHA-style verification challenges, so the default rules of firewalls such as Cloudflare, Akamai or AWS WAF can hand it a 403 or a 429 even when your robots.txt explicitly allows it.

This is exactly the kind of mismatch we cover in more detail in when robots.txt and the firewall disagree, and it is worth reviewing alongside your GPTBot configuration rather than separately.

How to check this on your own site

Whether your robots.txt, your firewall and what your server actually returns all agree is exactly the kind of mismatch few marketing teams get to cross-check by hand, because it means reading WAF logs as well as the text file.

The Parsigo diagnostic checks this as part of the access pillar, one of the three the analysis rests on (access 45%, readability 35%, structure 20%). The structure pillar, for example, reviews things like the structured data that helps crawlers interpret your content.

You can run the free diagnostic on your own domain to see whether GPTBot, OAI-SearchBot and ChatGPT-User are getting the answer you think you are giving them. It is not a promise that you will be cited in ChatGPT, it is a technical check of your site’s real configuration.

If you want to understand the full methodology first, you can read how it works.

Frequently asked questions about GPTBot

Does blocking GPTBot affect my Google rankings?

No. GPTBot is an OpenAI crawler and has nothing to do with Google’s algorithm or with Googlebot. Blocking it only affects whether your content can be used to train future OpenAI models.

Are GPTBot and ChatGPT-User the same thing?

No. GPTBot collects content to train models automatically. ChatGPT-User visits one specific page because a user or a Custom GPT asked for it during a conversation, and OpenAI notes that, as a user-initiated action, robots.txt rules may not apply to it in the same way.

What happens if I do not mention GPTBot in my robots.txt?

If there is no rule specific to GPTBot, the general rules in your file apply (those for user agent *), and if those do not exist either, the crawler takes it that it has permission to access your content for training purposes.

How do I verify that a GPTBot request is genuine and not spoofed?

Check the request’s source IP against the range OpenAI publishes at openai.com/gptbot.json. The user agent a request declares is easy to fake; the IP is the reliable validation.

Does blocking GPTBot take me out of ChatGPT’s answers?

No. What decides whether you appear in ChatGPT’s search answers is OAI-SearchBot, a different user agent controlled independently. GPTBot only governs the training permission.

What to check in your robots.txt before publishing

Before you call your GPTBot configuration finished, check three things: that the decision about GPTBot is deliberate and not a leftover from a generic template, that you have also reviewed OAI-SearchBot and ChatGPT-User separately, because the same rule does not control them, and that there is no silent contradiction between what your robots.txt allows and what your firewall actually lets through.