Should you block GPTBot? What blocking AI crawlers costs a small business
Half the robots.txt files we look at block at least one AI crawler, often by accident. Here is what each bot does, which ones you can block without losing anything, and which block makes you disappear from the answers.
Saurav Chamoli3 min read
In 2023 a lot of people added a few lines to robots.txt to keep OpenAI from training on their site. I understand the impulse. The problem is that the same lines, left in place, now also keep you out of the answers that ChatGPT gives when someone asks about your industry. Training and answering are different bots. Most people blocked both.
There is not one OpenAI crawler, there are three
GPTBot collects pages for training future models. Blocking it means your text does not go into the next model's training data. That is a legitimate choice and it costs you nothing you would notice.
OAI-SearchBot is the one that matters for visibility. It builds the index ChatGPT searches when a user asks a question with browsing on. Block it, and ChatGPT cannot find your page to cite it. Your competitor's page, if theirs is open, gets cited instead.
ChatGPT-User fetches a page when a user explicitly asks about it, pasting a URL or asking a follow-up. Blocking it means a user who asks "what does this company do" gets "I can't access that site".
OpenAI documents all three on their site. Anthropic has the same split: ClaudeBot for training, Claude-SearchBot for search, Claude-User for user requests. Perplexity has PerplexityBot and Perplexity-User. Google keeps it simpler and more confusing at the same time: there is no separate Gemini crawler, Gemini's search grounding uses the ordinary Google index, and Google-Extended is not a crawler at all but a token you add to robots.txt to say "don't use my pages for Gemini training". Blocking Googlebot to stop Gemini removes you from Google Search. People have done this.
What we actually see
We check robots.txt as part of every analysis. The pattern repeats: a WordPress security plugin or a CDN preset that added Disallow: / for a list of "AI bots" in 2023 or 2024, and nobody has looked since. The list usually includes the search bots, because at the time the distinction did not exist. The owner has no idea. They assume AI assistants ignore them because the site is small.
I am not going to give you a number for how common this is, because our sample is sites whose owners were curious enough to run a check, which is not everyone. It is common enough that it is the first thing I would look at if a site I cared about never shows up.
A reasonable robots.txt in 2026
For a company that wants to be found but has not decided how it feels about training:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: /
That blocks training, allows search. If you do not mind training either, delete the first three blocks. If you want to block everything, fine, but know that "everything" includes the answers.
