robots.txt for AI crawlers: how to block some, allow others, and check that it worked
The whole robots.txt standard fits on one page, and most AI visibility problems we see start there. How the matching rules actually work, which bot does what in 2026, copy-ready files for the three common cases, and how to tell whether a crawler obeyed you.
Saurav Chamoli··7 min read
robots.txt is a plain-text file at the root of your domain that tells crawlers, by name, which paths they may fetch. It is a request, not a lock: honest crawlers read it before every crawl and follow it, scrapers ignore it, and nothing in it is enforced by your server. That one sentence explains most of what follows, including why a file written in 2023 to keep AI companies out is now quietly keeping you out of their answers.
I wrote earlier about the GPTBot decision specifically. This is the longer reference: how the file is read, who the crawlers are, and the exact rules for the cases people actually have.
How the file is read
A crawler fetches https://yourdomain.ch/robots.txt once, then checks every URL it wants against the rules. The file is made of groups. A group starts with one or more User-agent: lines and is followed by Disallow: and Allow: lines that apply to those agents.
Five rules decide what happens, and two of them surprise people:
A crawler obeys exactly one group: the most specific one that names it. If there is a group for ExampleBot, that bot reads only that group and ignores the * group completely. This is the rule behind most accidents. Add a group for one AI bot to allow it, forget to repeat your /api/ disallow in it, and that bot now crawls /api/.
Paths are prefixes, and the longest matching rule wins.Disallow: /intern/ with Allow: /intern/public/ lets /intern/public/report.pdf through because the Allow is longer. When two rules are the same length, Allow wins in Google's implementation.
Paths are case-sensitive.Disallow: /Intern/ does nothing for /intern/.
* matches any run of characters and $ anchors the end.Disallow: /*.pdf$ blocks every PDF. Disallow: /*? blocks every URL with a query string. Without $, Disallow: /*.pdf also blocks /file.pdf?download=1 and /pdf-guide/.
An empty Disallow: allows everything, and a missing file allows everything. A that returns 404 means "crawl it all". A that returns a 5xx error makes Google treat the whole site as disallowed until it recovers, so a broken server rule here can empty your index.
robots.txt
robots.txt
Crawl-delay: exists in many files and Google ignores it; Bing honours it. Lines starting with # are comments. Encoding is UTF-8, and a byte-order mark at the start of the file has broken more than one User-agent line, because the first line then reads as \ufeffUser-agent.
The crawlers, by what they do
The operators now run separate bots for separate jobs, and the names matter more than the companies. Three jobs:
Training: collecting text to train the next model. Blocking this has no effect on whether you appear in answers.
Search index: building the index the assistant searches when a user asks something with browsing on. Blocking this removes you from the answers.
User fetch: retrieving a page because a user pasted the link or asked about it directly. These act on a person's behalf and the operators say they may not treat robots.txt as a crawl rule in the same way; check each operator's page.
Operator
Training
Search index
User fetch
OpenAI (ChatGPT)
GPTBot
OAI-SearchBot
ChatGPT-User
Anthropic (Claude)
ClaudeBot
Claude-SearchBot
Claude-User
Perplexity
PerplexityBot
PerplexityBot
Perplexity-User
Google (Gemini, AI Overviews)
Google-Extended (a token, not a bot)
Googlebot
Google-Extended also covers grounding
Microsoft (Copilot)
Bingbot
Bingbot
—
Apple (Siri, Apple Intelligence)
Applebot-Extended (token)
Applebot
—
Meta
Meta-ExternalAgent
—
Meta-ExternalFetcher
Amazon (Alexa)
Amazonbot
Amazonbot
—
Common Crawl (used by many)
CCBot
—
—
ByteDance
Bytespider
—
—
Two entries need a sentence each. Google has no Gemini crawler: Gemini's search grounding and AI Overviews use the ordinary Google Search index, so the only way to keep your pages out of them is to keep them out of Google Search. Google-Extended is a token you can name in robots.txt to say "do not use my pages for Gemini training or grounding"; it does not change Search ranking or AI Overviews. Perplexity uses one crawler for both indexing and training, so you cannot block one without the other.
These names change. Each operator publishes a page listing its current user agents, and most publish the IP ranges the bots come from. Check them once a year.
Case 1: block training, stay in the answers
The file most companies should have in 2026, if they have not decided how they feel about training but do want to be found:
# Training crawlers: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
Disallow: /
# Search and answer crawlers: yes, same limits as everyone
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /api/
Disallow: /account/
User-agent: *
Disallow: /api/
Disallow: /account/
Sitemap: https://yourdomain.ch/sitemap.xml
Note the second group repeats the disallows from the * group. Rule 1 above: a named bot never reads the * group. Perplexity sits in the "yes" list because blocking it blocks the index too.
Case 2: allow every AI crawler
If training does not bother you, the shortest correct file is the one with no AI-specific groups at all:
Everything not named falls under *. You do not need Allow: / lines; allowed is the default. Files that list twenty AI bots each with Allow: / are not wrong, just maintenance you do not need.
Disallow: /search without a trailing slash blocks /search, /search/ and /searchresults alike, because it is a prefix. Add the slash when you mean the directory.
What robots.txt cannot do
It is not security. Anything a browser can fetch without logging in, a script can fetch while ignoring robots.txt. Private material goes behind a login, not behind a Disallow. The file is public and lists exactly where you did not want people to look.
It does not remove pages from Google. A disallowed URL can still be indexed from links alone, shown with no description. To keep a page out of the index you need <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header, and the page must be crawlable so the crawler can see the tag. Disallowing and noindexing the same page is a contradiction: the crawler never reads the noindex. Google stopped honouring Noindex: lines inside robots.txt in 2019.
PDFs and images cannot carry a meta tag. The X-Robots-Tag response header is the tool for those.
It is not read by every AI feature. User-triggered fetches, browser extensions and "summarise this page" buttons work on the page the user is looking at. robots.txt governs crawling, not reading.
How to check it worked
In order of effort:
Open the file in a browser.https://yourdomain.ch/robots.txt should return plain text with a 200 status. A surprising number of sites return their 404 page, styled, with status 200, which crawlers parse as an empty file.
Google Search Console has a robots.txt report showing the last version Google fetched, when, and any parse errors. If you edit the file and the report still shows the old one, Google has not re-fetched yet; it usually does within a day.
Your server logs. Filter by the user agents above. If OAI-SearchBot fetched a hundred pages last week, it is allowed and active. If GPTBot keeps fetching after you disallowed it, either your file has an error, or the requests are not really from OpenAI. The operators publish IP ranges; a reverse DNS lookup on the request IP settles it.
A checker that reads the file as the bots do. The free analysis on this site fetches your robots.txt, parses it with the same prefix rules, and lists which of the AI crawlers may and may not reach your pages. The reference table with the current user agents lives in AI bot access.
Mistakes we see most
Blocking Googlebot to keep out Gemini. That removes you from Google Search. Use Google-Extended if training is the concern, and accept that AI Overviews come with Search.
A plugin's 2023 block list, never revisited. Several WordPress security plugins and CDN presets added Disallow: / for a list of "AI bots" that included what are now the search crawlers. Open the file and read it; it takes a minute.
A named group without the shared disallows. Covered above, and the most common way an /api/ or /account/ path ends up crawled.
Blocking CSS and JavaScript. Google renders pages; a disallowed stylesheet means it renders them broken and judges them that way. Only block assets you are certain nothing needs.
Testing with the wrong case or a trailing slash./Blog/ and /blog/ are different paths, /search and /search/ match differently. Test the exact URL that worried you.
If you also want to tell the crawlers what your site is, once they are allowed in, that is the job of llms.txt, which sits next to robots.txt and does the opposite: not where to go, but what to understand once there.
Saurav Chamoli7 min read
