AI bot access
If one of these bots is blocked in robots.txt, that platform can't use your pages for its answers. Make sure they are allowed, on the homepage and on your content pages:
| Bot / User-Agent | Platform |
|---|---|
| OAI-SearchBot, ChatGPT-User | OpenAI (ChatGPT) |
| PerplexityBot | Perplexity |
| Claude-SearchBot, Claude-User | Claude |
| Googlebot, Google-Extended | Gemini / AI Overviews |
| Bingbot | Copilot |
Some crawlers only collect training data: GPTBot, ClaudeBot and CCBot. Blocking them doesn't affect answers or citations. Google-Extended is different: it also controls Gemini app answers, so blocking it keeps you out of Gemini training and Gemini answers, while Google Search is not affected.
Example robots.txt rules
robots.txt is a small text file at the root of your site (visible4ai.com/robots.txt). Crawlers read it before fetching any page. Here are the two common cases:
Allow all major AI crawlers
This is the safe default if your goal is to be found in AI products. Explicit Allow rules make your intent obvious to anyone reading the file later.
# robots.txt — allow major AI crawlers # (Allow: / is also the default when no rules exist; # spelling it out makes intent explicit.) User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: bingbot Allow: /
Block one bot, keep the rest
If you want to stay visible everywhere except one platform (commonly because that platform also uses your content for training), block that specific user-agent and leave a wildcard Allow for everyone else.
# robots.txt — opt one bot out, allow the rest # Useful if you want to stay citable but exclude a specific # crawler from training your own content. User-agent: GPTBot Disallow: / User-agent: * Allow: /
robots.txt is a request, not enforcement. Well-behaved crawlers (the ones listed above) honour it; abusive scrapers ignore it. If you need real blocking, use server-level rules (firewall, rate limit, or 403 by user-agent).