Skip to content
visible4ai
E-E-A-T CheckerLearnAI File GeneratorPricing
en|
Sign inAccount
Loading
visible4ai
E-E-A-T CheckerLearnBlogAI File GeneratorPricingContact
PrivacyTermsRefundsImprintCancel contracts hereWithdraw from contract

visible4ai on Product Hunt

© 2026 visible4ai. A product of Siempi AG.

We measure whether AI tools cite your website and show you how to improve.

All articles·Technical·4 min read·Updated Nov 5, 2026Nov 5, 2026Updated November 5, 2026

AI bot access

If one of these bots is blocked in robots.txt, that platform can't use your pages for its answers. Make sure they are allowed, on the homepage and on your content pages:

Bot / User-AgentPlatform
OAI-SearchBot, ChatGPT-UserOpenAI (ChatGPT)
PerplexityBotPerplexity
Claude-SearchBot, Claude-UserClaude
Googlebot, Google-ExtendedGemini / AI Overviews
BingbotCopilot

Some crawlers only collect training data: GPTBot, ClaudeBot and CCBot. Blocking them doesn't affect answers or citations. Google-Extended is different: it also controls Gemini app answers, so blocking it keeps you out of Gemini training and Gemini answers, while Google Search is not affected.

Example robots.txt rules

robots.txt is a small text file at the root of your site (visible4ai.com/robots.txt). Crawlers read it before fetching any page. Here are the two common cases:

Allow all major AI crawlers

This is the safe default if your goal is to be found in AI products. Explicit Allow rules make your intent obvious to anyone reading the file later.

robots.txt
# robots.txt — allow major AI crawlers
# (Allow: / is also the default when no rules exist;
#  spelling it out makes intent explicit.)

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: bingbot
Allow: /

Block one bot, keep the rest

If you want to stay visible everywhere except one platform (commonly because that platform also uses your content for training), block that specific user-agent and leave a wildcard Allow for everyone else.

robots.txt
# robots.txt — opt one bot out, allow the rest
# Useful if you want to stay citable but exclude a specific
# crawler from training your own content.

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

robots.txt is a request, not enforcement. Well-behaved crawlers (the ones listed above) honour it; abusive scrapers ignore it. If you need real blocking, use server-level rules (firewall, rate limit, or 403 by user-agent).

Now audit your own site

See your GEO score, check if you're cited in Perplexity, ChatGPT, Gemini, and Claude, and get prioritized quick wins, in about a minute.

Analyze my site →Generate your AI files →
AI files: what crawlers readWhat works by provider