4 minUpdated Nov 12, 2026

AI standards: what providers actually document

Last checked: 12 November 2026, against each provider's own documentation. We check again every quarter.

Many files and headers claim to steer AI tools. Only some of them are documented by the companies that run the crawlers. This page sorts them into three levels, and says what visible4ai does with each.

1Documented by providers

The AI or search provider says in its own docs that it reads this. visible4ai checks most of these.

robots.txt user agents
OAI-SearchBot, ChatGPT-User, GPTBot (OpenAI); Claude-SearchBot, Claude-User, ClaudeBot (Anthropic); PerplexityBot, Perplexity-User; Googlebot and Google-Extended; Applebot, Amzn-SearchBot, Meta-WebIndexer, MistralAI-Index. The GEO score counts OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Google-Extended. ChatGPT-User and Perplexity-User are shown but not counted, because their providers say robots.txt may not apply to them; the rest are shown for information.
Shared AI training opt-out in robots.txt
User-agent: __usage_ai-training, from the IETF AIPREF work. Anthropic began recognising it on 30 October 2026 and is rolling it out over time, Bing honours it from 1 February 2027, and Google aims to support it in 2027. It is about training, not about whether AI search cites you; visible4ai does not score it.
Robots meta tag and X-Robots-Tag
noindex, nosnippet, data-nosnippet and max-snippet (Google: they also apply to AI Overviews and AI Mode). Bing: noarchive and nocache limit Copilot answers today. From 1 February 2027 Bing reads the new ai-grounding and ai-training preferences instead, and after 1 March 2027 it no longer reads noarchive and nocache.
Sitemaps with accurate lastmod
Google and Bing use lastmod to recrawl changed pages, as long as the dates are true.
IndexNow
Tells Bing and other search engines right away when a page changes.
Structured data (schema.org)
For example Organization markup with your name, URL and logo, documented by Google. Bing says in general that it reads structured data.
Published crawler IP ranges
Providers publish them so you can tell their real bots from fakes.

2Public proposals

Written down and open, but no AI search provider documents them as a signal for what it shows or cites. visible4ai shows them for information, with 0 points.

llms.txt
Proposal at llmstxt.org (v2, August 2026). Google says such files neither help nor harm.
Markdown for Agents
Cloudflare (February 2026): a page answers in Markdown when asked for it. According to Cloudflare, coding agents such as Claude Code already ask for Markdown; no AI search provider documents using it.
Content-Signal and Content-Usage in robots.txt
Cloudflare's Content-Signal and the Content-Usage line of the IETF AIPREF vocabulary (draft). The shared training opt-out from the same work is in level 1; these finer signals are not documented by any provider yet.
On the watch list
Web Bot Auth (IETF working group since October 2025), WebMCP (W3C community draft), MCP Server Cards (SEP-2127, final), RSL 1.0, tdmrep.json, UCP (commerce only).

3No provider-backed spec

No AI provider stands behind them or documents reading them. visible4ai shows them only when your site has them, with 0 points.

ai.txt, brand.txt, ai.json, identity.json, faq-ai.txt, developer-ai.txt, robots-ai.txt, llm.txt, llms.html
One agency's file set, with its own published spec (version 1.14.0). Harmless to keep, but don't expect them to change AI answers.
ecp.json
No named authors and no versioned spec. visible4ai no longer checks it.

The providers' own numbers

Two providers now report on their AI answers themselves: Google Search Console's generative AI performance report counts impressions of your pages in AI Overviews and AI Mode, and Bing Webmaster Tools' AI Performance counts how often Bing's AI answers cite them. Use them next to your GEO score: they count what happened on their side, we check whether AI tools cite you for your questions.

Sources

Now audit your own site

See your GEO score, check if you're cited in Perplexity, ChatGPT, Gemini, and Claude, and get prioritized quick wins, in under a minute.

Analyze my site →Generate your AI files →

Agent integration with the visible4ai API

All articles