- Google's readerGooglebot
- The program Google sends to read your pages for Google Search and its AI Overviews.Googlebot cannot crawl blocked pages to refresh their content; their URLs may still appear in search results.
- ChatGPT's web searchOAI-SearchBot
- The program ChatGPT uses to search the web when it answers a question.This opts blocked pages out of ChatGPT search answers; navigational links and answers from model knowledge are separate.
- Perplexity's readerPerplexityBot
- The program Perplexity sends to read your pages for its answers.If it is kept out, Perplexity leaves your pages out of its answers, though a person's live request may still get through.
- ChatGPT's page openerChatGPT-User
- The program ChatGPT uses to open a page on the spot when a person asks it to.If it is kept out, ChatGPT may fail to open your page when someone asks it to look; it can also ignore the rule.
- OpenAI's training collectorGPTBot
- The program OpenAI uses to collect text for training its models; it is not ChatGPT's search.Keeping it out opts you out of OpenAI training and does not change whether ChatGPT search can find you.
- Google's Gemini training switchGoogle-Extended
- A setting that tells Google whether your pages may be used to train and ground its Gemini models.Turning it off does not touch Google Search or AI Overviews; only keeping out Google's reader does that.
- Anthropic's training collectorClaudeBot
- The program Anthropic uses to collect text for training its Claude models.Keeping it out opts you out of Anthropic training and does not by itself change Claude's web answers.
- Perplexity's page openerPerplexity-User
- The program Perplexity uses to open a page on the spot when a person asks it to.Keeping it out changes little: it generally ignores your visitor rules anyway.
- Common Crawl's collectorCCBot
- The program that builds Common Crawl, a public copy of the web many AI models are trained on.Keeping it out leaves you out of that public copy, and so out of the models trained from it.
- your site's visitor rulesrobots.txt
- A small file on your site that tells automated readers which pages they may open.One line here can shut an AI engine out of your whole site.
- your page listsitemap
- A file listing the pages you want search engines to discover.It helps engines discover pages alongside the links between them.
- your page summary linemeta description
- A one-sentence summary of a page that sits in its code, not on the page itself.When it is missing, engines fall back to whatever text they can find to describe you.
- text that arrives with the pageserver-rendered text
- Words that are in the page the moment it loads, before any script runs.Text in the initial page is available even to readers that do not run scripts.
- quoting rulessnippet controls
- Settings on a page that tell engines whether, and how much, they may quote from it.A strict setting removes the page from AI Overviews and AI Mode and limits what AI answers can quote.
- how sure we areevidence tier
- Our label for how well proven a check is: documented by the engine, seen in large studies, or good practice.It tells you which problems are proven to cost you and which are simply worth tidying.
- Google's search summariesAI Overviews
- AI-generated summaries that Google may show alongside its regular search results.A supporting page must be indexed and eligible to show a snippet before Google can include it here.
- Google's conversational searchAI Mode
- Google's AI search experience for questions, comparisons and follow-up questions.Its supporting links require search indexing and snippet eligibility, but meeting these requirements does not ensure selection.