Be in the index first
AI answers are built from pages an engine can fetch, so the first job is letting the right crawlers in. Each engine has its own robots.txt name for search, and it’s often not the one people block or allow by habit.
| Engine | Lets your pages appear in answers | Training or other use only |
|---|---|---|
| Google AI Overviews and AI Mode | Googlebot, plus the Search generative AI setting in Search Console left on “include” | Google-Extended (Gemini Apps and Vertex AI; no effect on Google Search) |
| ChatGPT search | OAI-SearchBot | GPTBot (model training) |
| Perplexity | PerplexityBot | — |
| Claude | Claude-SearchBot; Claude-User for fetches a user asks for | ClaudeBot (model training) |
Sources: Google crawlers, Search Console Help, OpenAI, Perplexity, Anthropic.
Two traps. Blocking GPTBot or ClaudeBot opts you out of training, not out of search answers, so a site can block them and still appear. Blocking OAI-SearchBot or PerplexityBot does keep you out of those engines’ answers. The second trap sits below robots.txt: Google asks that crawling be allowed “by any CDN or hosting infrastructure” too, and OpenAI and Perplexity both recommend allowing their published IP ranges. A firewall that challenges unknown bots can undo a clean robots.txt.
JavaScript is worth a check as well. Google says it can process JavaScript content as long as it isn’t blocked. The OpenAI, Perplexity and Anthropic crawler pages don’t say whether their bots run it, so we’d keep the key answers in the HTML the server sends.
How do you get into Google's AI Overviews?
The same way you get into Google’s regular results. Google says there are no additional requirements and no special optimisations needed to appear in AI Overviews or AI Mode (Google Search Central). What decides it:
- The page is indexed and can show a snippet. That’s the stated technical requirement for being a supporting link. A
nosnippetor very lowmax-snippetsetting limits what Google can show from the page. - The site is included in Search generative AI features. It’s on by default. Search Console now has a setting to exclude a site, and it’s worth confirming nobody switched it off (Search Console Help).
- The page ranks well for the question and its neighbours. AI Overviews and AI Mode may run a “query fan-out”, issuing related searches across subtopics, so a page that answers a likely sub-question can be pulled in. Google warns that creating pages for every fan-out variation mainly to manipulate AI responses breaks its scaled content abuse policy (Google Search Central).
- An AI Overview shows for that query at all. Google says they appear only when its systems judge them additive to classic Search, and often don’t trigger. Search your target questions before planning around them.
Indexing and serving aren’t guaranteed even when everything above is in place. Google says so on the same page.
Make your facts easy to quote
Give clear, direct answers at the start of the section, and keep the facts about who you are and what you do the same everywhere they appear. A quotable passage states one fact in a self-contained sentence, names the source of every figure, and avoids pronouns that only make sense after the paragraph above.
There’s evidence for the sourcing part. In the 2023 study that named GEO, adding citations, quotations from relevant sources and statistics did best at making a page visible in AI answers, with gains of up to 40% on the authors’ test engine (arXiv 2311.09735). That’s a lab result on 2023 models, not a promise for any live engine.
Don’t overdo the format. Google says there’s no need to break content into tiny chunks or rewrite it for AI systems, and structured data isn’t required for its AI features. If you use it, it has to match the visible text (Google Search Central).
Build citable authority
Publish something an engine can’t get elsewhere. Google’s guide puts it as non-commodity content: first-hand experience, original data, a point of view, rather than a summary of what’s already online (Google Search Central). Its example contrasts a generic “7 Tips for First-Time Homebuyers” with a first-hand account of waiving an inspection.
Off your site, the aim is real coverage. Google notes its AI features can show what’s said about products and services in blogs, videos and forums, and that seeking inauthentic mentions isn’t as helpful as it might seem. So earn reviews, press and expert quotes, and keep the facts on profiles you control in line with your site. For local businesses and shops, Google points specifically to Business Profile and Merchant Center details.
The engines don't all pull the same way
“AI search” isn’t one target, and each engine documents only part of how it works.
- Google AI Overviews and AI Mode retrieve pages from the Search index with Google’s core ranking systems (Google Search Central). Classic ranking strength carries over directly.
- Gemini Apps can ground answers in the Google Search index; Google-Extended controls whether your content is used for that grounding and for training (Google crawlers).
- ChatGPT surfaces sites in its search features through OAI-SearchBot. When a user asks a question, ChatGPT may also visit a page with ChatGPT-User, and OpenAI says robots.txt rules may not apply to those user-initiated visits (OpenAI).
- Perplexity uses PerplexityBot to surface and link sites in its results, and Perplexity-User to visit pages a user’s question needs and link them in the answer (Perplexity).
Answers given without a live search come from what a model learned in training, and that can’t be edited page by page. If one engine ignores you, check that engine’s crawler access first, then whether it cites links for that kind of question, before rewriting content.
What can you skip?
For Google, a fair amount of popular AEO and GEO advice. Google’s own list of things you can ignore (Google Search Central):
- llms.txt and other “AI” files or markup: Google Search doesn’t use them. Keeping one for other services neither helps nor harms you in Google.
- Chunking content into small pieces for AI. Google says its systems can find the relevant part of a longer page.
- Rewriting just for AI, including chasing every long-tail phrasing. Google says its systems match synonyms and meaning.
- Inauthentic mentions across forums and blogs.
- Special schema. Structured data isn’t required for AI features, though it still helps with rich results.
This list covers Google Search only. Other engines haven’t published an equivalent, so test before you assume either way.
Common mistakes and how to measure it
The frequent errors are treating AI citation as a trick separate from being a good source, chasing a model’s training data instead of the live search layer you can influence, and blocking a search crawler while meaning to block a training one.
Measure in three places:
- For Google, Search Console’s Generative AI performance report shows impressions from AI Overviews and AI Mode by page, country and device, rolled out to all sites by August 31, 2026 (Search Console Help). It counts impressions; for clicks, use the Web search type in the regular Performance report, which includes these features.
- For other engines, ask your target questions on a fixed schedule, log whether you’re named or linked, and keep screenshots with dates. No first-party report exists for these yet.
- In analytics, watch referral traffic from chatgpt.com, perplexity.ai and similar, and whether branded search is rising.
A simple test before any of this: if a reader wouldn’t cite the page, no amount of markup will make an engine want to. If you’re buying this work, how to check your vendor is working covers what reports to ask for.