Optimizing content for Perplexity, Claude, and Gemini

Most advice about AI search talks about 'the AI engines' as if they were one thing. They aren't. Perplexity runs its own crawler and index. Claude searches the live web when a conversation needs it. Gemini sits on top of Google's index. Each one finds — or misses — your site through different plumbing, and the practical work splits into two unequal parts: a half-hour of checking that none of that plumbing is blocked, and the ongoing work of writing content these systems quote. The strategy piece and the ChatGPT specifics are covered elsewhere; this article is about the other three engines and the plumbing.

How each one reaches your site

Perplexity is the most search-engine-like of the three: it maintains its own index. Its documentation describes two crawlers. PerplexityBot builds the index and obeys robots.txt — the small text file on every website that tells crawlers what they may visit. Perplexity-User fetches a page live when a user's question calls for it, and Perplexity's docs say plainly that it generally ignores robots.txt, on the logic that it's acting for a person, the way a browser does. Both publish their server addresses so sites can verify who's really knocking.

Claude doesn't maintain a public search index; it searches the web during a conversation when the question needs current information. Anthropic lists three crawlers — ClaudeBot for model training, Claude-User for fetches a user triggered, Claude-SearchBot for search — and states that they honor robots.txt directives. Which index sits behind Claude's search, Anthropic doesn't disclose.

Gemini is the simplest to reason about: its answers are grounded in Google Search. If your pages rank in Google, they're in the pool Gemini draws from — the same pool as AI Overviews. There's no separate Gemini crawler to court; Google's regular crawling covers it.

The robots.txt check: one accident to find, one non-lever to skip

The plumbing work is mostly making sure nobody blocked these bots by mistake. In 2023–2024, many sites added blanket blocks against AI crawlers to keep their content out of model training. Understandable — but some of those blocks now also keep the site out of AI search answers, which is where customers increasingly ask for recommendations. Open yoursite.com/robots.txt and look for PerplexityBot, ClaudeBot, Claude-SearchBot, GPTBot, or OAI-SearchBot under a Disallow line. Training crawlers (ClaudeBot, GPTBot) and search crawlers (Claude-SearchBot, OAI-SearchBot — blocking that one removes you from ChatGPT's answers) are separate entries, so you can refuse training and still appear in answers. If you never added such blocks, you're fine; if an agency set up your site, it's worth the look.

The reverse confusion also costs people: a token called Google-Extended lets you block Gemini from using your content for training and grounding. Google's docs are explicit that it doesn't affect your Google Search inclusion or ranking — and, the part most posts get wrong, blocking it does not remove you from AI Overviews either, because Google treats AI answers as part of Search itself, fetched by the same Googlebot. So Google-Extended is purely a 'may Gemini use my content' switch. Blocking everything AI while staying visible in AI-assisted search is, by design, not really on offer from Google.

Blocking Google-Extended keeps Gemini from using your content. It does not remove you from AI Overviews — those are part of Google Search, and only blocking Googlebot opts you out.

While you're in the plumbing: AI referral visits are easy to spot. ChatGPT tags its outbound links with utm_source=chatgpt.com, and Perplexity, Claude, and Gemini traffic shows under their domains in your analytics. The counts will look small next to Google. Watch them anyway — they tell you when your niche's customers start asking assistants instead of searching, and those visitors arrive unusually far along in their decision.

Writing for engines that quote rather than rank

Here's the good news buried in the plumbing differences: the content work doesn't split three ways. All of these systems compose an answer and cite a handful of sources, which means they're all selecting for the same property — quotability.

The best evidence on what earns citations is still the Princeton-led study that tested nine content changes across a 10,000-query benchmark. Concrete statistics in the text lifted a page's visibility in AI answers by as much as 40%, the largest effect measured; direct quotations from named people and citing your own sources produced similar gains; keyword stuffing scored worse than doing nothing. AI systems quote pages that read like evidence.

For a small business site, that translates into habits more than projects:

• Lead sections with the answer. A question as a heading, a complete answer in the first sentence beneath it, detail after — extractable structure is what gets extracted.
• Put your real numbers on the page: prices, timelines, results. 'Bathroom renovations take us three to five weeks' is quotable; 'fast turnaround times' is not.
• Say where claims come from, even when the source is your own experience: 'across the forty kitchens we fitted last year' is a citation an AI answer can carry.
• Keep pages current. Assistants answering live questions favor pages whose information doesn't look abandoned.

What not to do

Skip llms.txt, the proposed 'menu file for AI systems' — Google has said it doesn't use it, John Mueller has compared it to the long-dead keywords meta tag, and the major assistants don't request it. Skip per-engine content strategies; where citation patterns have been measured (Semrush found Reddit appearing in roughly 40% of AI answers and Wikipedia in about 26%), the pattern's main feature was volatility — it reshuffles with every model update, so content tuned to today's favorite source is tuned to nothing. And skip any service guaranteeing placement in AI answers. Nobody controls these systems, including the companies that run them.

The half-hour version

Check robots.txt for accidental blocks. Confirm you rank in Google, which covers Gemini and AI Overviews in one move. Then put your numbers, your sources, and your directly-stated answers into the pages that matter, and let three different kinds of plumbing carry the same quotable content to wherever your next customer happens to ask.