llms.txt: what it is and whether you need one

Somewhere in the past year, an agency email, a LinkedIn post, or a checklist told you that serious websites now need an llms.txt file, "like robots.txt, but for AI". The pitch usually implies that without one, ChatGPT and Google's AI can't find you, and that adding one is an easy AI-visibility win.

Here is the two-sentence honest version: llms.txt is a genuine, well-intentioned proposal that some documentation-heavy sites have adopted, and there is no public evidence that any major AI system reads it, while Google has stated in writing that it ignores it. Everything else in this article is the detail behind those two sentences, so you can make the call in the time it deserves, which is about five minutes.

What llms.txt actually is

The proposal comes from Jeremy Howard, a well-known AI researcher at Answer.AI, who published it in September 2024 at llmstxt.org. The idea is simple. Language models that visit websites struggle with what websites are made of: navigation, cookie banners, scripts, and markup that bury the actual content. So a site can offer a clean map at a standard address, yourdomain.com/llms.txt.

The file itself is ordinary Markdown, which is the plain-text formatting that AI systems handle most naturally. The format calls for a title, a short summary in a quote block, and then organized lists of links to your important pages, each with a note about what it contains. A companion convention, llms-full.txt, goes further and packs the full text of your documentation into one big file, so an AI tool can ingest everything in a single request.

Note what this is: a menu offered to AI systems. Not a fence. It has no power to allow or block anything, which is the first thing the "robots.txt for AI" framing gets wrong.

Who actually uses it

On the publishing side, adoption is real but concentrated in one neighborhood: developer documentation. OpenAI's own docs reference an llms.txt index. Anthropic publishes one for its Claude documentation, with the llms-full.txt companion. Documentation platforms like Mintlify and ReadTheDocs generate the files automatically for the sites they host. If you spend your day in software docs, llms.txt is a normal sight.

The consuming side is the problem. For the standard to do anything, AI systems have to fetch and use the file, and there is no public confirmation that any major platform does. OpenAI, Anthropic, Google, and Perplexity have collectively published nothing saying "our assistant reads your llms.txt". The clearest statement on record points the other way. Google's guidance for AI-era site owners says: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search", and states specifically that llms.txt files are ignored for Search purposes and "will neither harm nor help your site's visibility or rankings".

Notice the asymmetry: the AI labs publish llms.txt files for their own docs, and none have committed to reading yours. That tells you the file's proven use case, which is real: it helps developers who deliberately point an AI tool at your documentation. It's an index for invited guests, not a beacon that summons traffic.

llms.txt is not robots.txt

The comparison that sells llms.txt also confuses it, so it's worth separating the two files cleanly.

robots.txt is about *permission*. It's the decades-old advisory standard where you tell crawlers what they may not fetch, and the AI companies do engage with it: they operate named crawlers with published identities, like OpenAI's GPTBot, precisely so sites can allow or block them there. Many do block: an analysis noted by Wikipedia found 306 of the 1,000 most-visited websites block GPTBot. Whether to let AI crawlers in is a genuine decision with real consequences for whether AI systems can cite you at all, and it happens in robots.txt, not llms.txt.

llms.txt is about *presentation*: assuming a system already wants your content, it offers a cleaner path to it. Permission has enforcement and adoption behind it. Presentation, so far, has hope.

So: do you need one?

For a small business website, the decision tree is short.

If your site is a normal business site, a blog, services, contact page, then no, you don't need one, and adding one will change nothing measurable. Google has said so for its half of the AI landscape, and no other platform has promised otherwise. The AI systems that matter already read your actual pages the way search engines do. The hour you'd spend on llms.txt is worth more spent on the things with evidence behind them: clear, extractable answers on your pages, the AEO fundamentals, consistent business information, and real content depth.

If you publish documentation, an API, or reference material that people feed into AI tools, then yes, add one, because you're serving the one audience with a demonstrated use for it, and your docs platform can probably generate it automatically. Cost: near zero. Benefit: modest, real, and aimed at developers, not rankings.

If you're deciding whether AI companies may use your content at all, that's the robots.txt conversation, and it's about GPTBot and its peers, not llms.txt.

The one wrong answer is paying someone meaningful money to "implement llms.txt for AI SEO". Anyone selling it as a visibility upgrade is selling ahead of the evidence, and Google's flat statement makes that pitch, at best, out of date. Treat it like any other claimed shortcut: AI search rewards the same substance search always did, and no index file changes what your pages contain.

If you want one anyway

Fair enough: it's ten minutes and it can't hurt, which is also Google's official assessment. Create a plain text file, format it as Markdown, and put it at yourdomain.com/llms.txt. A minimal, honest version for a service business looks like:

• An H1 line with your business name.
• One quoted sentence saying what you do and where.
• A "Key pages" section: links to your services, pricing, and contact pages, each with a one-line description.
• A "Guides" section: links to your handful of best articles, one line each.

Keep it accurate and short, update it when your key pages change, and file it mentally next to your XML sitemap: cheap plumbing, maintained out of tidiness, expected to move nothing on its own.

Quick answers to the follow-up questions

Does llms.txt affect Google rankings? No, by Google's own statement: Search ignores the file, so it neither helps nor harms. Anyone claiming ranking benefits is contradicting the vendor of the ranking.

Will ChatGPT or Perplexity read mine? Neither company has said so. Their search crawlers read your ordinary pages, the same ones Google reads, which is why normal content quality and accessibility is where AI visibility is actually decided.

Is llms.txt how I block AI from using my content? No, it has no blocking power at all. Blocking happens in robots.txt, against the named crawlers (GPTBot and its peers), and it's a real decision with a real trade-off: blocked crawlers can't train on your content, but blocking the search-focused bots also removes you from citation and recommendation in those tools.

Does the file need maintenance? Only the same kind your sitemap needs: update it when key pages change. A stale llms.txt pointing at dead URLs is worse than none, since its one plausible reader is someone deliberately trying to use your content.

My agency quoted me for an "llms.txt implementation." Fair? Ten minutes of work at whatever you consider a fair rate for ten minutes. If it's bundled inside genuine work, clean site structure, extractable content, entity consistency, judge the bundle on those parts. If it's sold alone as an AI-visibility unlock, you now know enough to decline politely.

The one-line version

llms.txt is a reasonable idea waiting for an audience: useful today if AI-tool users deliberately consume your documentation, inert for everyone else, and explicitly ignored by Google. Add it in ten minutes if you like tidy infrastructure, skip it without guilt if you don't, and spend the real effort on the content and clarity that AI systems verifiably reward.