llms.txt: what the file does, what Google says about it — and when you should create one

“Should we create an llms.txt?” The question comes up in almost every client project, usually from someone who read on LinkedIn that without this file you are invisible in AI answers. The expectation behind it: a small entry in the root directory, and ChatGPT cites your brand more often.

That is not how it works. And to explain why, you have to separate two questions that are constantly mixed up. First: does the file help you rank better in Google or in AI answers? Second: what, then, is it good for at all? The short answer up front. For ranking, llms.txt does nothing, and Google now says so itself. As a technical signpost for AI tools it can still make sense — for a specific, clearly bounded case. That case is what I work out here.

What llms.txt actually is

llms.txt is a Markdown file in the root directory of a domain, that is at yourdomain.com/llms.txt. It lists the most important pages in curated form and describes in a sentence or two what each is about. The purpose: to save a language model the orientation work it would otherwise have to do by laboriously translating an HTML page overloaded with navigation, ads and JavaScript into plain text.

An analogy helps. llms.txt is to a language model what a table of contents is to the reader of a non-fiction book: a quick overview of where the important things stand. Unlike a table of contents, though, no one is required to look inside. No model is obliged to read the file, and most do not. The numbers on that in a moment.

The format was proposed by Jeremy Howard in September 2024, co-founder of the research labs fast.ai and Answer.AI. His draft knows two files. The llms.txt links the central pages with short descriptions. The optional llms-full.txt packs the full content into a single, long Markdown file, including examples and how-tos. Both are meant as help at runtime, that is for the moment someone actively asks a model something — not as a training or ranking signal. That is not my interpretation, it stands so in the original proposal.

Important is the distinction from two files you know. The robots.txt controls which areas a crawler may enter; it contains instructions. The llms.txt contains not a single instruction — it permits nothing and forbids nothing, it only shows. The sitemap.xml, in turn, lists all indexable pages for search engines, potentially millions of URLs. The llms.txt is the opposite: a strict selection of the few pages that matter to a model. Whoever confuses the two and dumps their entire sitemap into the llms.txt has misunderstood the format.

What Google officially says

For a long time the position was ambiguous, and that fed the myth. In June 2026, Google added a dedicated section on llms.txt to its guide on optimizing for generative AI features. The core, in Google’s own words: you need no machine-readable files, AI text files, no markup and no Markdown to show up in Google Search, “because Google Search itself does not use them.” And further: whoever wants to maintain such files for other systems can do so, it “neither hurts your visibility or your rankings in Google Search, nor helps, since Google Search ignores them.”

It hardly gets clearer. No penalty, no bonus, simply not a factor. Search Engine Roundtable picked up the clarification in mid-June, with exactly this message: “won’t help or hurt.”

Interesting is the reasoning behind it, because it clears up a widespread misunderstanding about GEO, Generative Engine Optimization. From Google’s point of view, optimizing for AI features is not a separate discipline alongside SEO. It belongs to it. The AI Overviews and AI Mode draw on the same search index and the same ranking systems, via retrieval-augmented generation (RAG), that is looking up current pages in the index. Whoever is cleanly discoverable for classic search is thereby discoverable for the AI answers too. An extra file changes nothing about that.

How stubborn the myth is shows in an episode from Google’s own house. In December 2025 an llms.txt briefly appeared on Google’s own developer documentation — and was gone hours later, the URL returned a 404. The reason was no change of strategy. An internal content system had rolled the file out automatically. Gary Illyes made clear at Search Central Live that Google does not support llms.txt and does not plan to. John Mueller compared the format to the old keywords meta tag: a self-declared signal you would have to check against the real page anyway — and then you might as well read the page.

What the data shows about its use

Google is one thing. But do the others at least — ChatGPT, Perplexity, Claude, Gemini — use the file? Here it pays to look at real access data instead of advice articles.

Ahrefs analyzed around 137,000 websites with an llms.txt. The result is sobering: 97 percent of the files were never fetched by any bot. Of the bots that did access, 77 percent were not AI tools at all, but SEO crawlers, profiling services and unknowns. The AI fetchers that actually produce citations — OpenAI’s search bot, PerplexityBot, Claude’s crawler — accounted for about one percent of the requests. Across thousands of pages, these three together added up to only a few hundred fetches.

The simplest check comes from John Mueller and costs you five minutes: look into your server logs. Do not ask whether the models could use the file, but whether they request /llms.txt at all. In the vast majority of logs it says: no. Not a single AI answer system has so far officially announced that it supports the format. As a lever for citations in AI answers, llms.txt is therefore, soberly viewed, decoration.

The counter-check: 2,041 sources from a real project

Third-party studies are one thing. I wanted to see the question on a live project. For a Swiss pension and finance client, I took all 2,041 source domains that ChatGPT, Perplexity and the other models actually drew on for this client’s prompt set from early June to early July 2026, measured through Peec.ai. Not just any websites. Exactly the domains from which the models feed their answers in this market. Each one I checked for an llms.txt.

First result: 510 of the 2,041 domains carried one, that is 25 percent. Three of four sources the models rely on in this sector get by without an llms.txt. Broken down by domain type:

Seven competitor domains and the client page itself I leave out of the rate because of too small a base.

The real question is not who has a file. It is whether it does anything. Were domains with an llms.txt fetched or cited more often than those without? No. Rather less. The 510 domains with a file were on average even fetched less often than those without (8.5 versus 13.9 fetches per domain) and not cited more often (mean citation rate 0.26 versus 0.29). They make up a quarter of all sources but capture only 17 percent of the fetches. The statistical association between “has an llms.txt” and visibility lies between minus 0.02 and minus 0.05. That is zero, with a hint toward the negative.

An honest word on framing. This is a correlation, not causal proof, and it has an obvious reason: the most-cited sources in this market are editorial pages and reference works, and those in particular rarely maintain an llms.txt. One client, one sector, one month. That is not enough for a law of nature. For a direction, it is. If the file were the lever that brings citations, it would have to show up in exactly this data. It does not.

This is the point where many GEO checklists turn dishonest. They list llms.txt as a mandatory item because it is easy to tick off — not because it works. What really moves visibility in AI answers are other things: clean crawlability, clear entities and authority signals from third-party sources. How a model actually selects its sources I traced on ChatGPT’s raw data; the technical duty behind it stands in the piece on crawl budget. An llms.txt does not appear in that chain.

What the file is good for, then

No ranking effect does not mean worthless. It only means: a different purpose. llms.txt helps where you point an AI tool deliberately at your content — in a controlled context, not in the open web.

The clear winner is developer documentation. Whoever offers a public API or a product that other developers integrate via AI-assisted editors like Cursor or Claude has a tangible benefit from a good llms.txt. The developer asks their tool to use your API; the tool pulls your llms.txt together with the linked Markdown pages into context and writes correct code against your endpoints. That is exactly what Jeremy Howard devised the format for, and exactly how providers like Perplexity, Anthropic, Stripe or Cloudflare use it for their docs. Peec.ai, the tool I use to measure AI visibility, did the same and built an llms.txt for its own API — as help for AI tools, not as an SEO trick.

An example I carry through the practical part: the fictional Rheintal Software GmbH, a mid-sized company with a warehouse management system and a public API. Rheintal has two audiences. Customers who google — for them classic visibility counts, and no llms.txt helps there. And developers who build the API into their systems, often with an AI assistant at their side. For the second group, a lean llms.txt pointing to the API docs is a real advantage. Same company, two cases, one clear line.

A second sensible case: you want to feed your own AI applications with your knowledge, say an internal assistant or a RAG solution. Then an llms-full.txt is a convenient, clean source in one piece. The common denominator of both cases: you or your partner deliberately point a tool at the file. No one is waiting for an anonymous crawler to discover it by chance and therefore name your brand more often.

Practice: how to create one

If your case fits, the implementation is quickly done. The format is deliberately simple.

The structure follows a fixed order: an H1 with the name (the only mandatory field), below it a quote block with a short description, then arbitrary sections with H2 headings, under each a list of links in the format [Name](URL): short note. A section called ## Optional has a special role: what stands there a model may drop if space runs short. Here is how that looks for Rheintal Software GmbH:

# Rheintal Software

> Warehouse management for mid-sized businesses. This file helps AI tools
> find our API and the most important help pages.

## Documentation

- [API quickstart](https://rheintal.example/docs/quickstart.md): First request to the inventory API in five minutes.
- [Authentication](https://rheintal.example/docs/auth.md): Create API keys and sign requests.
- [Query inventory](https://rheintal.example/docs/inventory.md): Endpoints for stock, reservation, reordering.

## Policies

- [Return conditions](https://rheintal.example/returns.md): Deadlines and process for returns.

## Optional

- [Changelog](https://rheintal.example/changelog.md)

A real example to open: my own file at eullrich.com/llms.txt. Frankly, I run a personal brand, not an API product — by the logic of this article I would not need one. I keep one anyway, lean and curated, and treat it as what it is: a test case. I look into the logs to see whether a tool ever fetches it, rather than assuming it does.

Four steps get you there.

Step 1 – Define the purpose. Clarify which tool is supposed to read the file and which task it will solve with it. The selection of pages follows from that. Result: a sentence like “an AI coding tool should be able to develop against our API.” Without this sentence, the file becomes a random link list.

Step 2 – Curate the pages. Take the few pages that support exactly this task, and give each link a brief description. Less is more. Expected result: a handful of entries, not fifty. Most common mistake: copying in half the sitemap, whereupon the model can no longer find the important thing in the noise.

Step 3 – Place and serve it. Save the file as plain text at yourdomain.com/llms.txt. Open the URL in the browser; unformatted Markdown text must appear, served as text/plain or text/markdown. If your HTML page or a 404 comes up instead, it is usually a rewrite rule or a wrong path.

Step 4 – Check that it works. Copy the content into a fresh chat window, turn web search off and have the model answer exclusively from the file. If it answers your test questions correctly, the file is usable. If it stumbles, a page is missing or a description is too vague.

A word on the .md copies, because this is where the most common mistake starts. Some create a Markdown second version of every blog article and link them all from the llms.txt. That produces duplicate content, splits ranking signals across two URLs per piece and burns crawl budget — a real harm to your classic visibility, without a demonstrable gain on the AI side. Malte Landwehr, CPO of Peec.ai, calls this, for SEO purposes, “a distraction without benefit.” Markdown versions yes — but deliberately for doc pages an AI tool really consumes, not blanket for marketing content.

Frequently asked questions

Does an llms.txt hurt my Google ranking? The file itself does not. Google ignores it, neither penalty nor bonus. Harm only arises when you create and link an additional .md copy of every page: that can, as duplicate content, drag down your classic visibility.

Do ChatGPT, Perplexity or Gemini use the file for citations? No provider has announced it. The Ahrefs analysis of 137,000 pages shows about one percent of fetches by AI bots, 97 percent of files were never read. For citations in AI answers there is no demonstrated benefit.

Do I need a .md version of every page? No. That is the most expensive misconception around llms.txt. Create Markdown versions only for the doc or API pages an AI tool is meant to read deliberately.

Who does llms.txt concretely pay off for? For anyone whose audience includes AI agents, coding tools and developers with API access. For a classic marketing or shop page hoping for more ChatGPT mentions, it does not pay off — there, crawlability, clear entities and third-party sources count.

So, create one now?

Back to the opening question. “Should we create an llms.txt?” hangs on a single sub-question: who is supposed to read it? If you have a public API or docs that developers integrate with AI tools, build a lean, curated file — the effort is small, the benefit real. If, on the other hand, you are after more visibility in AI answers, spare yourself the file and put the time into the levers that measurably work. The concrete next step costs five minutes: open your server logs and count how often /llms.txt was requested in the last 90 days. That number ends the discussion faster than any LinkedIn post.