llms-full.txt Explained: What It Is and When You Need One
- llms.txt is an index. llms-full.txt is the full text of every page inlined into one file.
- The index stays small enough to read in a single pass. The full file can run to hundreds of thousands of words.
- Only documentation sites with a deep body of reference material genuinely need the full file.
- Adoption is partial. No major AI operator has publicly committed to honoring either file.
- Publishing a file is not the same as a crawler fetching it. Only your server sees which one happened.
llms-full.txt is a single plain-text file that holds your site's entire content in one place. You need it only if your site is a documentation site with a deep reference section. Almost nobody else does. Anyone searching for llms full txt is usually trying to work out which camp they fall into.
What llms-full.txt actually is
Two files share a name and do different jobs. The first, llms.txt, is an index. It sits at /llms.txt and lists your important pages as markdown links with a short description of each. It is built to be read in one pass, so a model gets a table of contents instead of crawling your navigation.
llms-full.txt inlines the content. Every page the index points at gets pasted into one file, in order, as plain text or markdown. A model that opens it reads your whole corpus without fetching anything else.
The index says where things are. The full file hands over the things themselves.
The index and the corpus, side by side
| llms.txt | llms-full.txt | |
|---|---|---|
| What it holds | Titles, URLs, short descriptions | The complete text of every page |
| Typical size | A few kilobytes | Can reach hundreds of thousands of words |
| How it reads | One pass, no follow-up fetches | One long context or a chunked pass |
| Best fit | Most sites | Documentation and reference material |
| Upkeep | Low | Regenerate on every content change |
How big the full file gets
Size depends on how much you have written. A five-page marketing site produces a file smaller than most emails. A documentation site with an API reference and a deep product manual can produce a file in the hundreds of thousands of words. Both are valid. The second is where the format earns its keep.
The value of a single file is context. A model putting together an answer about your product benefits from having the current reference material in front of it, rather than a link it has to fetch and a page it may or may not retrieve. Whether any given model opens the file is a separate question, and the section below gets to it.
Who actually needs llms-full.txt
Documentation sites. That is close to the whole list.
If your site's value is a body of technical writing, an API reference, or a detailed manual, a single-file dump gives a model the whole picture in one read. Picture a docs site where a reader's question spans three pages. The index sends the model to three URLs. The full file already has all three.
A site with a few dozen pages of the same quality does not gain much from it. The index handles that fine.
Who does not need it
Almost everyone else. Small business sites, blogs, portfolios, local service pages, SaaS marketing sites, and personal sites should skip it.
Two reasons. Your content is small enough that the index covers it. And a large file has a maintenance cost. Every time you change a page, the generated file goes stale until you regenerate it. For a docs site, that build step is already in your pipeline. For a blog, it is one more thing to forget.
If you are still working out what the field is called and how it differs from classic SEO, start with What Is SEO for AI Called? AEO, GEO, AIO Explained. The practitioner version is in The Small Business Guide to AEO.
The adoption problem worth being honest about
Neither file is a standard that AI companies have agreed to follow. No major operator has publicly committed to honoring llms.txt. OpenAI, Anthropic, and Google do not document it as a signal their crawlers follow. Several documentation platforms and developer tools do read it, which is why the format has traction at all.
Publishing the file costs a few minutes, so it is a cheap bet. It is not a ranking lever, and it is not a requirement. Treating it as either will cost you time you could spend on pages people actually read.
The distinction that matters is between publishing and being fetched. Your server sees which of the two happened. A crawler that downloads /llms-full.txt shows up in your logs. A crawler that ignores it does not. You can tell the difference only if you are looking.
That is where a crawler classifier earns its place. Kymo's AI crawler directory lists the agents it tracks, including GPTBot (OpenAI's training crawler), ClaudeBot (Anthropic's training crawler), and Bytespider (ByteDance). If a crawler requests your llms.txt or llms-full.txt, the fetch lands in the same dashboard as your pageviews and referrers. The same goes for ChatGPT-User (OpenAI's live fetcher), which fires when someone asks a question rather than when a model is being trained.
One thing that is not a crawler and never appears in a log: Applebot-Extended (a robots.txt control token, not a crawler). If you see it listed alongside real user agents somewhere, that source has it wrong.
How to tell whether any of it is working
Watch the log. Watch the dashboard. Both files sit at predictable URLs, so a fetch is easy to spot. What you are looking for is whether the crawlers you care about request either one.
Two groups behave differently. Training crawlers collect content for model development and send no traffic back. Indexers and live fetchers can lead to referral traffic down the line. The tradeoffs between allowing one group and blocking the other are in Should You Block AI Training Bots but Allow Answer Bots?. What you measure afterward is whether any of those visits turn into a session, which is what Does ChatGPT Send Traffic to Your Site? (How to Check) walks through.
If you want a rough sense of how visible your site already is to AI assistants, AI Visibility Score: What It Measures and What's a Good One explains what that number reflects.
For a site with no analytics installed yet, the free AI visibility check reads a URL from public signals and emails a report by magic link. The site does not need Kymo installed for that.
The short version
Publish llms.txt if you want a cheap bet on a format that some tools already read. Publish llms-full.txt only if you run a documentation site where the content is the product. Everyone else is maintaining a second file for no measurable return.
If you do publish either one, measure it. A file that gets fetched and a file that gets ignored look identical from the outside. Only your server can tell them apart.
See which crawlers actually show up
Install Kymo to see which AI crawlers fetch your site, which pages they read, and what traffic they send back, next to your normal human analytics. Solo is $9 a month or $90 a year for up to 10,000 events, and Studio is $29 a month or $290 a year for up to 100,000 events. Crawler tracking does not count against either limit, and setup is covered in the documentation.
Start free → 14-day free trial. No card required.