What an llms.txt File Is, and Whether AI Crawlers Fetch It

An llms.txt file is a plain markdown file at the root of a site that tells an AI model what the site is for and which pages matter most. That is the whole concept. A text file at /llms.txt, written in markdown, aimed at machines that may or may not read it.

If you have spent years tuning pages for Google, the shape feels familiar. A summary file, a bit like a sitemap, except the audience is a language model instead of a ranking engine. The catch is that the audience never agreed to read it.

The format is short and easy to get right

One H1 with the site name. One optional blockquote underneath, a sentence or two on what the site does. Then H2 sections, each holding a list of markdown links.

That is the whole spec. No XML, no schema, no build step.

A section called "Docs" lists your documentation. A section called "Blog" lists recent posts. A section called "Optional" holds pages a model can skip. The convention stays loose because nobody owns it, which is both the appeal and the weakness. Nothing validates the output, and no registrar ships it.

You can write one by hand in an editor. Kymo's free llms.txt generator and validator drafts the structure from a URL and checks the result, which beats re-reading a spec that keeps moving.

What the file looks like in practice

# Acme Widgets

> Acme sells industrial widgets and publishes a public widget spec.

## Main pages

- [Product overview](https://example.com/product)
- [Pricing](https://example.com/pricing)

## Docs

- [Widget spec](https://example.com/spec)
- [Compatibility list](https://example.com/compat)

## Optional

- [Changelog](https://example.com/changelog)

Each line is a markdown link with a short note after the colon. Models that read the file get a flat map of the site instead of guessing from HTML. Models that do not read it get nothing, and there is no error message either way.

No major AI operator has committed to honouring it

This is the part vendor pages skip. Adoption is partial.

OpenAI does not document llms.txt as a signal that GPTBot (OpenAI's training crawler) follows. Anthropic does not document it. Google does not. None of the three has publicly committed to reading the file, and none has said it ignores it either. Silence is where things sit.

Several documentation platforms and developer tools do read llms.txt today. That is real usage and a narrow slice of the audience you care about. The readers you want, the assistants answering questions about your category, have not signed on.

So when a post says llms.txt "tells ChatGPT what your site is about", treat it as marketing. It tells a model that only if the operator chose to look.

One more thing worth pinning down: blocking a training crawler in robots.txt does not remove your site from that operator's AI answers. Indexing agents and live-fetch agents run on separate tokens. llms.txt sits on top of an already tangled picture and does not untangle it.

How llms.txt compares to the files you already have

The older two earned trust over years. llms.txt is asking for it now, with no history and no enforcement.

The naming around this gets messy. You will see llms.txt filed under AEO, sometimes under AIO, and the terms get swapped freely. AEO vs AIO: Which One Are People Talking About? untangles that. If you want the short definition first, What Is AEO in Marketing? (Plain-English Answer) covers it in a few paragraphs.

Publishing the file is not the same as a crawler fetching it

This is the distinction that matters, and most posts blur it.

You can publish /llms.txt today. Whether PerplexityBot, ClaudeBot, GPTBot or any other agent fetches it is a separate fact. The only place that fact shows up is your own server logs. A file on disk proves nothing about who read it.

GA4 will not help you here, since it does not name AI crawlers at all. Seeing this requires a server-side receiver that classifies user agents into three categories: ai_answer for a live fetch made because one person asked a question right now, indexing for crawlers building a search index ahead of time, and training for bulk crawls gathering model data. Each category means something different for your site.

How to See AI Traffic in Your Analytics walks through what those hits look like once they arrive. The Kymo documentation covers how the classification works, and Reading the AI report explains the AI Visibility view: which of your pages assistants fetched, the likely prompts behind those fetches, and confirmed click-throughs from AI referrals.

Publish an llms.txt file. It costs a few minutes, several documentation platforms read it, and the worst case is a text file nobody fetches. Do not treat it as a ranking lever, and do not let it replace the work that actually moves numbers. Crawler access, clean page structure, and pages that answer a question well still do the heavy lifting. If you are still working out how much of your effort belongs to classic SEO, SEO vs AEO: Do You Need Both in 2026? covers the split, and The Small Business Guide to AEO covers the rest of the checklist.

Then check the logs. Until you see a fetch, the file is a bet, not a result.

See which crawlers actually show up

Point Kymo at your site and start tracking it to see which AI crawlers visit your pages and which send traffic back, alongside normal human analytics. AI crawler hits do not count against your event limit, so watching them adds nothing to your bill.

Start free → 14-day free trial. No card required.