llms.txt vs robots.txt vs sitemap.xml: What Each One Does
- robots.txt is a permission file. Every major crawler follows it. It says what may be fetched.
- sitemap.xml is a discovery file. It lists URLs you want found. It grants nothing and blocks nothing.
- llms.txt is a guide. It was proposed in 2024 and adoption is partial. Several documentation platforms and developer tools read it. No major AI operator has publicly committed to honouring it.
- All three can sit at your site root at the same time. They do not overlap.
- Publishing llms.txt is not the same as a crawler fetching it. Only your server sees which one happened.
robots.txt is a permission file, sitemap.xml is a discovery file, and llms.txt is a guide with partial adoption. They live at the root of the same site and do three separate jobs.
| File | Job | Who reads it | What it cannot do |
|---|---|---|---|
| robots.txt | Permission | Every major crawler: Googlebot, bingbot, GPTBot, ClaudeBot, PerplexityBot and the rest | It does not list pages or send crawlers anywhere |
| sitemap.xml | Discovery | Search engines and indexers that choose to fetch it | It cannot allow or deny access to anything |
| llms.txt | Guide | Some documentation platforms and developer tools. No major AI operator has committed to honouring it | It cannot make a crawler read your content |
robots.txt is the only one with real authority
It is the oldest of the three and the only one a crawler is expected to check before it fetches anything. Disallow a path and a compliant crawler skips it. That is the whole file.
What robots.txt does not do is describe your site. It has no list of your pages. It cannot point a crawler at something new.
For AI traffic, the token is what matters. GPTBot is OpenAI's training crawler. ClaudeBot is Anthropic's training crawler. Bytespider is ByteDance's. Block the training token and the operator's indexer and live fetcher are still allowed in, because they are separate tokens. Blocking one does not remove you from that operator's AI answers.
If you want the actual copy-paste rules, we wrote a separate walkthrough for robots.txt for AI crawlers. You can also look up any single agent by name, for example ClaudeBot (Anthropic's training crawler) or Bytespider (ByteDance), in the AI crawler directory.
sitemap.xml is about discovery, not permission
A sitemap is a list of URLs, usually with a last modified date. You submit it to Google Search Console or Bing Webmaster Tools, and crawlers use it to find pages they might otherwise miss.
It grants nothing. A URL in your sitemap can still be disallowed in robots.txt. A URL missing from your sitemap can still be crawled if something links to it.
Sitemaps mainly help indexing agents. They do little for live fetches. When someone asks ChatGPT a question and ChatGPT-User reaches your page in that moment, the sitemap did not cause it. The question did. That is why a sitemap alone will not tell you whether AI assistants are reading your pages.
llms.txt is a 2024 proposal with partial adoption
llms.txt was proposed in 2024 as a markdown file at your site root. It summarises your site for language models: what you do, which pages matter, links to longer documentation. The idea is that a model reads the summary instead of guessing from HTML.
Adoption is partial. Several documentation platforms and developer tools do read it. No major operator has publicly committed to honouring llms.txt. OpenAI, Anthropic and Google do not document it as a signal their crawlers follow.
That does not make it worthless. Publishing one costs a few minutes. Treat it as a cheap bet. It is not a ranking lever and it is not a requirement. If you skip it, nothing breaks.
The part people miss: publishing the file is not the same as a crawler fetching it. Only your own server sees which one actually happened. A file sitting at /llms.txt that nothing has requested since the day you uploaded it is doing nothing at all.
A site can carry all three, and they do not conflict
You can allow OAI-SearchBot in robots.txt, list your pages in sitemap.xml, and publish an llms.txt summary. All three coexist at the root.
You can also disallow GPTBot in robots.txt while keeping a sitemap and an llms.txt. The sitemap does not override the disallow. Neither does llms.txt. Permission wins, because it is the only file a crawler checks before fetching.
The clean mental model is a sequence. robots.txt answers "may I". sitemap.xml answers "where should I look". llms.txt answers "here is the short version". Different questions, different files.
What you still cannot see from these files
None of the three files tells you what actually visited. That information lives on your server, in the request log, one row per fetch.
That is the gap most sites sit in. They block a training crawler, publish a sitemap, maybe add an llms.txt, and have no idea which of those moves changed anything. The live Kymo demo dashboard shows what that looks like: crawler events sorted by category, next to normal human traffic, with no login required. Kymo identifies visitors with cookieless identity, so there is no cookie banner and no stored IP.
Two things worth measuring once you have that visibility. First, which AI agents send referrals back to your site, and whether the traffic converts. If you tag those links, UTM parameters will show you the source. Second, which pages get fetched live versus crawled for an index. Local pages behave differently from national ones here, and Local Businesses in AI Search covers that split in more detail.
If you want a wider list of agents to watch for, we keep an updated list of AI crawlers. It is long, and it grows.
Do these three things in this order
Get robots.txt right first. It is the only file with real authority, and a mistake there can lock out an indexer you wanted. Add or fix your sitemap second. It is cheap and search engines still use it. Publish llms.txt last, or never. It is a bet. It is not a foundation.
If you are a solo builder or a small team with a site that used to rank well, the priority is permission plus measurement. Blocking the wrong token by accident costs you AI referrals you never see. And if you want the fuller playbook on getting found inside AI answers, the small business guide to AEO is the next read.
See which crawlers actually show up
You can guess which of these files gets read. You can also look. Kymo shows every AI crawler that hits your site by name and category, next to your normal analytics, and the free AI visibility check at kymo.in/tools/ai-visibility-checker reads a URL from public signals so your site does not need Kymo installed, with the report arriving by email. Start tracking it or read the documentation on how Kymo classifies each one.
Start free → 14-day free trial. No card required.