Field notes / Blog
Notes on being found
when machines do the searching
AEO, AI crawlers, and analytics a small team can actually read.
AI crawlers are the automated readers that fetch your pages for chatbots and answer engines. These posts identify the main ones by user agent and separate the crawlers that train models from the ones fetching a page to answer a live question. They also cover what robots.txt can and cannot control.
FEATURED · CrawlersWhat an llms.txt File Is, and Whether AI Crawlers Fetch ItAn llms.txt file is a plain markdown file at the root of a site that tells an AI model what the site is for and which pages matter most. That is the whole.
Crawlersllms-full.txt Explained: What It Is and When You Need One
Crawlersllms.txt Best Practices, Straight From the Spec
CrawlersReal llms.txt Examples, Scored Against the Spec
Crawlersllms.txt vs robots.txt vs sitemap.xml: What Each One Does
CrawlersAre AI Crawlers Blocked by Default?
CrawlersHow to Tell If ChatGPT Is Reading Your Website
CrawlersWhich AI Crawlers Should You Allow?
CrawlersShould You Block AI Training Bots but Allow Answer Bots?
CrawlersDoes Blocking AI Crawlers Hurt Your Visibility?
CrawlersClaudeBot vs Claude-SearchBot: What Each One Does on Your Site
CrawlersEvery AI Crawler That Might Visit Your Site in 2026
SHOWING 12 OF 16OLDER →
See which AI engines already read your site.
Start free →14-day free trial. No card required.