Field notes / Blog
Notes on being found
when machines do the searching
AEO, AI crawlers, and analytics a small team can actually read.
AI crawlers are the automated readers that fetch your pages for chatbots and answer engines. These posts identify the main ones by user agent and separate the crawlers that train models from the ones fetching a page to answer a live question. They also cover what robots.txt can and cannot control.
FEATURED · CrawlersAre AI Crawlers Blocked by Default?No, AI crawlers are not blocked by default within the standard WordPress, static site, or web server environments most small businesses use. Unless you, or a.
CrawlersHow to Tell If ChatGPT Is Reading Your Website
CrawlersWhich AI Crawlers Should You Allow?
CrawlersShould You Block AI Training Bots but Allow Answer Bots?
CrawlersDoes Blocking AI Crawlers Hurt Your Visibility?
CrawlersClaudeBot vs Claude-SearchBot: What Each One Does on Your Site
CrawlersEvery AI Crawler That Might Visit Your Site in 2026
CrawlersPerplexityBot: What It Crawls and Why It Matters
CrawlersShould You Block AI Crawlers? (It Depends Which Kind)
CrawlersGPTBot vs OAI-SearchBot: One Trains, One Cites
Crawlersrobots.txt for AI Crawlers: A Copy-Paste Guide
SHOWING 11 OF 11
See which AI engines already read your site.
Start free →14-day free trial. No card required.