Are AI Crawlers Blocked by Default?

No, AI crawlers are not blocked by default within the standard WordPress, static site, or web server environments most small businesses use. Unless you, or a tool you use, have taken steps to block them, they can visit your site like any other bot.

The short answer is no, but there is a catch

A plain server, a shared host, or a site builder will not stop ClaudeBot, GPTBot, or PerplexityBot from visiting. They will happily crawl your pages and your content will be indexed or used in answers, depending on which bot it is.

You will not see them in Google Search Console, unless you look in the right place, and they are not affected by your standard .htaccess or nginx.conf file unless you have put them there yourself.

But there is a managed-default case we need to address. Many small business owners do not have a plain server anymore. They use Cloudflare, Netlify, Vercel, or a web host that has added bot management features on behalf of their customers.

Cloudflare, for instance, has been known to tighten controls on AI crawlers. Other CDNs and hosts have started shipping with bot blocking in the default configuration, or they offer a one-click toggle that is turned on when you sign in and you never bother to switch it off.

If you are on a managed platform, check whether it has a bot management setting. Some hosts block unnamed or unidentified bots by default, and many AI crawlers are new enough that they do not have a long-established reputation, so they get lumped in with the unknown traffic and blocked.

So the honest answer to “are AI crawlers blocked by default?” is this: no, not by the web itself, but possibly yes, by your hosting platform or CDN. You need to check the settings of the service in front of your site, not just the server behind it.

Why this matters beyond a yes or no

If you have ever run a business that relied on Google for leads, you know that being found is not optional. SEO still matters, but AI answers are increasingly the place where customers get their recommendations.

If a hosting provider silently blocks an AI crawler, you lose visibility before you ever had a chance to measure it. You will not know why your brand name appears in fewer answers, or why your referral traffic from ChatGPT or Perplexity has dropped to zero.

This is where Kymo comes in. It is a hosted web analytics platform that tracks both human visitors and AI crawlers in one dashboard. It is cookieless, which means it does not need GDPR consent banners, and it does not store raw IPs or human user agent strings. AI crawler user agents are kept on the crawler's own event row, so the classification stays auditable.

The comparison table below breaks out the key behaviours of the main crawlers so you can see why a blanket block is the wrong move.

There is more detail published on the PerplexityBot page and the OAI-SearchBot (OpenAI's search index crawler) page if you want to dig into specific user agents.

The three categories of AI crawlers and why you cannot treat them alike

Kymo classifies crawlers into three categories, and each one demands a different response.

ai_answer crawlers are the live fetches. When someone asks ChatGPT a question and the assistant decides it needs fresh information from a website, it uses ChatGPT-User to fetch that page in real time. The same goes for Claude-User and Perplexity-User. These fetches are important because they happen right at the moment a potential customer is asking for a recommendation.

indexing crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot are building search indexes ahead of time. They are not tied to one user query. They work like Googlebot and prepare content so the operator can reference it later.

training crawlers like GPTBot and ClaudeBot gather content to help build foundation models. There is a common pattern in the comparisons between Google-Extended (a robots.txt control token, not a crawler) and the various AI crawler tokens, and it is worth understanding.

Blocking a training crawler does not remove your site from that operator's AI answers. Training, indexing, and live fetches use separate tokens, so you can block GPTBot and still be cited by ChatGPT if OAI-SearchBot and ChatGPT-User are allowed.

What is actually crawling your site

You cannot improve what you do not measure. The Live Kymo demo dashboard shows how this works in practice. It demonstrates the event stream, the real-time map, and how AI crawler visits are flagged separately from human traffic.

If you are curious about what an AI visibility report looks like, Kymo offers a free AI visibility check at kymo.in/tools/ai-visibility-checker. No account is needed. It reads a URL from public signals only, and because it works from public data, the site does not need Kymo installed to run the check. The report arrives by a magic link over email. It is a useful way to see how search engines and AI assistants currently view your pages.

The managed-block problem in practice

Let's say you run a specialty retail site and you are on a popular managed WordPress host. The host enabled a bot firewall by default six months ago. One of the rules caught the AI crawlers, because they have unfamiliar user agents.

Meanwhile, your Google traffic is flat, but your referral traffic from Google has always been your main channel. You had no idea that you were being hidden from AI answers until you ran a free visibility check that showed which crawlers were hitting your homepage.

This is where the diagnostic value of Kymo shows. It tells you if a crawler visited, which category it fell into, and whether it was blocked before it got what it came for. Without that data, you are guessing.

How to decide what to block

The default position should be to allow indexing and ai_answer crawlers. They can send referral traffic. The exceptions are training crawlers like GPTBot, ClaudeBot, and Bytespider (ByteDance), which take content and do not send any traffic back.

That does not mean you must block all training crawlers. Some site owners care about being in AI answers and do not mind their content being used for training. Others block them all and sleep better at night. Both positions are defensible.

What is not defensible is letting a default firewall make that decision for you without telling you it has done so. Control should sit with the site owner.

If you decide to block a training crawler, it will not stop you from appearing in that operator's answers. The indexing crawler and the live fetcher are still allowed, unless you block them too.

Robots.txt is not the only place this happens

Robots.txt is the visible layer, but it is not the only layer. The managed-default case is about providers making proactive choices. Cloudflare published evidence in August 2025 of undeclared Perplexity crawlers, which Perplexity disputed, and it is worth noting that Perplexity has no training crawler at all. Its two agents are retrieval only: PerplexityBot indexes and Perplexity-User fetches live. There is no third agent and no "service" variant of PerplexityBot.

Perplexity-User is documented as exempt from robots.txt because a user asked for the live fetch. That is a disputed finding, and the cloudflare evidence about undeclared crawlers is in the same disputed category, but it tells you the ecosystem is messy.

Look at the data

If you want to know exactly what is happening, set up a session with Kymo and look at the "AI Crawlers" section. You will see a list of every AI-related user agent that has triggered an event, categorised as ai_answer, indexing, training, or other. You can match them against the crawler registry at /crawlers to check what each one does.

The OAI-SearchBot (OpenAI's search index crawler) page, for example, confirms that OAI-SearchBot is indexing and can lead to referral traffic. That is the one you want to allow if you want to be surfaced in ChatGPT search. The Google-Extended (a robots.txt control token, not a crawler) page explains that it is a control token, not a crawler. It has no user agent and will never appear in a log. Disallowing it only controls how content that Google has already crawled may be used.

The status of AI crawlers in 2025

Every major provider has at least one crawler with a robots.txt token. OpenAI has GPTBot for training and OAI-SearchBot for indexing. Anthropic has ClaudeBot for training and Claude-SearchBot for indexing. Meta has multiple agents, including FacebookBot for training and Meta-WebIndexer for indexing. Microsoft has bingbot, which is an indexing crawler.

The fact that these tokens exist does not mean they are honoured by default. It also does not mean the provider will respect your robots.txt preferences. Compliance is a gray area.

You can read the practical side of this in our How to Optimize for AEO (7 Steps, No Consultant) guide, and the more technical side in PerplexityBot: What It Crawls and Why It Matters. If you want to write pages that AI assistants tend to quote, the AEO Content: What to Write So AI Engines Quote You post is a good starting point.

The The Small Business Guide to AEO walks through the same ideas in a longer form.

A final word on defaults and data

Default configurations benefit the platform, not you. They are set in the interest of the host or the CDN, to reduce their support load and their bandwidth costs. They were not designed to optimise your visibility in AI answers.

If your hosting provider blocks AI crawlers by default, your options are to disable that rule or add an exception for the crawlers you care about. If your hosting provider does not block them, you still need a way to see who is visiting so you can measure whether AI visibility is improving.

Kymo is a diagnostician, not a strategist. It measures and reports. It does not write content, do outreach, or perform AEO for you. What it does is show you, in one dashboard, which AI assistants fetched your pages, which pages they fetched, and whether that has turned into clicks.

Look at the data, decide what you actually want to block, and stop letting a default setting make that decision for you.

Start seeing the bots that matter

Start tracking it if you want to see which of these crawlers actually visit your site, or read the documentation on how Kymo classifies them. You can also run the free AI visibility check to see your current status from public signals; no installation needed, and the report lands in your inbox.

Start free → 14-day free trial. No card required.