Does Blocking AI Crawlers Hurt Your Visibility?

The short answer to "does blocking AI crawlers hurt SEO" is: it depends on which ones you block, and you can only know the damage after you have measured what those crawlers were doing for you. Blocking a training crawler like GPTBot costs you nothing in direct traffic. Blocking a live fetcher like ChatGPT-User can remove you from an answer that a person is reading right now. Those are two different decisions with two different outcomes, and treating them as the same is how site owners make expensive mistakes.

The question itself is broken

"AI crawlers" is not one thing. It is three things, and they behave completely differently.

Live answer fetchers (ai_answer). When someone asks ChatGPT a question, the service may fetch your page in real time to ground the answer. The crawler is ChatGPT-User or Claude-User. This is a live fetch for one person's question. Blocking it means that person does not see your content in their answer.

Indexers (indexing). These bots build a search index ahead of time. OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot all do this. They crawl your site like Googlebot does, store what they find, and use it later when someone asks a question. Blocking them means you are absent from future answers, but there is no single moment where you can point to the lost visitor.

Training crawlers (training). GPTBot, ClaudeBot, Bytespider, CCBot and others download content in bulk to train foundation models. They send zero referral traffic. If they stopped visiting tomorrow, your analytics would not show a single lost visitor.

That is why the title question is unanswerable until you separate these three.

How much traffic do AI referrals actually send?

Not all AI crawlers send you visitors. Here is the direct comparison from Kymo's crawler registry:

The pattern is clear. Training crawlers take and give nothing back. Indexers and live fetchers can send people to your site. If you block the whole category because one training bot annoyed you, you also cut off the bots that could send paying customers.

The two kinds of blocking

There are two ways to block AI crawlers in your robots.txt file, and they have opposite effects.

Blocking the training crawlers: low risk

If you block GPTBot and ClaudeBot, you lose nothing in direct traffic. They do not send referrals. What you lose is influence over how those models understand your industry. That is a slow, diffuse loss, not a measurable drop in visitors. For many small business sites, blocking training crawlers is a reasonable privacy decision. There is no immediate cost.

One clarification about training crawlers and model behavior: blocking a training crawler does not remove your site from that operator's AI answers. OpenAI still has OAI-SearchBot and ChatGPT-User, which are separate tokens with separate rules. Anthropic still has Claude-SearchBot and Claude-User. If you block only GPTBot, your content can still appear in ChatGPT answers via the search bot. That is not a loophole, it is how the system is designed. The AI crawler directory explains each token's role.

Blocking the indexers and live fetchers: real cost

When you block OAI-SearchBot, PerplexityBot or Claude-SearchBot, you opt out of being indexed for future AI answers. When you block ChatGPT-User, Perplexity-User or Claude-User, you opt out of live answers entirely.

Traffic from AI referrals is still small for most sites, but it is growing. The question is whether you can afford to be absent from the channel that is replacing search. The AI visibility, explained doc walks through why a block on the indexing bot and a block on the live fetcher have different consequences even when they belong to the same company.

What blocking actually costs you

You cannot know what a block costs unless you measure the click it would have produced. That is the whole argument for tracking AI crawlers before you block them.

Here is the sequence that makes the decision clear:

1. A person asks an AI assistant a question relevant to your business.
2. The assistant's indexer already crawled your site and stored your page.
3. The live fetcher pulls your page to ground the answer.
4. The answer includes a citation to your site.
5. The person clicks through.

If you block step 2, you never reach step 3. If you block step 3, you are cited less often. But here is the thing: if your pages were not being cited anyway, blocking costs you nothing. If your pages are being cited every day, blocking removes a stream of visitors you cannot see in GA4 because GA4 does not categorize AI referrals properly. Kymo's Live Kymo demo dashboard shows what this data looks like when it is separated from human traffic.

Some people block AI crawlers because they worry about their content being used without permission. That is a legitimate position, but it is a value judgment, not a traffic calculation. If your concern is SEO, blocking everything is the worst of both worlds: you keep the training crawlers that give you nothing and lose the indexers that could send you people.

Make the decision after you measure, not before

The risk is not that you block a crawler. The risk is that you block a crawler without knowing what it was doing. That is the argument for measuring first.

Kymo classifies crawlers into three buckets automatically: ai_answer for live fetches, indexing for search-index builders, and training for bulk crawls. Each crawler's user agent is retained on its own event row, so you can audit the classification. You can see exactly which pages GPTBot, ClaudeBot, PerplexityBot and the rest are hitting. You can see which crawlers send you referral traffic and which ones never do.

The ClaudeBot vs Claude-SearchBot: What Each One Does on Your Site post shows how two Anthropic bots with similar names do completely different jobs. The robots.txt for AI Crawlers: A Copy-Paste Guide gives you the exact syntax for blocking one without blocking the other.

If your traffic is growing and you care about being found, check your crawler logs before locking the door. The data will tell you whether you are blocking a ghost or a salesman.

The rules change depending on who you ask

Some bots respect robots.txt differently. Perplexity-User is documented as exempt from robots.txt because a live fetch happens because a user asked for it. Cloudflare published evidence in August 2025 of undeclared Perplexity crawlers, which Perplexity disputed. That dispute is unresolved, and no one should cite the Cloudflare claim as settled fact.

The practical takeaway: robots.txt is not a force field. It is a request file. Most bots honor it, some interpret it differently, and a few have been accused of ignoring it. That is another reason to measure. You need to see what is actually hitting your server, not just what you asked to be blocked.

The strategic view

SEO is not dead. The mechanism of discovery is changing. People still search, but more and more of those searches happen inside AI assistants that do not show a blue link list. They show one answer with a few citations. The Small Business Guide to AEO covers how being cited in that single answer differs from ranking on page one.

The opportunity is not to fight AI crawlers. It is to understand which ones find you valuable and make it easier for them to do their job. If you are not being crawled, you cannot be cited. If you are not being cited, you are invisible to the people asking AI assistants what to buy.

What to actually block

Block what you know you do not want. Here is a sane default:

• Block training crawlers if you do not want your content used in model training. This costs you nothing in direct traffic.
• Do not block indexers or live fetchers unless you have looked at the data and decided the referral traffic is not worth it.
• Revisit the decision quarterly. The AI landscape changes fast. A crawler that sends zero traffic today may send meaningful referrals in six months.

Kymo's AI Visibility Score: What It Measures and What's a Good One explains how to tell whether your site is gaining or losing ground in AI referrals. That gives you a baseline. Without a baseline, you are guessing.

Measure before you block

Blocking AI crawlers is not inherently wrong. Doing it blind is. If you are going to make a decision that affects your visibility in the fastest-growing discovery channel, you need to know what you are giving up.

Kymo tracks both human visitors and AI crawlers in the same dashboard. You can see which crawlers visit, which pages they fetch, and which ones send you clicks. This is the same data that makes blocking an informed choice. It is simpler than GA4, which was not built for this problem, as Why Is GA4 So Complicated? (And What Small Sites Actually Need) points out.

Use the free AI visibility check at kymo.in/tools/ai-visibility-checker to see whether AI assistants currently reference your domain. It reads a URL from public signals only, so your site does not need Kymo installed. The report arrives by a magic link over email.

What to do next

Start tracking it. See which of these crawlers actually visit your site and which ones send you traffic. If you are thinking about blocking, check the data first. Block the training crawlers if you want, keep the indexers and live fetchers, and revisit the decision when the channel matures.

Do not block AI crawlers until you know what they are doing for you. Do block training crawlers if your only concern is privacy. If you want visibility in AI answers, the indexers and live fetchers are your channel. Treat them like it.

Start free → 14-day free trial. No card required.