To check whether AI crawlers are blocked, confirm three things: (1) robots.txt allows each role, (2) requests under the search/citation crawler names return 200, and (3) the returned HTML contains the body. Checking robots.txt alone is not enough — WAF or CDN bot protection can return 403 based on the name alone. The ones you should not block are the search/citation crawlers that fetch pages for AI search answers.
AI Crawlers Fall Into Three Roles
"AI crawler" gets used as one label, but each company runs separately named crawlers for separate purposes. What to allow and what to block is decided by role.
| Role | Main names | robots.txt | If you block it |
|---|---|---|---|
| Training | GPTBot (OpenAI) / ClaudeBot (Anthropic) / Google-Extended (Google)Collects pages for training AI models | Obeys | Blocking is a policy choice. It does not remove you from AI search answers |
| Search & citation | OAI-SearchBot (OpenAI) / Claude-SearchBot (Anthropic) / PerplexityBot (Perplexity)Collects and indexes pages so they can appear in AI search answers | Obeys | Blocking keeps you out of — or makes you less visible in — that company's AI search answers (wording varies by company). Allow it if you want to be cited |
| User-initiated | ChatGPT-User (OpenAI) / Claude-User (Anthropic) / Perplexity-User (Perplexity)Fetches a page on the spot, e.g. when a user pastes a URL into a chat | Varies by company (see note below) | Some companies do not stop at robots.txt. Where it does apply, the page cannot be fetched even when a user asks for it |
User-initiated fetchers handle robots.txt differently by company. OpenAI says robots.txt rules may not apply to ChatGPT-User because the actions are user-initiated, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic, by contrast, says its bots — including Claude-User — honor robots.txt.
Each setting is independent. You can, for example, block GPTBot for training while allowing OAI-SearchBot for search. For robots.txt syntax itself, see how to write robots.txt.
Google's AI Overviews Are Governed by Regular Googlebot
Google has a token called Google-Extended, but Google states it controls use for things like Gemini training and has no impact on inclusion or ranking in Search. AI Overviews and AI Mode are part of Search, so your robots.txt rules for regular Googlebot are what apply.
So blocking Google-Extended does not take you out of AI Overviews, and blocking Googlebot takes you out of Search altogether. If you only want to limit how you appear in AI Overviews, Google points to snippet controls such as nosnippet and data-nosnippet.
The 403 That Happens Outside robots.txt
robots.txt is a request about whether a crawler may come; it does not physically stop anything. What actually stops requests is WAF or CDN bot protection in front of the server. If that layer returns 403 or 503 based on an AI crawler's name, the crawler never gets through — no matter what robots.txt says.
The catch is that the site looks normal in a browser, so owners do not notice. Even if you never touched your CDN settings, the provider's defaults can change. Here is what Cloudflare, a common example, has done.
- In July 2025, Cloudflare announced it would block AI crawlers without permission or compensation by default on new domains.
- In July 2026, it introduced purpose-based controls — Search, Agent, and Training — on all plans, including Free.
- It has announced that from September 15, 2026, new domains will block Training and Agent by default on pages that display ads, while Search remains allowed.
A policy of "blocking by default" and what an individual setting actually blocks are separate questions. Check in your own dashboard that search/citation crawlers are allowed.
Four Steps to Check by Hand
Open https://your-site/robots.txt and check each name in the table above for a Disallow that applies. The common miss is a blanket block under "User-agent: *" with no explicit allow. Confirm that the path you care about is allowed for each name.
curl -s https://example.com/robots.txtEven when robots.txt allows them, a WAF or CDN bot protection may return 403 or 503 based on the name alone. Fetch the page with each of the three search/citation names and compare against the 200 a normal browser gets.
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" https://example.com/A 200 is not enough if the body only appears after JavaScript runs — the page may be treated as empty. Search the fetched HTML for a sentence that should be in the body.
curl -s -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://example.com/ | grep -c "a sentence from the body"Steps 2 and 3 only claim a name. The final answer lives in your server or CDN access logs. Filter by name and confirm the requests were served with 200. You can verify they are genuine against the IP lists each company publishes.
grep "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -cNote: steps 2 and 3 only claim a name (User-Agent). Cloudflare says it identifies verified bots by a published IP list, reverse DNS, or a cryptographic signature — a request that merely uses the same name is not treated as verified. So with settings that let only verified bots through, the real crawler may pass even when your test gets 403. The reverse can happen too, which is why step 4's access logs are the final check.
Reached but Unreadable: Body Rendered by JavaScript
Google officially says Googlebot runs JavaScript to render pages. The crawler documentation from OpenAI, Anthropic, and Perplexity, however, does not say whether they execute JavaScript. A third-party measurement published in December 2024 (Vercel and MERJ) reported that GPTBot, ClaudeBot, PerplexityBot, and others were not executing JavaScript, while AppleBot and Gemini did render it.
If the body only appears after JavaScript runs, the page may be treated as empty even when it returns 200. For details, see whether SPAs are visible to AI crawlers.
What Changes Between Doing It by Hand and Using a Tool
Doing steps 1–3 for a single page means reading robots.txt, making requests under three search/citation names, and checking the body. The more pages you care about, the more you repeat it. Direbase's AIO Readiness Check does this from a single URL entry.
| Task | By hand | AIO Readiness Check |
|---|---|---|
| Reading robots.txt | Open the file and find the lines for each name | Judges allow/deny for 14 names across training, search, and user-initiated roles |
| Requests under each name | Run a command per name (3 for search/citation alone) | Requests under all 3 names at once and lists the status codes |
| Is the body there? | Open the fetched HTML and look for the body | Automatically judges whether the initial HTML contains the body |
| Comparing results | Line up three responses and bodies side by side | One screen, with a suggested direction for anything flagged |
| Repeating per page | Repeat all of the above for every page you care about | Just enter the next URL |
What gets shorter is the checking. Fixing robots.txt or WAF settings is done by you — or by the AI you work with. The tool never rewrites your site directly. And because the tool's requests are also name-only tests, step 4 — checking your access logs — still applies.
Check Your Own Site
Enter a URL to check robots.txt permissions by role, requests under the three search/citation names, and whether the initial HTML contains the body — all at once. No sign-up required.
Once You Know They Get Through
Reachability is the precondition; whether you get cited comes after. For the terminology and where to start given your site's stage, see what AIO, AEO, GEO, and LLMO mean. For how to shape pages that get cited, see how to optimize for AIO.
Related Tools
Sources (checked September 2026)
- OpenAI — Overview of OpenAI Crawlers
- Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity — Perplexity Crawlers
- Google Search Central — Google's common crawlers
- Google Search Central — AI features and your website
- Cloudflare — AI traffic options changelog (2026-07-01)
- Cloudflare Docs — Verified bots
- Cloudflare — Press release (2025-07-01)
- Vercel — The rise of the AI crawler (2024-12-17)
FAQ
What is the difference between GPTBot and OAI-SearchBot?▾
Is it safer to block all AI crawlers?▾
robots.txt allows them, so why don't AI crawlers show up?▾
Does blocking Google-Extended remove me from AI Overviews?▾
If curl with a spoofed User-Agent gets 403, is the real crawler blocked too?▾
Do AI crawlers execute JavaScript?▾
Visible in a Browser Does Not Mean Reachable by AI
Check robots.txt, WAF/CDN, and the initial HTML with a single URL entry (confirm with your access logs).
Run a free AIO Readiness Check
Written by
今井政和SEO Director / Frontend Developer
SEO Director with 20+ years of web industry experience. Creator of Direbase and the official WordPress plugin "ORECTIC SEO CHECK." Author of a book on web strategy inspired by Edo-era merchant principles.
@imai_directorRelated Articles
AI Search Is a Selection Layer | The Four-Layer GEO Model and the Layer That Breaks First
GEO works in one order only: Access, Orientation, Understanding, Quotability. Most sites fail where crawlers can't reach them, yet nearly all investment goes into content. A practitioner's reading of The GEO Readiness Manual.
Does llms.txt Work as a GEO Strategy? | CDN Log Data Analysis
Does deploying llms.txt get you cited in AI search? Analyzing CDN logs from ChatGPT, Perplexity, and Claude plus 300K domain studies to verify llms.txt GEO effectiveness.
Cited in AIO Despite No Organic Ranking | Verifying the Gap Between Search Position and AI Overview
Our site didn't appear on page 1 organically for 'seo score check free,' yet was cited first in AIO. We verify the gap between search rankings and AIO citations with real data and Ahrefs research.
What Are AIO, AEO, GEO, and LLMO? Differences and Where to Start
A breakdown of the four AI-search abbreviations by the audience each addresses, plus a measurement-based framework for judging your site's stage and deciding where to start.