AI Search Readiness Checker
On September 15, 2026, Cloudflare changes the default AI-crawler policy for any site running ads: Training and Agent crawlers get blocked by default — and because Googlebot is a multi-purpose crawler (it indexes for Search and feeds Google's AI), an overly broad block can catch it too. Most site owners have never heard of this. Paste your URL below to check where you stand right now — takes about 3 seconds, no sign-up.
What this checks
- Search engine blocks — is Googlebot, Bingbot, or Applebot accidentally disallowed?
- AI crawler blocks — is GPTBot, ClaudeBot, Google-Extended, PerplexityBot, or similar blocked?
- Content-Signal directive — the new (2026) robots.txt standard for declaring search/ai-input/ai-train preferences separately
- llms.txt presence — the emerging standard that helps AI assistants understand your site faster
- Sitemap directive — whether your robots.txt points crawlers to your sitemap
Why this matters more than it used to
Google now shows an AI Overview on roughly half of US searches, and ChatGPT, Perplexity, and Apple Intelligence all actively crawl the web to answer questions in real time. If your site blocks these crawlers — even by accident, via an overly broad wildcard rule — you're invisible to a growing share of how people actually find things online. At the same time, Cloudflare's upcoming default change means some site owners will end up blocking AI crawlers without ever deciding to.
Fixing what this tool finds
Most issues have a one-line fix in robots.txt. If you want AI assistants to cite you in answers but don't want your content used to train models, add: Content-Signal: search=yes, ai-input=yes, ai-train=no to your robots.txt. If a specific AI crawler is blocked and you want to allow it, add an explicit User-agent block for that bot with Allow: /.