When your site blocks AI crawlers

What this check means

To answer questions, AI assistants send crawlers to read pages on the live web. This check looks at your robots.txt file, and also whether your server itself turns those crawlers away, and flags when either blocks them. If they're blocked, your pages can't appear in the assistant's answers, no matter how good they are.

How to read it

If you see this in your Actions or SEO scan, your robots.txt is blocking the crawlers AI assistants use to fetch pages. It's usually accidental, the result of a broad "block all bots" rule or an over-eager firewall.

What to do

• Review your robots.txt and make sure it doesn't disallow the crawlers AI assistants use to read your site.
• If a firewall or bot-protection tool sits in front of your site, check that it isn't blocking them either.
• Note that the crawlers used to train AI models are separate from the ones that fetch pages to answer questions. Blocking training is a valid choice on its own; it's the answer-time crawlers you want to keep allowed.

Once they're allowed again, your pages become eligible to appear in AI answers. Re-run a scan to confirm the flag clears.