AI Crawler Checker: See Which AI Bots Can Crawl Your Website
Check, bot by bot, which AI crawlers your robots.txt allows or blocks, and create a starter llms.txt in the same place.
Many website owners do not know which AI bots can reach their site. Hosting companies, CDNs, security plugins and SEO plugins sometimes add AI-blocking rules on their own โ or leave everything open. The free SeoSpidie AI crawler checker reads your robots.txt and tells you, bot by bot, what is allowed.
If you have not decided what you want yet, read should I block AI crawlers first.
Step 1: Enter your website address
Type your domain, such as example.com. The checker fetches your robots.txt file from the root of the site.
Step 2: Read the results bot by bot
The checker lists the main AI crawlers โ such as GPTBot, ClaudeBot, PerplexityBot, CCBot and the Google-Extended token โ and shows whether each is allowed or blocked by your current rules. Look for surprises:
- Blocked when you wanted visibility โ a plugin or host may have added a blanket block.
- Allowed when you wanted protection โ no rules exist yet for training crawlers.
- A rule for one bot that accidentally affects others โ group order and wildcards can be tricky.
Step 3: Change the rules if needed
To block or allow a bot, edit robots.txt with a group for that bot's name. The robots.txt generator builds the rules without syntax mistakes. Remember that a bot with its own group ignores the general User-agent: * group, so repeat any shared rules. Then run the checker again to confirm.
Step 4: Create a starter llms.txt
The same tool generates a starter llms.txt file from your site name, address, a one-line summary and your key pages. Edit the descriptions in your own words and upload it to your site's root folder. Read what an llms.txt file is to understand what it can and cannot do.
Remember what robots.txt can and cannot do
- Reputable AI companies follow robots.txt; some unknown scrapers do not.
- Rules affect future crawling only, not content collected earlier.
- A firewall or CDN can block bots too โ if a bot is allowed in robots.txt but still cannot reach you, check your security settings.
Check visibility, not just access
Being crawlable is step one. To be cited in AI answers, pages also need clear, quotable content. Test pages with the GEO checker, and measure results by tracking ChatGPT traffic in GA4.
Example: rules that surprise people
A site owner believed their site was open to all AI tools. The checker showed GPTBot and ClaudeBot as blocked. The cause was a security plugin that had added AI-blocking rules to robots.txt automatically during an update. Another site found the opposite: a hosting control panel had removed a custom robots.txt, leaving everything allowed when the owner wanted training crawlers blocked.
Run the checker after any plugin, theme, hosting or CDN change, not just once.
Related guides
More free tools: browse all our free AI search tools, no sign-up needed.
Frequently asked questions
Which AI crawlers does the checker look at?
The major ones, including GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended, plus others commonly found in robots.txt files.
Does the checker change my robots.txt?
No. It only reads it. You make changes on your own server.
Why is a bot shown as allowed when I blocked it?
Check the spelling of the bot name, the group order and that the file is at the root of the exact domain you tested.
Is the AI crawler checker free?
Yes, with no sign-up.