Free tools › robots.txt AI-bot checker
Free tool · $0, no LLM · no signup
robots.txt AI-bot checker
Enter a domain and CitePulse fetches its robots.txt and reports, bot by bot, whether the major AI crawlers are allowed or blocked. If you are accidentally blocking the assistants that drive citations, you get a corrected robots.txt to copy.
Email me the report and a fixed robots.txt
Want to know if AI assistants actually cite you, not just whether they can? Run the free CitePulse audit →
Frequently asked questions
Which AI bots does the checker test?
It checks the major AI crawler user-agents: OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User; PerplexityBot and Perplexity-User; Anthropic's ClaudeBot; Google-Extended and Google-CloudVertexBot; Applebot-Extended; plus training/scraper bots like CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot and Meta-ExternalAgent.
Should I allow all AI bots?
Allowing the assistant-facing bots (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, OAI-SearchBot) is generally what lets AI assistants read and cite your content. Pure training-corpus scrapers such as CCBot or Bytespider are optional — some owners block them while still allowing the assistant bots. The tool marks which are which.
Does this tool cost anything or use AI?
No. It only fetches your public robots.txt and parses it deterministically. There is no AI model involved and no cost — it is free with no signup.