Crawl Control & AI
Check & Block AI Bots in robots.txt
Audit whether your website allows access to AI training crawlers and LLM search engines. Generate the perfect robots.txt code with recommended presets.
robots.txt AI Directives Generator
Select a recommended preset based on your copyright and visibility preferences:
How AI companies differentiate their web crawlers
Unlike traditional web crawlers (Googlebot or Bingbot), AI companies like OpenAI, Anthropic, and Google use specific user-agent strings to distinguish LLM model training from direct user search queries:
- OpenAI: Differentiates
GPTBot(ChatGPT model training) fromChatGPT-User(real-time browsing when requested by a user). - Anthropic: Uses
ClaudeBotfor general model training andClaude-SearchBotfor grounded search responses (RAG). - Google: Allows using
Google-Extendedin robots.txt to prevent Gemini and Vertex AI training on your content without impacting your traditional organic Google Search rankings.
Technical note: The robots.txt file is a standard crawling directive, not a hard security firewall. Major AI providers officially comply with these directives.