Check & Block AI Bots in robots.txt

How AI companies differentiate their web crawlers

Unlike traditional web crawlers (Googlebot or Bingbot), AI companies like OpenAI, Anthropic, and Google use specific user-agent strings to distinguish LLM model training from direct user search queries:

• OpenAI: Differentiates GPTBot (ChatGPT model training) from ChatGPT-User (real-time browsing when requested by a user).
• Anthropic: Uses ClaudeBot for general model training and Claude-SearchBot for grounded search responses (RAG).
• Google: Allows using Google-Extended in robots.txt to prevent Gemini and Vertex AI training on your content without impacting your traditional organic Google Search rankings.

Technical note: The robots.txt file is a standard crawling directive, not a hard security firewall. Major AI providers officially comply with these directives.