robots.txt Generator

A standard robots.txt consists of one or more directive blocks structured by specific crawler targets:

• User-agent: Defines the target crawler for the rules that follow. Using User-agent: * establishes universal rules for all compliant crawlers. Target-specific rules can be set for dedicated engines, such as User-agent: Googlebot or User-agent: Bingbot.
• Disallow: Directs the bot not to crawl URLs matching the specified relative path prefix (e.g., Disallow: /admin/). Leaving the field empty (Disallow:) explicitly permits crawling of all pages.
• Allow: Explicitly permits crawling of a specific sub-path located inside a broader disallowed folder (e.g., Disallow: /wp-admin/ followed by Allow: /wp-admin/admin-ajax.php).
• Sitemap: Declares the absolute URL of your XML sitemap (e.g., Sitemap: https://example.com/sitemap.xml). You can specify multiple sitemaps to help search engines discover newly published pages. You can generate optimized sitemaps using our XML Sitemap Generator.

A frequent misconception in Technical SEO is treating Disallow as a de-indexing tool. Disallowing a URL only prevents search engines from crawling and downloading its contents. If the disallowed URL is linked internally or externally from other websites, Googlebot may still index the URL without fetching its HTML, displaying the URL in search results with a generic snippet stating "No information is available for this page".

If your goal is to completely remove a page from search results, you must allow Googlebot to crawl the page and output a <meta name="robots" content="noindex, follow"> tag or an X-Robots-Tag: noindex HTTP response header. You can configure and validate these tags with our Meta Tags Generator. Note that Google officially deprecated the proprietary Noindex: directive inside robots.txt in September 2019.

Under the official RFC 9309 standard, search engines evaluate path rules using specific length matching:

• Longest Matching Path Wins: The crawler compares all applicable rules and enforces the one with the longest character count.
• Allow Precedence on Tie: When an Allow rule and a Disallow rule match the exact same character length on a URL, the Allow directive wins.
• Wildcard (*): Matches zero or more arbitrary characters (e.g., Disallow: /*.pdf$).
• End-of-path Anchor ($): Signifies the strict end of a URL path, distinguishing /file.pdf from query parameter variations like /file.pdf?download=1.
• Case Sensitivity: Directives are case-sensitive. Disallow: /Admin/ will not block access to /admin/.

The Crawl-delay directive is not part of the RFC 9309 core standard and is treated differently across search engines:

• Googlebot: Completely ignores Crawl-delay inside robots.txt. Google dynamically adapts its crawl rate based on server responsiveness or custom settings in Google Search Console.
• Yandex: Has officially ignored Crawl-delay for main search indexing since February 2018. Crawl speed for Yandex is configured inside Yandex Webmaster.
• Bingbot, AhrefsBot & SemrushBot: Fully recognize and honor Crawl-delay (specified in seconds) to limit the frequency of requests to your server.

Modern websites often manage search engine crawlers alongside commercial SEO bots and AI data scrapers:

• Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider.
• AI & LLM Crawlers: GPTBot (OpenAI), CCBot (Common Crawl), ClaudeBot (Anthropic), Google-Extended (Gemini training).
• SEO Tool Crawlers: AhrefsBot, SemrushBot, DotBot.

You can audit bot accessibility and identify bot request behavior on your domain using our AI Bots & Crawler Checker.

Before deploying changes to production, verify your configuration with the following steps:

1. Direct HTTP 200 Check: Open https://yourdomain.com/robots.txt in your browser and confirm it loads as plain UTF-8 text.
2. Google Search Console: Use the robots.txt report and URL Inspection tool to confirm Googlebot has retrieved the latest version.
3. Technical SEO Health: Audit your site for crawl issues, broken links, and structured data integrity using our Broken Link Checker and Schema JSON-LD Validator.

WordPress Optimized Preset

Blocks the administrative backend while ensuring Googlebot can render dynamic AJAX content required for page layout.

E-Commerce / PrestaShop Preset

Prevents crawler index bloat from duplicate faceted search results and cart URLs while keeping product catalog URLs crawlable.

Blocking Specific File Formats with Regex Wildcard ($)

The end-of-path anchor ($) ensures all URLs ending strictly in .pdf are blocked from crawling.