robots.txt Generator

Complete Guide to robots.txt & RFC 9309 Standard

The robots.txt file is a standardized plain-text file placed at the root of a domain (e.g., https://example.com/robots.txt) that instructs web search crawlers and automated bots which parts of a site they should or should not crawl. In 2022, the Internet Engineering Task Force (IETF) codified the protocol as RFC 9309 (Robots Exclusion Protocol), establishing clear rules for directives, pattern matching, precedence, and parsing.

1. Crawling vs. Indexing: A Critical Distinction

A common misconception in Technical SEO is confusing crawling with indexing. A Disallow rule prevents search engines from fetching/crawling a URL. However, if that disallowed URL is linked from other external or internal pages, Google may still index the URL without crawling its content. If your goal is to prevent a page from appearing in search results, you must allow Googlebot to crawl the page and serve a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP response header. Note that Google does not support a Noindex: directive inside robots.txt (deprecated since 2019).

2. Directive Precedence & Path Matching (RFC 9309 §2.2.2)

Under RFC 9309, crawlers calculate the length of matching characters for all applicable rules. The longest matching path pattern wins. In the event of a tie where an Allow directive and a Disallow directive match the exact same character length on a URL, the Allow directive takes precedence.

• Prefix matching: Disallow: /admin/ blocks /admin/settings, /admin/users, etc.
• Wildcard (*): Disallow: /*.pdf matches any path containing .pdf
• End-of-path anchor ($): Disallow: /*.pdf$ matches /doc.pdf but NOT /doc.pdf?preview=1
• Case sensitivity: Paths are strictly case-sensitive. Disallow: /Admin/ does not block /admin/.

3. Crawl-delay Realities: Google & Yandex vs. Other Engines

The Crawl-delay directive is not part of the core RFC 9309 standard. Google completely ignores Crawl-delay in robots.txt (Googlebot crawl rate is handled dynamically or via Search Console). Similarly, Yandex has ignored Crawl-delay for main web indexing since February 2018 (crawl rate is configured in Yandex Webmaster). However, search engines like Bingbot and crawlers like AhrefsBot and SemrushBot recognize and respect Crawl-delay to throttle request frequency.

4. robots.txt is NOT a Security or Access-Control Mechanism

robots.txt is a voluntary advisory protocol for well-behaved crawlers. It is publicly visible to everyone on the Internet. Never list confidential folders, private backend paths, or secret tokens in robots.txt, as malicious bots and attackers will specifically scan robots.txt to discover hidden endpoints. To secure sensitive resources, use server authentication (HTTP Basic Auth, OAuth, session logins, or firewall IP whitelists).