Product guide

How it works

Check which URLs each bot can crawl, find out the exact rule that decides access, and download a corrected robots.txt before publishing.

Method

01
Separates user-agent groups and Allow/Disallow rules.
02
Select the most specific group and apply the longest path match.
03
Shows the applied rule and warnings. The download preserves the entered text, including comments, Sitemap and line endings.

Practical example

Situation A · Access blocked

Your inputs

Contents of robots.txt: User-agent: * Disallow: /private/ Allow: /private/public/ Sitemap: https://example.test/sitemap.xml
User-agent: Googlebot
local route: /private/report.html

Result

The most specific rule is disallow: /private/.

Decision: DISALLOW
Rule groups: 1
Matching rules: 1

robots.txt targets cooperative crawlers; It does not authenticate, does not protect content and does not guarantee deindexing.

Situation B · Access allowed

Your inputs

Contents of robots.txt: User-agent: * Disallow: /private/ Allow: /private/public/ Sitemap: https://example.test/sitemap.xml
User-agent: Googlebot
local route: /private/public/report.html

Result

The most specific rule is allow: /private/public/.

Decision: ALLOW
Rule groups: 1
Matching rules: 2

robots.txt targets cooperative crawlers; It does not authenticate, does not protect content and does not guarantee deindexing.

Situation C · Access blocked

Your inputs

Contents of robots.txt: User-agent: * Allow: /page Disallow: /*.htm
User-agent: Googlebot
local route: /page.htm

Result

The most specific rule is disallow: /*.htm.

Decision: DISALLOW
Rule groups: 1
Matching rules: 2

robots.txt targets cooperative crawlers; It does not authenticate, does not protect content and does not guarantee deindexing.

Situation D · Access blocked

Your inputs

Contents of robots.txt: User-agent: * Disallow: /café
User-agent: Googlebot
local route: /caf%C3%A9

Result

The most specific rule is disallow: /café.

Decision: DISALLOW
Rule groups: 1
Matching rules: 1

robots.txt targets cooperative crawlers; It does not authenticate, does not protect content and does not guarantee deindexing.

robots.txt is not authentication, security, or a deindexing command. The simulator implements a practical subset of RFC 9309 and does not query remote URLs.

Download result · JSON

Responsible scope

Sources and scope

robots.txt is not authentication, security, or a deindexing command. The simulator implements a practical subset of RFC 9309 and does not query remote URLs.

RFC 9309

Try the tool