Test robots.txt rules for Googlebot, GPTBot and other agents. See which rule allows or blocks a path, then download the text with comments and Sitemap preserved.
Practical example
01
Audit changes before publishing robots.txt.
02
Explain why a route is allowed or blocked.
03
Detect out-of-group rules and basic contradictions.
The most specific rule is allow: /private/public/.
Decision
ALLOW
Rule groups
1
Matching rules
2
robots.txt targets cooperative crawlers; It does not authenticate, does not protect content and does not guarantee deindexing.
Situation C · Access blocked
Your inputs
Contents of robots.txt
User-agent: *
Allow: /page
Disallow: /*.htm
User-agent
Googlebot
local route
/page.htm
Result
The most specific rule is disallow: /*.htm.
Decision
DISALLOW
Rule groups
1
Matching rules
2
robots.txt targets cooperative crawlers; It does not authenticate, does not protect content and does not guarantee deindexing.
Situation D · Access blocked
Your inputs
Contents of robots.txt
User-agent: *
Disallow: /café
User-agent
Googlebot
local route
/caf%C3%A9
Result
The most specific rule is disallow: /café.
Decision
DISALLOW
Rule groups
1
Matching rules
1
robots.txt targets cooperative crawlers; It does not authenticate, does not protect content and does not guarantee deindexing.
robots.txt is not authentication, security, or a deindexing command. The simulator implements a practical subset of RFC 9309 and does not query remote URLs.
robots.txt is not authentication, security, or a deindexing command. The simulator implements a practical subset of RFC 9309 and does not query remote URLs.