Method
How it works
- Separates user-agent groups and Allow/Disallow rules.
- Select the most specific group and apply the longest path match.
- Displays the winning rule, warnings, and a downloadable normalized robots.txt.
robots.txt · crawler simulator
CrawlerControl
Check which URLs each bot can crawl, find out the exact rule that decides access, and download a corrected robots.txt before publishing.
Free robots.txt simulator for Googlebot, GPTBot and other crawlers, with conflicts, winning rule and normalized file.
No accountLocal processingVisible assumptions
01 · Your inputs
Paste your robots.txt and try a route
Edit assumptions and review the result instantly.
The file is read locally.
Don't paste secrets; robots.txt is public.
No network request is made.
The record is deliberately short and versioned.
02 · Result
Access, winning rule and conflicts
Local calculation readyResult updated with your data
The scenario has been recalculated with the visible data. Review metrics, detail, and limits before using or downloading the result.
DISALLOWDecision
3Groups
1Coincidences
- User-agent tested
- Google-Extended
- local route
- /training/private/report.html
- Winning rule
- disallow:/training/
- Conflicts
- 1
- There are conflicting Allow/Disallow rules in: /.
- robots.txt targets cooperative crawlers; It does not authenticate, does not protect content and does not guarantee deindexing.
When to use it
Use cases
- Audit changes before publishing robots.txt.
- Explain why a route is allowed or blocked.
- Detect out-of-group rules and basic contradictions.
Limits
Sources and scope
robots.txt is not authentication, security, or a deindexing command. The simulator implements a practical subset of RFC 9309 and does not query remote URLs.
Privacy by design
Privacy →Your data stays on this device
The tool calculates and creates files in your browser. It needs no account and does not send your input to Chapa Lab.