Tool

robots.txt tester and user-agent checker (RFC 9309)

Polite scrapers check robots.txt before every crawl. Paste the file, pick a bot and see exactly which rule decides each URL.

robots.txt

User-agent or product token

Matched product token:

URLs or paths, one per line

User-agent analyzer

User-Agent string

Pick the group whose User-agent matches the crawler's product token (case-insensitive). Several matching groups are combined. If none matches, use the * group.
Within the group, the rule with the longest matching path wins. On a tie, Allow wins.
* matches any sequence of characters; $ anchors the end of the URL path.
Crawl-delay is not part of the standard but many crawlers honor it; polite bots should.

robots.txt is a request, not access control. It does not replace terms of service, authentication or rate limiting.