Start typing a directive to see the translation...
Symbol Cheat Sheet
*
Wildcard: Matches any sequence of characters.
$
End Anchor: Forces the match to the very end of the URL path.
/
Directory: Designates root or folder paths.
The Query String Pitfall
Blocking ? too broadly (e.g., Disallow: /*?) is a major risk. It can prevent bots from crawling vital CSS/JS files that use query parameters for cache-busting.
Does robots.txt help my "Crawl Budget"?
Yes. It prevents bots from wasting time on low-value pages like internal search results or admin panels, focusing their energy on your important pages instead.
Why do some bots ignore my robots.txt file?
Robots.txt is a "voluntary" protocol. While reputable search engines honor it, aggressive scrapers or rogue AI bots often bypass these rules entirely.