How does Google's robots.txt matching algorithm work?

Google finds all rules that match the URL (where the URL starts with the rule path), then selects the most specific one (longest path). If an Allow and a Disallow match at the same specificity, Allow wins. User-agent-specific groups override the wildcard group.

Does robots.txt prevent indexing?

No. Robots.txt blocks crawling, not indexing. Google can still index a URL it has never crawled if other sites link to it or if it was previously in the index. Use a noindex meta tag to prevent indexing.

What happens if my robots.txt file is missing?

If Google cannot fetch robots.txt (404 or connection error), it assumes the file has no restrictions and crawls the entire site. A 500 error causes Google to temporarily pause crawling until the file is accessible again.

What is the difference between Disallow: / and Disallow: /*?

They are treated identically by Google. Both block all URLs for that user-agent. The wildcard * at the end of a path in robots.txt is a Google extension and matches any continuation of the path. Tracking this metric alongside conversion data gives a more complete picture of how changes affect actual business outcomes.

Should I block CSS and JavaScript files?

No. Google needs to render your pages to understand their content, which requires access to CSS and JavaScript files. Blocking these files can prevent Google from understanding your page layout and content correctly.

How do I test my robots.txt with multiple URLs at once?

Use the batch mode: enter one URL per line in the multi-URL textarea. The tool runs the matching algorithm against all of them and returns a results table showing allowed or blocked status for each URL and the specific rule that matched.

robots.txt Tester

The Robots.txt Tester lets you paste your robots.txt content and test any URL against any user-agent to see whether crawlers are allowed or blocked. The tool applies Google's exact matching algorithm, including longest-path precedence and Allow-over-Disallow tie-breaking rules.

robots.txt content

Syntax preview

User-agent: *
Disallow: /admin/
Allow: /public/
 
User-agent: Googlebot
Disallow: /staging/

URL to testUser-agent to test

Allowed

Matched: No matching rule

Matched group: googlebot

Test multiple URLs (one per line)

Batch results

URL	Result	Matched rule
/admin/page	Allowed	No matching rule
/public/page	Allowed	No matching rule
/staging/home	Blocked	Disallow: /staging/

Google robots.txt spec

This tester evaluates only the rules you paste. It does not fetch robots.txt from your live domain.

How this tool works

The robots.txt tester validates whether a specific URL is accessible or blocked for a named user-agent according to the rules in your robots.txt, using the same rule-precedence logic that compliant crawlers apply. The parsing order: more-specific user-agent blocks override more-general ones; within a block, when both Allow and Disallow rules match a URL, the longer (more specific) path wins; if both match at equal path length, Allow takes precedence. The tool highlights which specific rule triggered the allow-or-block decision and displays the matched pattern, so you can identify conflicting rules without guessing. Crawl-delay and Sitemap directives are parsed and shown as informational but are not evaluated for URL access status. Key assumption: real-world crawlers cache robots.txt for up to 24 hours after each fetch; a change made today may not be honored by Googlebot until its next scheduled crawl of your robots.txt file. Edge case: Google ignores the Crawl-delay directive entirely, so it has no effect on Googlebot's crawl rate, which is controlled via crawl rate settings in Google Search Console. The tool surfaces Crawl-delay for reference but cannot model its actual effect on any crawler.

Worked example

With Disallow: /admin/ and Allow: /admin/public/ in the same user-agent block, the URL /admin/public/file.html is allowed. The Allow rule path (/admin/public/) is longer than the Disallow path (/admin/), so it takes precedence. This is the most common source of confusion in robots.txt configuration.

Frequently asked questions

How does Google's robots.txt matching algorithm work?
Google finds all rules that match the URL (where the URL starts with the rule path), then selects the most specific one (longest path). If an Allow and a Disallow match at the same specificity, Allow wins. User-agent-specific groups override the wildcard group.
Does robots.txt prevent indexing?
No. Robots.txt blocks crawling, not indexing. Google can still index a URL it has never crawled if other sites link to it or if it was previously in the index. Use a noindex meta tag to prevent indexing.
What happens if my robots.txt file is missing?
If Google cannot fetch robots.txt (404 or connection error), it assumes the file has no restrictions and crawls the entire site. A 500 error causes Google to temporarily pause crawling until the file is accessible again.
What is the difference between Disallow: / and Disallow: /*?
They are treated identically by Google. Both block all URLs for that user-agent. The wildcard * at the end of a path in robots.txt is a Google extension and matches any continuation of the path. Tracking this metric alongside conversion data gives a more complete picture of how changes affect actual business outcomes.
Should I block CSS and JavaScript files?
No. Google needs to render your pages to understand their content, which requires access to CSS and JavaScript files. Blocking these files can prevent Google from understanding your page layout and content correctly.
How do I test my robots.txt with multiple URLs at once?
Use the batch mode: enter one URL per line in the multi-URL textarea. The tool runs the matching algorithm against all of them and returns a results table showing allowed or blocked status for each URL and the specific rule that matched.