OnSumoTools

robots.txt Tester

The Robots.txt Tester lets you paste your robots.txt content and test any URL against any user-agent to see whether crawlers are allowed or blocked. The tool applies Google's exact matching algorithm, including longest-path precedence and Allow-over-Disallow tie-breaking rules.

Syntax preview

User-agent: *
Disallow: /admin/
Allow: /public/
 
User-agent: Googlebot
Disallow: /staging/

Allowed

Matched: No matching rule

Matched group: googlebot

Batch results

URLResultMatched rule
/admin/pageAllowedNo matching rule
/public/pageAllowedNo matching rule
/staging/homeBlockedDisallow: /staging/
Google robots.txt spec

This tester evaluates only the rules you paste. It does not fetch robots.txt from your live domain.

How does the robots.txt Tester work step by step?

The robots.txt tester validates whether a specific URL is accessible or blocked for a named user-agent according to the rules in your robots.txt, using the same rule-precedence logic that compliant crawlers apply. The parsing order: more-specific user-agent blocks override more-general ones; within a block, when both Allow and Disallow rules match a URL, the longer (more specific) path wins; if both match at equal path length, Allow takes precedence. The tool highlights which specific rule triggered the allow-or-block decision and displays the matched pattern, so you can identify conflicting rules without guessing. Crawl-delay and Sitemap directives are parsed and shown as informational but are not evaluated for URL access status. Key assumption: real-world crawlers cache robots.txt for up to 24 hours after each fetch; a change made today may not be honored by Googlebot until its next scheduled crawl of your robots.txt file. Edge case: Google ignores the Crawl-delay directive entirely, so it has no effect on Googlebot's crawl rate, which is controlled via crawl rate settings in Google Search Console. The tool surfaces Crawl-delay for reference but cannot model its actual effect on any crawler.

What does a typical robots.txt Tester result look like?

With Disallow: /admin/ and Allow: /admin/public/ in the same user-agent block, the URL /admin/public/file.html is allowed. The Allow rule path (/admin/public/) is longer than the Disallow path (/admin/), so it takes precedence. This is the most common source of confusion in robots.txt configuration.

Frequently asked questions

  • How does Google's robots.txt matching algorithm work?

    Google finds all rules that match the URL (where the URL starts with the rule path), then selects the most specific one (longest path). If an Allow and a Disallow match at the same specificity, Allow wins. User-agent-specific groups override the wildcard group.

  • Does robots.txt prevent indexing?

    No. Robots.txt blocks crawling, not indexing. Google can still index a URL it has never crawled if other sites link to it or if it was previously in the index. Use a noindex meta tag to prevent indexing.

  • What happens if my robots.txt file is missing?

    If Google cannot fetch robots.txt (404 or connection error), it assumes the file has no restrictions and crawls the entire site. A 500 error causes Google to temporarily pause crawling until the file is accessible again.

  • What is the difference between Disallow: / and Disallow: /*?

    They are treated identically by Google. Both block all URLs for that user-agent. The wildcard * at the end of a path in robots.txt is a Google extension and matches any continuation of the path. Tracking this metric alongside conversion data gives a more complete picture of how changes affect actual business outcomes.

  • Should I block CSS and JavaScript files?

    No. Google needs to render your pages to understand their content, which requires access to CSS and JavaScript files. Blocking these files can prevent Google from understanding your page layout and content correctly.

  • How do I test my robots.txt with multiple URLs at once?

    Use the batch mode: enter one URL per line in the multi-URL textarea. The tool runs the matching algorithm against all of them and returns a results table showing allowed or blocked status for each URL and the specific rule that matched.

Related tools

A tiny file with big consequences

robots.txt tells crawlers where they can go. One wrong line can hide your whole site from Google. It happens more than you'd think, usually after a staging file gets copied to production.

This tester takes your file, a URL and a user-agent, and says whether the URL is allowed. No guessing, no waiting for a crawl.

How matching works

Google picks the most specific rule, meaning the longest matching path. If Disallow: /shop/ and Allow: /shop/sale/ both match, the longer Allow wins. If two rules tie in length, Allow wins.

Wildcards help. The star matches any run of characters, and a dollar sign anchors the end of the URL. Disallow: /*.pdf$ blocks PDFs and nothing else.

The tool follows these rules and shows which line decided the result. That's the useful part. You see why, not just yes or no.

Disallow: / versus Disallow: /*

Both block everything for that agent in practice, but they read differently, and an empty Disallow allows everything. Mixing them up is a classic mistake. Test the exact lines you wrote.

Batch checks

Paste a list of URLs and test them all at once. Run your key templates: home, category, product, blog, search, login. Add a few that should be blocked, like admin paths. If any result surprises you, fix the file before deploying.

What robots.txt doesn't do

It stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it, just without a description. To keep a page out of the index, use a noindex tag, and let crawlers see it.

If the file is missing, crawlers assume everything is open. Don't block CSS or JavaScript that pages need to render, since Google uses them to understand layout.

After every deploy

Make testing a habit. A stray Disallow: / on a live site is a classic disaster. Check the live file, not your local copy. Your input stays in your browser.