Free Robots.txt Checker & Validator
Fetch and validate any site’s robots.txt in seconds. We parse every directive, test which crawlers can reach your pages, check your sitemap declarations, and flag the rules quietly costing you visibility.
Why validate your robots.txt?
Crawl control
Confirm Googlebot can reach the pages that matter, and that low-value URLs are not eating your crawl budget.
AI crawler policy
See which of the fourteen major AI crawlers you allow and which you block — and whether that was a decision or an accident.
Security and privacy
robots.txt is public. We flag any disallowed path that advertises an admin area, backup or staging environment to anyone reading it.
Sitemap discovery
Check your Sitemap directive is present and absolute. If it is missing, we look for the sitemap anyway and tell you what to add.
How it works
Enter your domain
We find and fetch /robots.txt for you. No signup, no limits.
Automated fetch and analysis
Every directive is parsed against the Robots Exclusion Protocol, with correct longest-match precedence and multi-group merging.
Crawler and sitemap discovery
We test 14 AI crawlers and 5 search crawlers against your rules, and check your sitemap declarations.
Review the results
A score, a full checklist, an access matrix, syntax errors by line number, and a cleaner robots.txt to start from.
Why this matters
robots.txt is four lines of plain text that can remove an entire website from Google. We have seen it happen more than once — a staging site goes live with its Disallow: / intact, and nobody notices until traffic has been gone for three weeks.
It is also the file that decides whether ChatGPT, Claude, Perplexity and Google’s AI Overviews can read your content at all. Most sites have never made a deliberate decision about that; their AI policy is whatever their wildcard rules happen to imply.
This checker fetches your live file, parses every directive against the Robots Exclusion Protocol, and tells you exactly what it does — including which of the fourteen major AI crawlers you are currently allowing or blocking, and why.
What we check
- Does the file exist and parse. A surprising number of sites serve an HTML 404 page at /robots.txt, which crawlers treat as “no file”.
- Syntax errors. Lines without a colon, Sitemap directives with relative URLs, and unknown directives that will be silently ignored.
- Homepage crawlability for Googlebot. The check that matters most, done with correct longest-match rule precedence.
- Blanket blocks. Any
Disallow: /applied toUser-agent: *. - Search crawler access. Googlebot, Bingbot, DuckDuckBot, Slurp and Applebot tested individually.
- AI crawler access. All fourteen major agents, with whether each is named explicitly or just falling through to your wildcard rules.
- Sitemap declarations. And if none are declared, whether one exists anyway that you should be pointing to.
- Sensitive paths. robots.txt is public. Listing /backup/ or /wp-admin/ in it tells anyone reading exactly where to look.
- Deprecated directives.
Noindex:in robots.txt has done nothing since 2019, andCrawl-delayis ignored by Google entirely.
The AI crawler question
There are two defensible positions here and one indefensible one. You can allow AI crawlers, accepting that your content feeds answer engines in exchange for being citable in them. You can block them, accepting invisibility in AI answers in exchange for not contributing. What you should not do is neither — leaving it to whatever your wildcard rules happen to say, and finding out eighteen months later.
For most businesses selling a service, allowing them is the obvious call. Being the source an answer engine cites is the closest thing to a recommendation that exists at that stage of the funnel. Publishers with a licensing strategy have a genuinely different calculation.
Either way, the tool names each agent explicitly so you can see what your file currently does, and gives you a starter file that states the policy on purpose.
Blocking is not security
A Disallow rule is a request, not an enforcement. Well-behaved crawlers honor it; nothing else does. And because robots.txt is publicly readable by anyone who types the URL, disallowing a path advertises its existence to exactly the people you were worried about.
If a path needs protecting, protect it — authentication, IP rules, or server-level access control. Use robots.txt for crawl budget management, not for hiding things.
Best practices
- Always declare your sitemap with an absolute URL as the last line.
- Name AI crawlers explicitly so your policy is a decision rather than an accident.
- Block low-value URLs that waste crawl budget: internal search results, cart and checkout, faceted filter combinations.
- Never block CSS or JavaScript that Google needs to render your pages.
- Keep it short. Every additional rule is another thing that can be wrong.
- Re-check after any plugin update or migration — this file changes without anyone touching it.
Common mistakes to avoid
- Shipping a staging
Disallow: /to production. - Using
Noindex:in robots.txt, which Google has ignored since 2019. - Listing sensitive directories, which advertises them to anyone reading the file.
- Blocking a page you also want de-indexed — a blocked page cannot be crawled, so the noindex tag on it is never seen.
- Relative sitemap URLs. The Sitemap directive requires an absolute URL.
Frequently asked questions
Do I even need a robots.txt file?
Not strictly. Without one, crawlers assume everything is allowed and your site will be indexed normally. But you lose the ability to point crawlers at your sitemap, to keep them off low-value URLs, and to state an explicit policy for AI crawlers. It is a small file with real leverage.
Will blocking a page in robots.txt remove it from Google?
No, and this is the most misunderstood part of the file. Blocking prevents crawling, not indexing. Google can still index a blocked URL from external links, showing it with no description. To remove a page, allow it to be crawled and add a noindex meta tag — Google has to be able to read the tag for it to work.
Should I block GPTBot and the other AI crawlers?
It depends on your model. If you sell services and want to be recommended when someone asks an AI for options, allow them — a citation in an AI answer is close to a referral. If you monetize content directly and are pursuing licensing, blocking is a coherent strategy. The wrong answer is not deciding.
What is the difference between Disallow and noindex?
Disallow controls crawling; noindex controls indexing. They live in different places — Disallow in robots.txt, noindex in a meta tag or X-Robots-Tag header — and using them together on the same URL is self-defeating.
Does Google respect Crawl-delay?
No. Google has never supported it. Bing and Yandex do. If Googlebot is hitting your server too hard, adjust the crawl rate in Search Console instead.
Where does the file have to live?
At the root of the domain, exactly: https://example.com/robots.txt. It cannot live in a subfolder, and each subdomain needs its own.
Can robots.txt affect my Google Business Profile or map pack rankings?
Not directly — your Business Profile is a separate property and Google does not need to crawl your site to show it. But the map pack is influenced by your website’s relevance and authority for the service and area, and that requires your service pages to be crawlable. If your location or service-area pages are sitting behind a Disallow rule, the signals that support your local ranking cannot be read at all. It is an indirect problem with a very direct cost.
Want this fixed, not just found?
RevSurge Digital is a digital marketing agency in Sarasota, Florida, running
technical SEO and answer engine optimization and
paid media for small and mid-sized businesses across
Sarasota, Bradenton and the Gulf Coast. Everything this tool flags is work we do every week.
More tools like this
GEO Audit
Find out whether AI answer engines can actually read, understand and cite your website.
SEO Competitor Analysis
Put your site next to a competitor and see, metric by metric, where you win and where you lose.
Missing Keywords
Find the keywords and topics your site should be targeting and never mentions.
Sitemap Checker
Find, parse and validate your XML sitemap, then live-test the URLs inside it.
Meta Description Checker
Check any description against Google’s real pixel limits, desktop and mobile.
