Your robots.txt file is four lines of text that can quietly decide whether Google sees your site at all. One misplaced character — a stray slash after Disallow: — and you have told every search engine to ignore your entire website. It happens more often than you would think, usually after a site migration or a developer forgetting to remove a staging block.

Here is how to check your robots.txt, the errors we find most often when auditing Sarasota and Bradenton business sites, and how to fix each one without hurting your rankings.

What Robots.txt Actually Does

Robots.txt lives at yourdomain.com/robots.txt and gives crawlers instructions about which parts of your site they may request. It is a gatekeeping file, not a security file — and it does not remove pages from Google’s index. A page blocked by robots.txt can still appear in search results (as a bare URL with no description) if other sites link to it. De-indexing is noindex‘s job; crawl control is robots.txt’s job. Confusing the two is the root of most robots.txt damage.

How to Check Your Robots.txt in 60 Seconds

1. Load the file directly. Visit yourdomain.com/robots.txt. If it 404s, that’s fine (everything is crawlable by default). If it returns a 5xx server error, Google may stop crawling your whole site — fix that immediately.
2. Run it through a checker. A robots.txt checker parses every directive, flags syntax errors, and tests specific URLs against your rules so you can see exactly what is blocked.
3. Cross-check Search Console. The Pages report shows “Blocked by robots.txt” — if important URLs appear there, you have found real damage.

The 7 Robots.txt Errors We Find Most

1. Disallow: / Left Over From Staging

The site-killer. Developers block staging sites (correctly), then launch without removing the rule. If traffic cratered after a redesign, check this first.

2. Blocking CSS and JavaScript

Google renders pages like a browser. Block /wp-includes/ or your theme assets and Google sees a broken page — mobile-friendliness and Core Web Vitals assessments suffer accordingly.

3. Using Robots.txt to “Hide” Sensitive Pages

Robots.txt is public. Listing /admin-login/ or /customer-data/ is an advertisement, not protection. Use authentication and noindex instead.

4. Blocking Pages You Want De-Indexed

If a page is blocked by robots.txt, Google cannot crawl it — which means it can never see the noindex tag on it. Unblock first, let Google recrawl, then the noindex takes effect.

5. Missing or Wrong Sitemap Line

Add Sitemap: https://yourdomain.com/sitemap_index.xml (your real sitemap URL). It is the cheapest crawl-discovery win available.

6. Case and Path Mismatches

Robots.txt is case-sensitive. Disallow: /Blog/ does not block /blog/. Wildcards (*) and end-anchors ($) are supported but frequently misused — test rules against real URLs rather than assuming.

7. Blocking AI Crawlers Unintentionally (or Not Deciding at All)

GPTBot, ClaudeBot, PerplexityBot and others respect robots.txt. In 2026, blocking them means invisibility in AI answers — a growing share of local discovery. Make it a deliberate policy decision, not an accident of a copy-pasted template.

A Healthy Robots.txt for Most WordPress Sites

For a typical business site, less is more:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap_index.xml

Everything else — crawl-delay lines, long disallow lists copied from 2012 blog posts, per-bot rules you cannot explain — usually creates more risk than value.

Real-World Damage: Three Robots.txt Postmortems

The migration wipeout. A Florida home-services company relaunched on a new theme; the developer copied staging’s robots.txt — Disallow: / included — to production. Organic traffic fell 70% over three weeks before anyone looked at the file. Recovery took two months of recrawling after a ten-second fix. The lesson: add robots.txt to your launch checklist, and set a calendar reminder to verify it the day after any site change goes live.

The invisible noindex. An e-commerce site blocked its filtered category URLs in robots.txt and added noindex tags to them — then wondered why thousands of blocked URLs still appeared in Google. Because the crawler could not fetch the pages, it never saw the noindex. Unblocking for six weeks let the noindex process, then the block went back on. Order of operations matters.

The rendering penalty. A WordPress site with a 2014-era robots.txt blocked /wp-content/ entirely. Google rendered every page without CSS — and mobile-friendliness flags quietly suppressed rankings sitewide. Deleting three legacy lines produced a measurable recovery within a month.

Robots.txt Maintenance: A Quarterly 5-Minute Routine

1. Load /robots.txt in a browser — confirm it returns 200 and looks intentional.
2. Run it through a checker for syntax errors and surprise blocks.
3. Open Search Console’s Pages report and scan “Blocked by robots.txt” for anything valuable.
4. Confirm the Sitemap line points at your live sitemap URL.
5. Review your AI-crawler policy against current business goals — the bot landscape changes fast enough that last year’s decision may be stale.

Five minutes a quarter is cheap insurance against the only file on your site that can silently turn off Google — and if you run a local business in Sarasota, Bradenton, or anywhere on the Gulf Coast, it is the first thing we check in every site audit.

The Bottom Line

Robots.txt is the highest-stakes-per-line file on your website: invisible when right, catastrophic when wrong. The full defense takes minutes — load the file, run it through a checker, scan Search Console for surprise blocks, confirm the sitemap line, and make a deliberate call on AI crawlers. Put it on your quarterly calendar and on every launch checklist, because nearly every robots.txt disaster we see follows a redesign or migration nobody double-checked. Five minutes of verification versus two months of traffic recovery is not a close call.