Robots.txt Checker
Is your site accidentally blocking Googlebot?
Chexter fetches robots.txt and analyzes all directives: syntax, Allow/Disallow conflicts, and Sitemap links. Finds blocks that prevent important pages from being indexed.
Loading page...
Analyzing...
Rate limit reached
You have reached the check limit. Try again in:
Methodology
How Robots.txt Checker works
Is robots.txt reachable at /robots.txt? Returns 200 with text/plain Content-Type?
All directives follow the Robots Exclusion Protocol. Unknown directives are flagged.
Conflict detection: does an Allow rule override Disallow for the same path?
Critical check: is Disallow: / present for User-agent: * or Googlebot?
Does robots.txt include a Sitemap: directive pointing to a valid sitemap URL?
What Robots.txt Checker checks
Why use Chexter Robots.txt Checker
Frequently Asked Questions
What is robots.txt?
robots.txt is a plain-text file at the root of a website that tells search crawlers which pages they may and may not access. It follows the Robots Exclusion Protocol standard.
What happens if robots.txt blocks Googlebot?
Googlebot will not crawl or index the blocked pages. Disallow: / for User-agent: * is the most critical error — it prevents all crawlers from accessing the entire site.
Does blocking a page in robots.txt remove it from Google?
No. Blocking in robots.txt prevents crawling but not indexing if the page has backlinks. Use noindex meta tag to fully exclude a page from search results.
Is robots.txt required?
Not required, but strongly recommended. Without it, all crawlers have unrestricted access. Google treats a missing robots.txt as "allow all".
Get started
Try Robots.txt Checker now
Free, no signup required. Get your report in seconds.
Free forever · No credit card · Instant results