How to Fix Crawl Errors in Google Search Console: 8 Common Errors Explained

You open Google Search Console, click into the Page indexing report, and you are met with a wall of labels: Server error (5xx), Soft 404, Redirect error, Crawled – currently not indexed. Each one sounds vaguely alarming, but Google never tells you what to actually do about it.

This guide is built as a diagnostic tool. Find the label you see in your report, read what it really means, then apply the fix. No theory dumps, no generic advice about “improving site quality”. Just the error, the cause, and the repair. semrush.com walks through the specifics.

Quick reference: match your error to its fix

Use this table to jump straight to the section you need.

Error label in Search Console	What it really means	Priority
Server error (5xx)	Your server broke or timed out while Googlebot was requesting the page	Critical
Failed: Robots.txt unreachable / DNS error	Google could not resolve your domain or fetch robots.txt, so it paused crawling entirely	Critical
Blocked by robots.txt	A rule in your robots.txt file is stopping the crawl	High if it hits money pages
Not found (404)	The URL does not exist. Sometimes normal, sometimes a broken link	Medium
Soft 404	The page returns 200 OK but looks empty or dead to Google	High
Redirect error	Redirect loop, chain that is too long, empty URL in the chain, or a bad final target	High
Blocked due to unauthorized request (401) / Forbidden (403)	Googlebot is being asked to log in, or your firewall is blocking it	High
Discovered / Crawled – currently not indexed	Not a crawl failure. A crawl budget or content quality signal	Medium

Where crawl errors actually live in Search Console today

The old standalone “Crawl Errors” report is long gone. If someone tells you to open it, they are working from outdated notes. In the current interface, crawl and indexing problems are split across three places:

• Indexing > Pages (the Page indexing report): every URL Google knows about, grouped by why it is or is not indexed. This is where you will spend 90% of your time.
• Settings > Crawl stats: host-level health. Response codes over time, average response time, file types fetched, and crawl requests by purpose. This is where you spot server-wide problems.
• URL Inspection tool: single-URL forensics. It tells you the last crawl date, the crawl result, the canonical Google picked, and whether the page is indexable right now.

Site errors versus URL errors

Before you touch anything, work out which category you are in. It changes the urgency completely.

• Site errors affect your entire domain: DNS failures, an unreachable robots.txt, a server that returns 5xx across the board. Google will slow down or stop crawling everything. These are emergencies.
• URL errors affect individual pages: a single 404, one soft 404, a broken redirect. Annoying, fixable, rarely fatal.

A quick way to tell them apart: open Crawl stats. If your “By response” breakdown shows a spike of 5xx or a flat line where crawl requests should be, you have a site error. If crawl volume looks normal but the Pages report lists specific URLs, you have URL errors.

1. Server error (5xx)

What Google is telling you

Googlebot requested the page and your server responded with a 500, 502, 503, or 504 status. Google could not see the content at all. If this happens repeatedly across many URLs, Google reduces its crawl rate to avoid making things worse.

What causes it

• PHP fatal errors, plugin conflicts, or a memory limit that is too low
• Shared hosting that throttles or drops connections under load
• Database connection failures (very common on WordPress at peak traffic)
• Aggressive bot protection or rate limiting that returns 503 to Googlebot
• A CDN or reverse proxy timing out before your origin responds (502 and 504)

How to fix it

1. Identify the exact code. Open Crawl stats > By response and look at whether you are seeing 500, 502, 503 or 504. Each points to a different layer.
2. 500: application-level. Check your server error log, disable recently added plugins one by one, raise the PHP memory limit, and look for a fatal error in your theme functions file.
3. 502 / 504: gateway or timeout. Increase PHP-FPM timeout values, check whether your CDN origin timeout is shorter than your server response time, and look for slow database queries.
4. 503: often intentional. Confirm you did not leave a maintenance mode plugin active. Also check whether your security layer is serving 503 to Googlebot as a rate limit.
5. Verify Googlebot is not being blocked by your firewall. Cross-check the request in your raw server logs and confirm the IP resolves to Google via reverse DNS.
6. Once fixed, use URL Inspection > Test live URL on a few affected pages. If it returns a clean fetch, click Validate fix in the report.

Note on 503: if you are doing planned maintenance, a 503 with a Retry-After header is the correct response. Google will come back. What you must never do is leave it running for days.

2. DNS error and “Failed: Robots.txt unreachable”

What Google is telling you

Google could not resolve your domain name, or it could resolve it but received a 5xx response when asking for /robots.txt. In the second case Google will not crawl your site at all, because it cannot confirm what it is allowed to access.

This is the single most dangerous entry in this list. A missing robots.txt returning 404 is fine (Google assumes everything is crawlable). A robots.txt returning a server error is not, and it can freeze crawling of your whole domain.

How to fix it

1. Load yourdomain.com/robots.txt in a browser. It should return either valid text with a 200 status, or a clean 404. Anything in the 5xx range needs immediate attention.
2. Check your DNS records with your registrar or DNS provider. Confirm the A or AAAA record points to the right IP and that nameservers are propagating correctly.
3. If you recently migrated hosts or changed nameservers, allow for propagation but verify from multiple locations rather than assuming.
4. Make sure your WAF or bot filter is not challenging requests to robots.txt. Some security products serve a JavaScript challenge page to non-browser agents, which Googlebot cannot pass.
5. In Settings > Crawl stats > Host status, confirm all three checks (robots.txt fetch, DNS resolution, server connectivity) show green.

3. Blocked by robots.txt (and “Indexed, though blocked by robots.txt”)

What Google is telling you

A Disallow rule matched the URL, so Googlebot did not fetch it. This is only a problem if the blocked URL is a page you want in search results.

The related warning Indexed, though blocked by robots.txt means something more awkward: Google found enough external signals (links pointing at the URL) to index it, but cannot read the page. The result is usually an ugly listing with no meta description.

How to fix it

1. Export the affected URLs and sort them. Ask a simple question for each: do I want this in Google?
2. If yes, remove or narrow the Disallow rule in robots.txt. Test the change with the robots.txt tester before deploying.
3. If no, and the page is currently indexed, you have a contradiction to resolve. Robots.txt blocks crawling, which prevents Google from ever seeing a noindex tag. The correct sequence is:
   • Remove the robots.txt block temporarily
   • Add <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header
   • Wait for Google to recrawl and drop the page
   • Only then re-add the robots.txt disallow if you want to save crawl budget
4. Watch out for accidental broad rules. A single Disallow: / left over from a staging environment is one of the most common catastrophic SEO mistakes we see during audits.

Common robots.txt patterns that cause trouble

Rule	Unintended effect
Disallow: /wp-content/	Blocks CSS, JS and images, breaking how Google renders your pages
Disallow: /*?	Blocks every URL with a query string, including paginated and filtered pages you may want indexed
Disallow: /product	Matches /products/ too, since there is no trailing slash or anchor

4. Not found (404)

What Google is telling you

The URL returns a 404 status. Contrary to a persistent myth, 404s are not a ranking penalty. Google expects them. A site with zero 404s is more suspicious than a site with a few hundred.

They only matter when the 404 URL had value: inbound links, traffic, or a place in your internal navigation.

How to triage them

1. Export the full list from the Page indexing report.
2. For each URL, check the Referring page data in the URL Inspection tool or your crawler. Where did Google find this link?
3. Sort into three buckets:
   • Deleted pages with backlinks or old rankings: 301 redirect to the closest equivalent live page. Not to the homepage by default, that often gets treated as a soft 404.
   • Broken internal links or typos: fix the link at the source. Redirecting a typo instead of correcting the link just hides the mess.
   • Junk URLs (spam links, malformed parameters, URLs that never existed): leave them as 404s. This is exactly what a 404 is for.
4. If a page was deleted permanently and has no replacement, consider returning 410 Gone. Google processes it slightly faster than a 404.

5. Soft 404

What Google is telling you

The URL returned a 200 OK status, but the content looks like an error page or an empty page to Google. This is a mismatch between what your server says and what the page actually contains.

Soft 404s are worse than real 404s because Google wastes crawl budget on them and may deindex pages you care about.

Typical causes

• A custom error page that says “Sorry, this page does not exist” while returning 200
• Empty category, tag, or search result pages with no items
• Out of stock product pages with the content stripped out
• Thin pages: a heading, a sentence, and nothing else
• JavaScript-rendered content that never loads for Googlebot, leaving an empty shell
• Redirecting a mass of deleted URLs to the homepage

How to fix it

1. Open each flagged URL in URL Inspection > Test live URL > View crawled page and look at the rendered HTML. If it is empty, you have a rendering problem, not a content problem.
2. If the page genuinely does not exist, return a real 404 or 410 status code.
3. If the page should exist but is thin, add substantive content: descriptions, specifications, related items, useful copy. Enough that a human would find the page worth landing on.
4. For out of stock products, keep the page live with full content, add availability schema, and link to alternatives. Do not blank the template.
5. For empty filtered or paginated views, either noindex them or prevent them from being generated when there are no results.
6. For JavaScript sites, check that the content is present in the server response or that server-side rendering is functioning for bots.

6. Redirect error

What Google is telling you

Googlebot followed a redirect and something went wrong along the way. The label covers four distinct problems:

• Redirect chain too long: Google follows a limited number of hops, generally around five, then gives up
• Redirect loop: A points to B, B points back to A
• Empty or malformed URL in the chain
• Bad final destination: the chain ends on a 404 or a 5xx

How to fix it

1. Crawl your site with a desktop crawler and export the full redirect map. You need to see every hop, not just the endpoint.
2. Flatten every chain to a single hop. If A goes to B goes to C, rewrite the rule so A goes directly to C. Then delete the intermediate rule if nothing else uses it.
3. Find loops by looking for URLs that appear on both sides of a redirect rule. The classic causes are conflicting HTTP-to-HTTPS and www-to-non-www rules, or a trailing slash rule fighting a CMS rule.
4. Order matters in server config. Protocol and hostname canonicalisation should happen first, in a single rule, before any content-level redirects.
5. Update internal links to point at final destinations. Redirects should be a safety net for external links, not your internal architecture.
6. Check your XML sitemaps. They should contain only 200-status canonical URLs, never redirects.

Chain audit example

Situation	Verdict
http://site.com → https://www.site.com/page	Fine, one hop
http://site.com → https://site.com → https://www.site.com → https://www.site.com/page/	Fix, three hops of wasted crawl budget
/old-page → /new-page → 404	Fix urgently, all link equity is lost

7. Blocked due to unauthorized request (401) and Forbidden (403)

What Google is telling you

Your server refused Googlebot’s request. A 401 means authentication was required. A 403 means access was denied outright.

What causes it

• HTTP basic auth left on a staging site that has since gone live
• Geo-blocking or IP filtering that excludes the ranges Googlebot crawls from
• Bot protection or a WAF classifying Googlebot as a scraper
• Content behind a login wall or paywall with no crawlable version
• Overly strict file permissions or a misconfigured directory rule

How to fix it

1. Test the URL with URL Inspection > Test live URL. If it fails there but loads in your browser, the block is agent or IP based, not a page problem.
2. Remove basic auth from production. Check for leftover .htpasswd references in your server config.
3. Whitelist verified Googlebot in your security layer. Most managed WAFs have a one-click option for verified search engine crawlers. Always verify by reverse DNS, never by user agent string alone.
4. Check your server logs for the exact response returned to Googlebot’s IP. This settles the argument faster than any guesswork.
5. For genuinely gated content, either accept it will not be indexed, or serve a crawlable preview with the appropriate structured data for paywalled content.

8. Discovered – currently not indexed, and Crawled – currently not indexed

What Google is telling you

These are the two most misunderstood labels in the whole report, and they are not technically crawl errors at all. Anyone digging further should read Understanding Google SEO emails and console errors.

• Discovered – currently not indexed: Google knows the URL exists but has not fetched it yet. This is usually a crawl budget or server capacity signal. Google is deliberately holding back, often because it expects your server to struggle or because it does not consider the URL a priority.
• Crawled – currently not indexed: Google fetched the page, evaluated it, and decided not to index it. This is a quality or duplication judgement.

How to fix “Discovered – currently not indexed”

1. Check server response times in Crawl stats. If your average is above roughly 600ms, Google will crawl less. Fix performance first.
2. Reduce the number of low-value URLs competing for crawl budget: parameter variations, infinite filter combinations, thin tag archives, internal search result pages.
3. Improve internal linking to the affected pages. Orphaned pages sitting only in a sitemap get deprioritised.
4. Make sure the URLs appear in a clean, current XML sitemap with accurate lastmod values.

How to fix “Crawled – currently not indexed”

1. Read the page as a stranger would. Does it add anything that is not already available on ten other pages? If not, that is your answer.
2. Look for near-duplicate content across your own site. Product variants, location pages built from a template, and syndicated content are the usual suspects.
3. Consolidate. Three thin pages on a topic almost always perform better as one strong page.
4. Add internal links from pages Google crawls often. A page with no internal links from indexed content sends a clear signal about its importance.
5. Once improved, request indexing through URL Inspection. Do this for genuinely updated pages, not as a bulk tactic.

Reality check: if these pages are auto-generated archives, filtered views, or thin variations that you never intended to rank, the correct response is to noindex them or stop generating them. Not every URL deserves to be indexed, and clearing them out of the report makes real problems visible.

Bonus: the labels that are not errors

Plenty of entries in the Page indexing report are working exactly as intended. Do not waste a sprint fixing these:

• Page with redirect: the URL redirects. That is what you asked it to do.
• Alternate page with proper canonical tag: Google respected your canonical. Perfect.
• Excluded by noindex tag: you told Google not to index it and it complied. Only investigate if a page you want indexed appears here.
• Duplicate, Google chose different canonical than user: worth reviewing. Google disagreed with your canonical choice, usually because internal links and content signals point elsewhere. Align your internal linking, sitemap and canonical tags on one URL.

The validation workflow: how to close an error properly

Fixing the underlying problem is only half the job. Search Console needs to confirm it.

1. Fix the root cause across all affected URLs, not just the samples Google showed you. The report displays a limited sample of a potentially much larger set.
2. Verify manually using URL Inspection on three or four URLs from different parts of the affected group. Use Test live URL, not the cached result.
3. Click Validate fix in the relevant issue panel. Google then recrawls a sample of the URLs.
4. Wait. Validation typically takes anywhere from a couple of days to a few weeks depending on site size and crawl frequency. Starting it and abandoning it is common and pointless.
5. If validation fails, Google shows you the first URL that failed. That URL is your debugging starting point. Fix it, then restart validation.

One important caveat: do not start a validation before the fix is actually deployed to production and cached versions have cleared. A failed validation resets the clock.

Prevention: a simple monthly routine

Most crawl errors are the residue of changes nobody audited afterwards. A short recurring check catches them before they compound. There is a practical rundown of it online.

• Weekly: glance at Crawl stats host status. All three checks green, no 5xx spikes.
• Monthly: open the Page indexing report and compare the trend line against last month. A sudden jump in any category is more informative than the absolute number.
• Monthly: run a full site crawl and check for new redirect chains, broken internal links, and orphaned pages.
• Monthly: confirm your XML sitemap contains only indexable 200-status URLs.
• After every deployment, migration or plugin update: check robots.txt, spot-check a handful of key URLs with URL Inspection, and confirm no noindex tags shipped from staging.

Priority order when you have dozens of issues at once

If your report is a mess, work in this sequence. Fixing errors in the wrong order wastes time because upstream problems mask downstream ones.

1. Host-level errors: DNS, robots.txt unreachable, site-wide 5xx
2. 403 and 401 blocks affecting large sections of the site
3. Robots.txt rules blocking pages that should rank
4. Redirect loops and chains ending in errors
5. Soft 404s on commercially important pages
6. 404s on URLs with backlinks or historical traffic
7. Crawled and Discovered not indexed, on pages you actually want ranked
8. Everything else

FAQ

How do I fix “Crawled – currently not indexed” in Google Search Console?

Treat it as a quality verdict rather than a bug. Google fetched the page and chose not to index it, usually because the content is thin, duplicates something else, or has almost no internal links pointing at it. Strengthen or consolidate the content, add internal links from pages Google crawls regularly, make sure the URL is in your sitemap, then request indexing. If the page was never meant to rank, noindex it instead.

What are crawl stats in Google Search Console?

The Crawl stats report, found under Settings, shows how Googlebot interacted with your server over the last 90 days. It breaks down total crawl requests, total download size, average response time, and splits requests by response code, file type, purpose (discovery versus refresh) and Googlebot type. It also includes a host status panel covering robots.txt fetch, DNS resolution and server connectivity. It is the fastest way to distinguish a server-wide problem from isolated page issues.

How do I trigger a Google crawl of my site?

Use the URL Inspection tool, enter the URL, and click Request indexing. This adds the page to a priority crawl queue, though it does not guarantee indexing. For larger batches, submit or resubmit an XML sitemap with accurate lastmod dates. Manual requests are rate limited, so use them for genuinely important updates. Consistent publishing, strong internal linking and fast server responses influence crawl frequency far more than manual requests.

Why is Google not crawling my website?

Work through this list in order: check that robots.txt does not contain a site-wide Disallow, confirm your property is verified in Search Console, check host status in Crawl stats for DNS or connectivity failures, verify your server is not returning 5xx or 403 to Googlebot, confirm no site-wide noindex tag is present (a very common leftover from staging in WordPress), and make sure a valid sitemap has been submitted. If everything is clean and the site is new, it may simply be a matter of time and building a few external links.

Do 404 errors hurt my rankings?

No. Google has stated repeatedly that 404s are a normal part of the web and do not carry a ranking penalty. They become a problem only when they affect URLs that had value: pages with backlinks, pages that used to rank, or pages still linked from your navigation. Redirect those. Leave genuine dead URLs as 404s.

How long does it take for fixed crawl errors to disappear from the report?

The Page indexing report is not real time. After you click Validate fix, Google recrawls a sample of URLs, which usually takes a few days to a few weeks depending on your site’s crawl rate. Large sites with slow crawl frequency can take longer. The count also drops naturally over time as Google recrawls without any validation request, though starting validation speeds it up.

Should I fix every error listed in the Page indexing report?

No. Several labels describe correct behaviour, including pages with redirects, alternate pages with proper canonicals, and pages excluded by noindex. Focus on errors affecting URLs you want ranked. A report showing thousands of excluded URLs is normal for large sites. What matters is whether your important pages are indexed and served.

Final word

Crawl errors in Google Search Console are rarely mysterious once you separate the true failures from the labels that just describe how your site is configured. Start with anything affecting your whole host, work down to individual URLs, and resist the urge to chase a zero in every category. The goal is not an empty report. The goal is that every page you care about gets crawled, understood and indexed without friction.

If your report has grown into something you no longer want to open, our technical SEO team runs full crawl diagnostics and fixes the root causes rather than the symptoms. Get in touch and we will take a look.