Crawlability failures are the most underdiagnosed class of SEO problem because they produce no visible symptoms on the front end. The site looks fine. It loads correctly. The user can navigate normally. Google cannot.
This checklist covers the 8 most common crawlability failures we encounter on WordPress sites. It is ordered by severity โ the first three can cause catastrophic invisible ranking loss; the remainder are structural inefficiencies that accumulate silently over time.
1. The accidental sitewide noindex
WordPress has a setting under Settings โ Reading called "Discourage search engines from indexing this site". It is designed for staging environments. It is also the most common cause of a brand-new website producing zero organic traffic despite being technically live. When a staging site is migrated to production and the developer forgets to uncheck this box, every page on the site carries a noindex directive. Google stops indexing immediately. Traffic falls to zero. The site owner assumes the SEO work has not had time to take effect and waits โ for months, sometimes โ while Google is being politely told to stay out.
How to check: In Google Search Console, go to Coverage โ Excluded โ "Excluded by noindex tag". Any pages listed there are actively telling Google not to index them. Also check via Ahrefs Site Audit โ HTTP response codes โ filter for pages with a noindex meta tag.
2. Robots.txt blocking key page types
The robots.txt file instructs crawlers which sections of the site to avoid. A well-intentioned developer adding a rule to block tracking parameters, session IDs, or admin directories can accidentally introduce a rule that blocks service pages, product categories, or the checkout flow. This is particularly common on sites that have been through multiple development handovers.
The specific pattern to watch for is an overly broad 'Disallow' rule. Disallow: /services/ when the intention was to block /services/internal/ will prevent Google from crawling every page in the services directory. Check robots.txt via Google Search Console โ Settings โ robots.txt tester. Test every high-value URL pattern against the current robots.txt before and after any site changes.
3. Canonical tags pointing to the wrong URL
The canonical tag tells Google which version of a page is the authoritative one. When canonical tags are misconfigured โ pointing to a different domain, pointing to a redirected URL, or self-referencing incorrectly โ Google treats the page as a duplicate of something else and may choose not to rank it. This happens most commonly after site migrations or domain changes or when staging environments are used and the staging canonical is left on production.
Verify canonical tags via Ahrefs Site Audit โ Canonicals โ Non-canonical pages receiving organic traffic. Any page receiving traffic that has a canonical pointing elsewhere is a misconfiguration risk.
4. Redirect chains longer than two hops
A 301 redirect is the correct way to handle a URL change. A redirect chain โ where URL A redirects to URL B which redirects to URL C โ loses link equity at each hop and increases crawl budget consumption. Google recommends keeping redirect chains to a single hop. Chains of three or more hops should be treated as technical debt to be resolved.
Screaming Frog identifies redirect chains automatically. Ahrefs Site Audit flags them under Redirects โ Redirect chains. The fix is straightforward: update the first redirect to point directly to the final destination, bypassing the intermediate hops.
5. Orphan pages with no internal links
Google discovers pages primarily through links. A page that has no internal links pointing to it โ an orphan page โ will be discovered infrequently if at all, will receive no PageRank flow from the site's link equity, and will rank weakly regardless of its content quality. Orphan pages are extremely common on sites with large blog archives where older articles are never updated to link to newer ones.
Rank Math Pro's Link Genius identifies orphan pages automatically. AIOSEO's Link Assistant does the same. The fix โ adding one or two internal links from relevant existing pages โ takes minutes and has an immediate crawlability and authority impact.
6. Paginated content with incorrect handling
E-commerce sites, news publications, and large blog archives use pagination. Incorrect pagination handling โ either blocking paginated pages in robots.txt or failing to connect them with canonical or rel=next/prev signals โ can cause Google to index only the first page of a large content set, leaving subsequent pages undiscovered.
The modern correct approach for paginated content is to allow Google to crawl all paginated pages, ensure each page has a canonical pointing to itself (not to page 1), and ensure the content on paginated pages is substantive rather than thin. Google discontinued support for rel=next/prev in 2019 but crawls pagination patterns intelligently. The main risk is robots.txt blocking or noindex tags applied to paginated pages.
7. Excessive crawl budget consumed by low-value URLs
Google allocates a crawl budget per domain โ a limited number of URLs it will crawl in a given time period. On large sites, this budget can be consumed by low-value URLs: session-based parameters, printer-friendly versions, search results pages, and sort and filter combinations on e-commerce sites. When the crawl budget is exhausted on low-value URLs, high-value pages (product pages, service pages, and blog articles) are crawled less frequently.
The fix is two-part: block low-value URL patterns in robots.txt or via meta noindex and submit the XML sitemap with only the pages you want crawled. Ahrefs Site Audit surfaces crawl budget issues under the Crawl report. Google Search Console's Crawl Stats report shows the actual distribution of crawl activity.
8. JavaScript-rendered content invisible to Google
Single-page applications and sites that load key content via JavaScript present a specific crawlability challenge. Googlebot renders JavaScript, but it does so asynchronously and does not always execute all JavaScript before indexing a page. Content that appears on screen but is loaded dynamically โ through React, Vue, Angular, or custom JavaScript โ may not be present in the version of the page that Google indexes.
The diagnostic check is simple: view the page source (Ctrl+U in any browser) and look for the main body content. If the source shows a near-empty body with a large JavaScript bundle reference, and the content only appears after JavaScript execution, there is a rendering risk. The fix depends on the site architecture โ server-side rendering or pre-rendering are the standard solutions for critical content pages.
This checklist is not exhaustive. It covers the 8 most common causes of invisible crawlability failures on WordPress sites. Before running any ranking optimisation campaign, verify that none of these eight issues are present. The most sophisticated keyword strategy in the world cannot overcome a site that Google is being told not to crawl.
Senior Editor ยท Revenue desk
