Technical SEO — the health of your site’s infrastructure, crawlability, and indexation — is the foundation everything else rests on. You can have great content and a clean backlink profile, but if Google cannot crawl and index your pages efficiently, you don’t rank. Screaming Frog is the industry standard tool for technical SEO audits. It is a desktop crawler that behaves like a search engine bot, identifying broken links, duplicate content, redirect chains, missing metadata, orphan pages, canonicalization issues, and more. This article walks through how to set up and run a comprehensive technical SEO audit with Screaming Frog, how to interpret the results, and how to prioritize fixes — for sites of any size.
What Screaming Frog does and why it matters
Screaming Frog crawls your website like Google’s crawler does — following links, discovering pages, analyzing on-page elements. Unlike manual checking or simple link validators, it gives you a bird’s-eye view of every page on your site and exposes patterns of problems that are otherwise invisible because they are scattered across thousands of URLs.
The recurring pattern in audits: a few categories of issue dominate. Broken internal links pointing to 404 pages. Duplicate content where multiple URLs serve the same template. Orphan pages with no internal links pointing to them. Redirect chains where URL A goes to B which goes to C which goes to D. Missing or duplicate title tags and meta descriptions. Canonical tags pointing somewhere odd. None of these are individually catastrophic; collectively they cap how much of the site Google indexes and how authority flows internally.
Installation and initial setup
Download Screaming Frog from screaming-frog.com. Available for Windows, Mac, and Linux. Installation is straightforward — it’s a desktop application that you run locally. The free version crawls up to 500 URLs per project, which covers most small-to-medium sites or single sections of larger sites. The paid licence (around 200 GBP per year as of 2026) lifts the URL cap, adds JavaScript rendering, scheduled crawls, integrations with Google Search Console and Analytics, and custom extraction.
The first crawl: enter your root URL, click Start, and let it run. Small sites finish in a few minutes; larger sites take 30 minutes to several hours. For sites larger than the free tier’s 500-URL limit, plan to crawl section by section or invest in the licence — the audit value scales rapidly with site size.
Understanding crawl results: the Overview tab
After the crawl completes, the Overview tab summarizes total pages crawled, response code distribution (200 OK, 3xx redirects, 4xx not found, 5xx server errors), and basic on-page element counts. The first diagnostic glance is the response code breakdown:
A healthy site has the vast majority of internal-linked URLs returning 200. Significant numbers of 4xx errors indicate broken internal links worth fixing. Significant numbers of 5xx errors indicate server problems that need infrastructure attention before any SEO work makes sense. High counts of 3xx redirects can be normal (intentional redirects after a migration) or symptomatic (chained or unnecessary redirects worth flattening).
The Overview tab also shows counts of pages with title tags, meta descriptions, H1 tags, and other on-page elements. If 30 percent of pages have duplicate meta descriptions, that’s a snippet-level CTR problem worth fixing.
Finding and fixing 404 errors and broken links
In the Response Codes section, click 4xx Client Error to see all 404 pages your site is serving. Every 404 wastes crawl budget — Google could be discovering and crawling new content instead. Three options for each 404:
Delete it if it’s truly no longer relevant and not linked from anywhere external.
Restore it if it was accidentally broken (template change, plugin update, deployment error).
301-redirect it to the most relevant active page. Don’t redirect everything to the homepage — that’s a soft 404 pattern Google’s documentation explicitly discourages. Redirect to the closest topical match: an old product page to its replacement, an old article to a related current article.
The Inlinks tab on each 404 URL shows which internal pages are linking to it. Update or remove those internal links so the 404 stops being requested.
Identifying duplicate content and canonicalization issues
Duplicate content reports (in the paid version’s URL tab and Content section) show URLs serving the same or near-identical content. This is common in e-commerce (the same product accessible via multiple category paths), content sites (pagination, sort parameters, session IDs), and sites with tracking parameters appended to URLs.
The fix is canonical tags (rel=”canonical”) that consolidate the variants onto a single primary URL. Screaming Frog also flags pages with no canonical, self-referential canonicals that are formatted incorrectly, and canonical chains. Each pattern needs a different fix; the Canonical column in the URL tab tells you which.
For URLs with parameters (filters, sort orders, tracking), the fix is canonical tags pointing to the parameter-free version. The old “URL Parameters” tool in Search Console was deprecated in April 2022; canonical tags and clean internal linking are the supported mechanisms now.
Finding orphan pages and internal link structure
An orphan page is a page on your site with no internal links pointing to it. Google may still discover it via external links or sitemaps, but it earns little authority from your internal link graph and is unlikely to rank.
Screaming Frog identifies orphan pages by cross-referencing the crawl with your XML sitemap and (in the paid version) Search Console export. Orphans usually fall into two categories: pages that should be deleted (truly obsolete, no longer relevant) and pages that should have internal links added (still valuable but disconnected from the navigation or content structure).
The internal link structure — how pages connect to each other — is critical for SEO. Pages with many internal links pointing to them tend to inherit authority and rank better. Pages buried at depth 5+ from the homepage struggle. The Crawl Depth column in Screaming Frog shows how many clicks each URL is from the entry point; pages at depth 5 or deeper deserve a review of whether they need a more direct internal link path.
Redirect chains and their impact
A redirect chain is when URL A redirects to URL B, which redirects to URL C, possibly through more hops. Chains slow page load (each redirect is a round trip), waste crawl efficiency, and can reduce link equity transfer. Google generally follows up to 5 redirects in a chain; beyond that they stop following.
The Redirect Chains report (in the Reports menu) lists every chain detected during the crawl with their hops. The fix is straightforward: collapse each chain into a single direct redirect from the original URL to the final destination. After a couple of site migrations, chains accumulate; a quarterly cleanup keeps them manageable.
Missing metadata and on-page SEO elements
Screaming Frog analyzes title tags, meta descriptions, H1 tags, and structured data. The relevant reports:
Page Titles → Missing, Duplicate, Over Limit (>60 chars), Below Limit (<30 chars), Multiple. Each row tells you which pages need attention.
Meta Descriptions → same breakdown.
H1 → Missing, Duplicate, Multiple. Pages should have exactly one H1 that reflects the page topic.
Each page should have a unique, descriptive title (50 to 60 characters, the keyword and a benefit), and a unique meta description (around 150 to 155 characters that compels the click). Duplicates dilute snippets and reduce CTR.
Exporting and prioritizing fixes
Screaming Frog exports to CSV for collaboration with developers and tracking. File → Export → choose the report (Response Codes, Redirect Chains, Duplicate Titles, etc.). The export pattern that works:
One spreadsheet per audit. Tabs for each issue category. For each row: URL, issue, recommended action, owner, status, date fixed.
Prioritize by impact. Critical (blocking crawl or indexation): 4xx errors, 5xx errors, robots.txt blocking pages you want indexed, sitemap errors, broken canonicals. High (impacting rankings): redirect chains on high-traffic URLs, duplicate content on commercial pages, orphan pages with commercial intent. Medium (best practices): missing or duplicate meta descriptions, title tag length issues. Low (polish): minor structured data warnings, image alt text gaps.
Don’t try to fix everything at once. The 80/20 of any audit is usually 10 to 20 critical issues that produce most of the recovery; address those first.
Integration with Search Console and GA4
Screaming Frog tells you what is on your site; Google Search Console tells you what Google has actually crawled and indexed. The two are complementary.
If Screaming Frog shows 50 pages returning 404 but GSC shows 800 “Excluded” pages in the Pages report, your audit underestimated the problem. Cross-reference the two: Screaming Frog’s crawl is the on-site truth, GSC’s reports are Google’s perspective.
The paid version of Screaming Frog can connect directly to GSC and GA4 APIs to enrich crawl data with impressions, clicks, sessions, and conversions per URL. The combined view lets you prioritize by traffic value: a 404 on a URL that used to drive 1,000 monthly sessions is a higher priority than a 404 on a URL that has never received traffic.
Mobile-first crawling
Screaming Frog can crawl as Googlebot Smartphone (Configuration → User-Agent → Googlebot Smartphone). This is critical because Google primarily crawls and indexes the mobile rendering of pages — mobile-first indexing has been the default for new sites since 2019 and was extended to virtually all sites by July 2024.
If your mobile rendering differs from desktop — different content, different navigation, different internal links, different schema — the mobile crawl will surface those differences. Common findings: mobile templates that strip product details to “speed up” the page, mobile navigation menus that omit internal links present on desktop (creating mobile orphan pages), JavaScript that renders desktop-only blocks. The fix is content parity between mobile and desktop.
JavaScript rendering and modern frameworks
The paid version of Screaming Frog renders JavaScript, simulating how Google’s crawler handles modern frontend frameworks (React, Vue, Angular, Next.js). Without JavaScript rendering, a single-page application looks empty to a crawler — and to Google’s initial indexing pass.
Compare the rendered HTML to the raw HTML. If critical content (product descriptions, internal links, schema) appears only in the rendered version, you depend on Google’s JavaScript rendering pipeline to surface it. Google does render JavaScript, but rendering is queued and slower than HTML parsing. Server-side rendering or static generation surfaces the content immediately to the first crawl pass and tends to be more reliable for SEO than pure client-side rendering.
Comparing crawls over time: catching regressions
The real power of technical audits emerges when you run them regularly and compare results over time. Screaming Frog lets you save crawl files (.seospider format) and reload them for comparison. The discipline:
Run a full crawl quarterly at minimum, monthly for active sites. Save the crawl file with a date stamp. Compare against the previous crawl: are total pages crawled growing? Are 4xx error counts decreasing? Are redirect chains under control? Are new orphan pages appearing?
Regressions are common after deployments. A developer changes URL structure without setting up redirects; an editor publishes 200 articles without internal linking; a plugin update breaks canonical tags across the site. Quarterly crawls catch these regressions while they are still cheap to fix.
The ROI of technical SEO audits
Technical SEO doesn’t have the visibility of content production or link building, but the foundation it provides determines how well everything else works. Sites that crawl cleanly, index efficiently, and connect their pages internally produce more value from every other SEO investment.
The recurring pattern in audits is that fixing technical issues alone — without producing any new content — typically produces a meaningful traffic lift over the following quarter, sometimes substantial. The lift comes from pages that were previously orphan or broken being discovered, ranked, and earning impressions; from snippet-level CTR improving as duplicate metadata is fixed; from crawl budget being spent on real content rather than 404s and redirect chains.
Make Screaming Frog audits a quarterly ritual. The compounding effect of catching regressions early and keeping the technical foundation clean is one of the highest-ROI investments in an SEO program.
LaFactory runs technical SEO audits, integrates the findings with Search Console and GA4 data, and ships the prioritized fix list to your dev team. Contact us to scope a quarterly audit cadence matched to your site size.