Discover the Website
Follow crawlable internal links or process a supplied URL list to build an auditable view of the site.
Side Info Heading
Side Info Text
Gallery Heading
Follow Us
GuestPost.UK Technical SEO Playbook
Screaming Frog SEO Spider Guide: Technical SEO Audits, Crawling & Site Migrations
A practical command centre for crawling websites, finding broken links, auditing redirects, reviewing metadata, diagnosing indexability and planning safer migrations. Learn which configuration to use, which reports matter and how to turn crawl data into a prioritised technical SEO audit.
✓500 URLs free✓300+ issue checks✓Migration workflows✓GuestPost.UK methods
SFSEO Spider Command Centre
Audit Map
Crawl a WebsiteSpider mode and configuration
Find Broken LinksResponse codes and inlinks
Audit Page ElementsTitles, descriptions and headings
Check IndexabilityRobots, canonicals and directives
Find Duplicate ContentExact and near duplicates
Render JavaScriptIntegrated Chromium crawler
Audit a MigrationList mode and crawl comparison
Extract Custom DataXPath, CSSPath and regex
Technical audit map
What do you need to investigate?
Start with the technical question, configure the smallest useful crawl and follow the evidence back to the affected URL, its source and the underlying template or rule.
300+Issues checked
Run a Complete CrawlChoose scope, storage, user agent and rendering
Repair Broken Journeys4xx errors, 5xx errors and no-response URLs
Clean RedirectsChains, loops, temporary redirects and migration maps
Review Page ElementsTitles, descriptions, H1s, H2s and images
Diagnose IndexabilityRobots directives, canonicals and blocked resources
Find Content ProblemsDuplicates, similarity, word count and spelling
Validate a Site MigrationURL lists, redirect mapping and before-after checks
Connect and Extract DataAPIs, custom extraction, automation and exports
The free version currently crawls up to 500 URLs. Advanced configuration, saved crawls, JavaScript rendering, crawl comparison and other professional features require a licence; confirm current access on the official pricing page.
Crawler overview
What Screaming Frog SEO Spider Actually Does
SEO Spider is desktop crawling software for Windows, macOS and Linux. It discovers URLs and their relationships, requests pages and resources, extracts technical signals and organises findings into tabs, filters, issues, reports and bulk exports. It reveals evidence; an SEO professional still needs to interpret impact and recommend the correct fix.
Collect Technical Signals
Record status codes, directives, canonicals, metadata, headings, content, links, images and more.
Isolate the Problem
Use tabs, filters and issue summaries to narrow thousands of URLs into relevant investigation groups.
Build the Action List
Export affected URLs with sources and supporting fields for content, development or migration teams.
Start at a URL and follow eligible links through the configured crawl scope.
Upload or paste an exact URL set for redirects, migrations and controlled checks.
Keep crawl information in RAM for suitable smaller or faster local projects.
Use disk-based storage for larger crawls, saved projects and practical continuity.
NOTE
A crawler does not see the whole of Google
SEO Spider shows what its configured crawler can discover and request. Combine it with XML sitemaps, Google Search Console, analytics, server logs and manual checks to find orphaned, indexed or valuable URLs outside the normal link graph.
Crawl configuration
Configure the Crawl Before You Press Start
A technically correct crawl begins with scope. Decide which hostname, subdomains, folders, parameters, resources and rendering method belong in the investigation. An oversized crawl wastes time; an undersized crawl can hide the exact issue you need to diagnose.
Starting Point
Choose Spider or List Mode
Use Spider mode to discover a connected website and List mode when you already have the exact URLs to test.
Best for: Matching discovery behaviour to a full audit, redirect check, migration list or targeted validation.
Boundaries
Set the Crawl Scope
Control whether the crawler stays on one subdomain or folder and whether linked subdomains, sitemaps and external resources are included.
Best for: Avoiding incomplete audits and preventing unrelated environments from consuming the crawl.
Crawler Identity
Select the User Agent
Choose the appropriate crawler identity when investigating bot-specific output, access rules or parity between search engines and users.
Best for: Reproducing crawler behaviour without assuming every user agent receives identical HTML.
Access
Respect or Test Robots Rules
Default behaviour should reflect real crawler access. Custom robots testing can support diagnosis but does not change what search engines are permitted to crawl.
Best for: Separating discovery problems from robots.txt restrictions and safe staging tests.
JavaScript
Choose HTML or JavaScript Rendering
Use the fastest method that reveals the necessary content and links; render JavaScript when critical output depends on client-side execution.
Best for: Comparing source and rendered experiences on React, Vue, Angular and other dynamic websites.
Capacity
Choose the Storage Mode
Database storage is generally suited to larger saved crawls on an SSD, while memory storage depends more directly on available RAM.
Best for: Preventing a large crawl from failing because the machine and storage configuration were overlooked.
Courtesy
Control Crawl Speed
Reduce threads or requests when a server is fragile, rate-limited or shared with live users; schedule intensive work during suitable periods.
Best for: Collecting evidence without creating avoidable load or triggering security controls.
Private Sites
Handle Authentication Carefully
Crawl authorised staging or protected environments with the correct credentials while preventing accidental exposure and external indexing.
Best for: Pre-launch audits where the live website cannot yet reveal the new templates and URLs.
Repeatability
Save the Configuration
Store proven settings and crawl data so recurring audits use comparable scope, filters and evidence instead of an improvised setup.
Best for: Monthly monitoring, team handovers and defensible before-and-after comparisons.
Write down the issue, affected market and evidence needed before configuring the crawler.
Choose hostname, folders, subdomains, resources, user agent and rendering method.
Crawl representative pages and confirm expected content, links and status information.
Complete the crawl, save the project and document the date and configuration.
Response codes and redirects
Find Broken Links, Server Errors and Redirect Waste
Response-code reports reveal whether discovered URLs return content, redirect elsewhere, fail or cannot be reached. The strongest repair workflow uses the Inlinks information to identify every source page, then fixes the source rather than merely recording the destination error.
Broken
Client Errors
Locate internal or external URLs returning 4xx responses, including missing pages that still receive links from crawlable content.
Action: Restore, redirect or remove the source link according to the page’s purpose and available replacement.
Server
Server Errors
Identify pages and resources returning server-side failures and investigate whether the problem is persistent, intermittent or crawl-load related.
Action: Confirm with server monitoring and logs, repair the cause and retest under realistic load.
Connection
No Response URLs
Review timeouts, DNS failures, connection refusals, malformed URLs and blocked requests that do not produce an HTTP response.
Action: Retest representative examples and distinguish invalid links from network, firewall or rate-limit behaviour.
Permanent
Permanent Redirects
Find internal links that still point to redirected URLs and replace them with the final preferred destination where appropriate.
Action: Link directly to the canonical destination and preserve redirects needed for users and old external links.
Temporary
Temporary Redirects
Review 302 and 307 responses to ensure the temporary status matches the intended long-term behaviour.
Action: Keep genuinely temporary routing; change permanent moves to an appropriate permanent response.
Efficiency
Redirect Chains
Trace URLs through multiple hops and identify the original sources still linking into an unnecessarily long route.
Action: Update internal links and consolidate safe redirects to the final relevant destination.
Critical
Redirect Loops
Detect circular routing where a crawler or visitor cannot reach a final content response.
Action: Correct the rule order or mapping, then test every affected entry URL and final destination.
Sources
Inlinks and Source URLs
Use the lower Inlinks tab or bulk export to see exactly which pages, elements and anchor text lead to a problematic URL.
Action: Repair the component or content source so the problem is removed across every affected page.
External
Broken External Links
Check outgoing citations, partner pages and referenced resources that no longer resolve or now lead somewhere unsuitable.
Action: Replace with a strong current source, update the destination or remove the link when it no longer helps readers.
Open Response Codes and isolate internal 4xx, 5xx, redirects or no-response URLs.
Find every source page, link position, anchor and template creating the journey.
Restore, redirect, replace or remove according to intent, relevance and value.
Confirm direct 200-status journeys and check that no new chain or broken destination remains.
AUDIT
Fix causes, not spreadsheet rows
Get a Technical SEO Audit →Hundreds of broken links may come from one header, footer, product template or content component. Group URLs by source and pattern before assigning hundreds of individual fixes.
Page elements
Audit Titles, Meta Descriptions, Headings and Images at Scale
Screaming Frog extracts essential page elements into separate tabs and filters, making it easier to find missing, duplicated or unusual patterns across thousands of URLs. Treat length warnings as prompts for review rather than universal rules: clarity, uniqueness, intent and usefulness matter more than forcing every element into an arbitrary character count.
Search Display
Page Titles
Find missing, duplicate, unusually short or long title elements and review whether each one identifies a distinct page purpose.
Best for: Detecting template repetition and pages competing with weak or indistinguishable titles.
SERP Copy
Meta Descriptions
Isolate absent, repeated or problematic descriptions and decide where a stronger search-result summary could improve relevance and clicks.
Best for: Prioritising important landing pages rather than bulk-writing generic descriptions for every crawlable URL.
Main Heading
H1 Headings
Review missing, duplicate or multiple H1s in context and confirm the visible main heading accurately introduces the page.
Best for: Finding broken templates and vague headings—not enforcing one technical pattern without inspecting the design.
Structure
H2 Headings
Use extracted H2s to spot repeated template labels, missing topic structure and pages whose subheadings do not support the intended query.
Best for: Reviewing large content types before improving individual pages manually.
Address
URL Structure
Filter long, non-ASCII, duplicated or parameter-heavy addresses and investigate whether they reflect a wider architecture or faceting problem.
Best for: Finding inconsistent patterns while preserving useful existing URLs unless a migration is justified.
Media
Image Size and Alt Text
Locate large image files, missing alt attributes and image links that return errors, then review each case according to purpose.
Best for: Improving accessibility, load efficiency and image understanding without stuffing keywords into alternative text.
Sharing
Social Metadata
Use custom extraction or search to identify Open Graph and social-card fields on templates where consistent sharing previews matter.
Best for: Checking editorial and campaign pages before promotion across social platforms.
Documents
PDF Auditing
Find linked PDFs, review response behaviour, file size and available document metadata, and decide whether each resource remains useful.
Best for: Managing guides, reports and downloads that are part of the searchable website estate.
Templates
Pattern-Based Prioritisation
Group metadata and heading problems by directory, content type or shared value to identify the template causing the largest impact.
Best for: Turning thousands of warnings into a few scalable fixes for content and development teams.
Choose a page-element tab and isolate a specific missing, duplicate or unusual condition.
Compare directory, page type and repeated values to expose the common source.
Open representative pages and inspect the live search results before rewriting anything.
Correct the template or priority pages, then verify unique and useful output.
Crawlability and indexability
Understand Why a URL Can or Cannot Enter Search
The Indexability columns combine signals found during the crawl, but a crawl alone cannot confirm whether Google has indexed a page. Use these reports to diagnose access, directives and canonicalisation, then verify important URLs with Google Search Console and live search evidence.
Summary
Indexability Status
Separate potentially indexable pages from URLs excluded by response, directives, canonicalisation or crawler access.
Best for: Building an investigation queue while remembering that “indexable” does not mean “indexed”.
Access
Robots.txt Blocks
Find URLs or resources prevented from crawling and determine whether the restriction is intentional, inherited or outdated.
Best for: Diagnosing discovery and rendering gaps without confusing crawl blocking with a noindex directive.
Directive
Meta Robots and X-Robots-Tag
Review noindex, nofollow and other directives supplied in HTML or HTTP headers, including conflicting implementations.
Best for: Finding important pages unintentionally excluded and low-value pages incorrectly left open.
Preferred URL
Canonical Elements
Locate missing, multiple, conflicting, non-indexable or redirected canonical targets in HTML and supported HTTP headers.
Best for: Confirming that duplicate and parameterised pages consistently identify a valid preferred version.
Consistency
Self-Referencing Canonicals
Review whether indexable standalone pages consistently declare themselves where that is part of the site’s canonical strategy.
Best for: Detecting omissions across templates without assuming every non-self-canonical URL is wrong.
Discovery
XML Sitemap Auditing
Compare submitted URLs with crawl data and identify non-indexable, redirected, broken or missing canonical pages inside the sitemap.
Best for: Keeping sitemaps focused on important, canonical and indexable destination URLs.
Architecture
Orphan Pages
Connect XML sitemap, analytics and Search Console sources to find known URLs absent from the normal internal-link crawl.
Best for: Finding valuable or legacy pages that users and crawlers cannot reach through the website structure.
Clicks
Crawl Depth
Measure how many internal-link steps separate pages from the crawl start and identify priority destinations buried too deeply.
Best for: Improving access and hierarchy while considering alternate entry points and contextual relevance.
Verification
Search Console Confirmation
Use URL Inspection and indexing reports to check Google-specific canonical selection, rendering and indexing evidence.
Best for: Confirming crawler findings on priority URLs before declaring the issue fixed or unresolved.
RULE
Accessible, indexable and indexed are different states
A crawler may access a URL that carries noindex, and an indexable URL may still be excluded by Google for canonical, duplication, quality or demand reasons. Report the precise evidence rather than collapsing every problem into “not indexed”.
Content quality signals
Find Duplicate, Thin and Overlapping Pages
Content reports help isolate pages that share identical or similar copy, contain unexpectedly little indexable text or fall outside a site’s topical patterns. They do not judge factual accuracy, originality, expertise or commercial usefulness, so every finding needs direct page and search-intent review.
Duplicates
Exact Duplicate Pages
Identify pages whose analysed content matches exactly and investigate parameters, print versions, faceting or repeated publishing.
Best for: Choosing consolidation, canonicalisation, redirects or intentional differentiation according to purpose.
Similarity
Near-Duplicate Content
Use configured similarity analysis to find pages that overlap heavily without being exact copies.
Best for: Detecting location, product or topic pages differentiated only by a few substituted words.
Word Count
Low-Content Pages
Filter pages below a useful investigation threshold while recognising that tools, contact pages and focused answers may be intentionally concise.
Best for: Finding unexpected template failures and weak pages—not enforcing one minimum word count site-wide.
Extraction
Content Area Configuration
Include the main content and exclude repeated navigation or footer elements so similarity and word-count analysis reflect the page body.
Best for: Preventing shared templates from distorting duplicate-content and language checks.
Language
N-Gram Analysis
Inspect recurring phrases across pages or groups to understand vocabulary patterns and possible internal-link opportunities.
Best for: Finding repeated language and unlinked topic mentions without turning frequency into a keyword-density target.
Editorial
Spelling and Grammar
Use language checks to find likely mistakes at scale, then review names, brands, technical terms and regional usage manually.
Best for: Supporting editorial quality control without automatically replacing valid British English or specialist language.
SERP Review
Search Intent Overlap
Compare similar pages with the live result set and performance data to determine whether they compete or serve distinct needs.
Best for: Separating real cannibalisation from multiple useful pages that target different tasks or funnel stages.
Semantic
Semantic Similarity
Advanced similarity workflows can identify closely related pages and topical outliers for further content or migration analysis.
Best for: Prioritising human review across large sites, not allowing an automated similarity score to make publishing decisions.
Decision
Consolidate or Differentiate
Merge competing pages when one destination can satisfy the shared intent, or strengthen unique value when separate pages are justified.
Best for: Turning similarity findings into a clearer content architecture and internal-link plan.
CONTENT
Similarity is evidence, not the verdict
Improve SEO Content →Two pages can share language and still serve different users; two very different pages can compete for the same query. Combine crawl similarity with intent, rankings, clicks, conversions and direct page review.
Rendered websites
Crawl JavaScript Without Mistaking the Simulation for Google
JavaScript rendering uses an integrated Chromium environment to execute pages before extracting rendered HTML, links and content. It is essential when meaningful output depends on client-side code, but it is slower and more resource intensive than an HTML crawl. Compare both versions and confirm critical findings with Google’s own tools.
Rendering
JavaScript Rendering Mode
Render pages in headless Chromium before crawling the resulting DOM and collecting JavaScript-dependent elements.
Best for: React, Vue, Angular, single-page applications and pages whose source HTML is incomplete.
Source
Raw vs Rendered HTML
Compare the server response with the executed DOM to see whether key content, metadata, canonicals and links are added, removed or changed.
Best for: Finding parity gaps that make crawling and indexing depend entirely on rendering.
Discovery
JavaScript-Only Links
Identify navigational elements that appear after execution and confirm they use crawlable anchor elements with destination URLs.
Best for: Finding critical pages hidden behind click handlers, buttons or unsupported link patterns.
Resources
Blocked Resources
Review scripts, styles, APIs and other resources the rendering environment cannot request because of robots rules, errors or access controls.
Best for: Diagnosing incomplete rendering while avoiding unnecessary crawling of every third-party resource.
Visual QA
Rendered Page and Screenshots
Inspect the rendered view to confirm visible content, consent overlays, lazy loading and interactive components appear as expected.
Best for: Connecting extracted-data differences with what the rendering browser actually displayed.
Timing
AJAX Timeout
Adjust timing only when necessary for slow or delayed applications, then verify the setting does not hide genuine performance problems.
Best for: Pages that have not completed meaningful rendering within the configured wait period.
Parity
User Agent and Viewport
Use relevant crawler identities and viewport settings when checking mobile output, responsive behaviour or bot-specific differences.
Best for: Controlled comparisons without assuming one desktop render represents every search context.
Confirmation
Google Rendering Checks
Confirm important URLs using URL Inspection, Rich Results Test and other first-party evidence when crawl simulations reveal discrepancies.
Best for: Distinguishing a local rendering configuration issue from a problem Google actually encounters.
Resilience
Reduce Rendering Dependency
Where practical, provide essential content, metadata and crawlable navigation in reliable server output or robust rendering patterns.
Best for: Making important SEO signals available consistently to users and multiple search crawlers.
Record what the server supplies before executing client-side code.
Run JavaScript mode on representative templates with controlled settings.
Review links, text, directives, canonicals, structured data and status behaviour.
Test priority URLs and repair meaningful parity or access gaps.
International SEO
Audit Hreflang and International Page Relationships
SEO Spider can extract hreflang annotations from HTML, HTTP headers and XML sitemaps, then report common implementation problems. The audit must consider complete language and regional clusters: a single page can appear correct while its reciprocal target, canonical or status code breaks the relationship.
Annotations
Hreflang Discovery
Collect alternate-language relationships supplied in page markup, headers or XML sitemaps.
Best for: Building a site-wide view of declared language and regional alternatives.
Reciprocal
Missing Return Links
Find pages that reference an alternate URL whose cluster does not link back appropriately.
Best for: Repairing incomplete relationships that search engines may ignore.
Validation
Invalid Language Codes
Identify malformed, unsupported or inconsistent language-region values and compare them with the intended audience.
Best for: Correcting syntax without using country codes alone as language identifiers.
Destinations
Broken or Redirected Alternates
Locate hreflang targets that redirect, fail, are blocked or return no indexable content response.
Best for: Ensuring annotations resolve directly to live, accessible destination pages.
Canonical
Canonical Conflicts
Review alternates that canonicalise elsewhere or otherwise send contradictory preferred-URL signals.
Best for: Aligning international clusters with the site’s indexable canonical strategy.
Fallback
X-Default Review
Check whether a language-selector or fallback page is declared appropriately where the international experience needs one.
Best for: Guiding users whose language or region is not represented by a more specific alternate.
INTL
Validate the complete cluster, not one annotation
Open the Hreflang SEO Guide →Review every language version, return link, canonical, indexability state and destination response together. International SEO fails at the relationship level even when one page appears correctly marked.
Migration control
Protect Rankings During a Website Migration
A migration can change URLs, templates, navigation, rendering, canonicals and indexability at the same time. Screaming Frog is most useful when it preserves evidence before launch, tests the proposed destination set and compares the new crawl with the old one. The redirect spreadsheet is only one part of the process.
Before Launch
Preserve the Live-Site Baseline
Crawl and save the existing website before development replaces URLs, internal links or page elements.
Check: Status, indexability, titles, headings, canonicals, directives, depth, inlinks and structured data.
Pre-Launch
Crawl the Staging Website
Test the new environment with authorised access and a controlled scope while keeping it unavailable to public indexing.
Check: New templates, navigation, mobile output, rendered content and unintended noindex or robots rules.
URL Mapping
Map Every Valuable Old URL
Combine crawl, sitemap, analytics, Search Console and backlink data so important URLs are not lost simply because the link crawl missed them.
Action: Map each changed URL to the closest useful new destination, not automatically to the homepage.
Controlled Test
Validate Redirects in List Mode
Upload the legacy URL set and crawl only those addresses to confirm destination, response and redirect path.
Check: Direct permanent redirects, relevant final URLs, no loops, no chains and no soft-404 destinations.
Internal Routes
Update Internal Links
Replace links to redirected legacy URLs with direct links to final canonical destinations.
Check: Navigation, breadcrumbs, body links, image links, hreflang, canonicals and structured-data URLs.
Signal Match
Check Technical Parity
Compare old and new templates to find changes that were not part of the approved migration plan.
Check: Content, metadata, canonical targets, directives, language annotations, schema and indexable URL counts.
Discovery
Replace XML Sitemaps Cleanly
Publish new sitemaps containing only canonical, indexable destination URLs and retain the old URL evidence outside the submitted file.
Check: No redirected, blocked, broken, duplicate or staging URLs remain in submitted sitemaps.
Comparison
Compare Old and New Crawls
Use crawl comparison and change detection to isolate URLs, fields and issue counts that changed between saved crawls.
Check: New, missing and altered URLs rather than manually comparing two unrelated spreadsheets.
Launch Day
Run a Post-Launch Control Crawl
Recrawl priority templates and the mapped URL set immediately after DNS, routing and cache changes settle.
Check: Server responses, final HTML, rendering, analytics tags, conversion journeys and accidental staging references.
Save crawl data and merge all known URL sources before anything changes.
Audit new templates and validate each changed URL in List mode.
Test priority journeys, redirects, internal links, canonicals and indexability.
Watch Search Console, analytics, logs and rankings while defects are still recoverable.
RULE
Do not remove redirects because a crawl looks clean
Internal links can be updated quickly, but old bookmarks, citations, backlinks and search results may continue to request legacy URLs for years. Keep useful migration redirects while the old addresses still carry value or receive legitimate traffic.
Advanced analysis
Extract Custom Data, Connect APIs and Automate Repeat Audits
The standard tabs cover common technical signals. Paid features extend the crawler into a configurable audit workstation: extract page-specific fields, search source code, join external performance data, schedule repeatable crawls and export evidence for reporting. API quotas and third-party costs remain separate from the Screaming Frog licence.
Extraction
Custom Extraction
Collect values from HTML using XPath, CSSPath or regular expressions when standard tabs do not contain the field.
Use it for: Product details, author names, dates, breadcrumbs, stock status, schema fields and template markers.
Source Search
Custom Search
Find pages that contain or omit selected words, phrases, tags or code patterns in raw or rendered output.
Use it for: Analytics tags, old branding, legal copy, internal-link mentions and template QA.
Search Data
Google Search Console
Append clicks, impressions, position and inspection data to crawl URLs where the connected property and API permit it.
Use it for: Prioritising pages with search demand and finding known URLs outside the internal-link crawl.
Behaviour
Google Analytics
Join relevant traffic, engagement and conversion metrics to crawled landing pages for prioritisation.
Use it for: Finding valuable orphan pages and separating high-impact defects from unused URL noise.
Performance
PageSpeed Insights
Collect available Lighthouse and field-performance metrics through the API and connect them to crawlable templates.
Use it for: Grouping performance issues by page type while checking lab and field evidence separately.
Authority Data
Link-Metric Integrations
Connect supported link-data providers to combine crawl architecture with page and domain authority metrics.
Use it for: Prioritising broken URLs, redirects and orphan pages that have meaningful external links.
Prompt Data
AI Prompts and Embeddings
Connect supported AI providers to run configured prompts or embedding analysis against selected crawl data.
Use it for: Classification, language, intent, entities, alt-text assistance and semantic similarity—with manual review.
Automation
Scheduling and Command Line
Run recurring crawls with saved settings and automate exports for agreed monitoring workflows.
Use it for: Weekly health checks, release QA, client reporting and large crawls outside business hours.
Delivery
Bulk Exports and Looker Studio
Export focused issue sets, inlinks, redirect reports and crawl summaries instead of sending an unfiltered crawl dump.
Use it for: Giving developers the source URL, failing URL, evidence and expected outcome needed to act.
Crawl links, sitemaps, lists and connected sources to assemble the URL estate.
Join response, directives, canonicals, content, rendering and link relationships.
Append search, analytics, speed and link metrics for business-aware prioritisation.
Export the source, evidence, owner, recommended fix and recrawl result.
Practical playbooks
Six Screaming Frog Workflows Worth Saving
Repeatable configurations make the crawler more useful than a long list of one-off warnings. Save the scope and exports for each job, record assumptions, and recrawl the affected sources after implementation.
Technical Audit
Full Website Health Crawl
Spider the canonical production scope, connect sitemaps and Search Console, then group problems by template and impact.
Output: An evidence-backed priority list, not an exported catalogue of every available warning.
Release QA
Template Change Validation
Crawl a representative URL list before and after a release to compare headings, canonicals, directives, schema and links.
Output: A focused change report that separates intentional updates from regression defects.
Internal Links
Find Pages Needing More Support
Combine crawl depth, inlink count, anchor text and Search Console data to locate valuable pages with weak contextual support.
Output: Relevant source pages and natural anchor opportunities for each priority destination.
Content Estate
Consolidate Duplicate Topics
Group exact, near-duplicate and semantically similar pages, then inspect rankings, intent, links and conversion value.
Output: A keep, improve, merge, redirect or remove decision supported by evidence.
Link Campaigns
Audit Guest-Post Placement Pages
Use List mode to check that published placements resolve, remain indexable, retain the agreed link and point to the correct destination.
Output: A placement-quality report covering status, directives, canonical, anchor, rel attribute and destination.
Monitoring
Recurring Priority-URL Watch
Schedule a controlled list crawl of commercial, editorial and campaign URLs and compare it with the previous result.
Output: Fast alerts for new errors, redirects, noindex, canonical changes, missing content or lost links.
GP
Use crawl data to qualify link-building work
Explore Link-Building Services →Before outreach or placement renewal, verify that candidate and live pages are accessible, relevant, indexable and technically stable. Combine that evidence with editorial quality, traffic and topical fit.
Free and paid access
Screaming Frog Pricing and Licence Choice
The free version is a genuine crawler for small jobs, but it is restricted to 500 URLs per crawl and excludes many configuration, saving, comparison, integration and automation capabilities. A paid SEO Spider licence is currently £199 per user per year; confirm the live price and currency before purchasing.
| Capability | Free Version | Paid Licence | Practical Meaning |
|---|---|---|---|
| Crawl size | Up to 500 URLs per crawl | No software crawl limit* | Large crawls still depend on machine resources, storage, scope and server tolerance. |
| Core audits | Broken links, metadata, directives, duplicates, hreflang, sitemaps and visualisations | Included | The free version is useful for small sites and controlled URL samples. |
| Configuration | Restricted | Advanced crawl configuration | Paid access is better for precise scope, rendering, extraction and repeatability. |
| Saving and comparison | No saved-crawl workflow | Save, open, compare and detect changes | Essential for migrations, regression testing and recurring audits. |
| JavaScript rendering | Restricted | Included | Needed when important content or links depend on client-side rendering. |
| Custom analysis | Restricted | Custom search, extraction and JavaScript | Useful for site-specific templates, QA and data collection. |
| APIs and automation | Restricted | Integrations, scheduling and reporting | Useful for joining business data and running consistent monitoring. |
| Licence model | Free | Annual licence per user | One licensed user can install on multiple personally used devices, subject to the licence terms. |
Start Here
Choose Free for Small, Focused Crawls
Use it for compact websites, samples, quick broken-link checks and learning the core interface.
Good fit: A crawl that remains meaningful within 500 discovered URLs.
Professional
Choose Paid for Serious Audit Work
Use it when crawl size, saved projects, JavaScript, comparison, custom extraction, integrations or scheduling matter.
Good fit: Consultants, in-house teams, agencies and repeat technical SEO workflows.
Licensing
Count Users, Not Shared Computers
Licences are assigned per person, with bulk discounts listed for larger purchases on the official pricing page.
Good fit: Teams budgeting access for every person who will operate licensed features.
*
“Unlimited” is not infinite
The paid software removes the product’s 500-URL cap, but crawl capacity still depends on available memory, database storage, configuration, website size, rendering method and the server being crawled. Define scope before adding hardware.
Crawler comparison
Screaming Frog vs Sitebulb, Semrush and Ahrefs
These products overlap, but they are not interchangeable. Screaming Frog is a highly configurable desktop crawler; Sitebulb emphasises visual technical reporting; Semrush and Ahrefs provide cloud crawlers inside broader research platforms. The strongest choice depends on workflow, scale, collaboration and the data already used by your team.
| Tool | Operating Model | Strongest Use Case | Main Trade-Off |
|---|---|---|---|
| Screaming Frog | Desktop crawler for Windows, macOS and Linux | Deep configuration, migrations, controlled URL lists, extraction and technical investigation | Capacity uses local resources and collaboration needs an agreed file/report workflow. |
| Sitebulb | Desktop and cloud technical auditing options | Visual explanations, prioritised hints and stakeholder-friendly technical reporting | Teams should compare crawl flexibility, limits and reporting needs directly. |
| Semrush Site Audit | Cloud crawler inside a broad marketing platform | Recurring project monitoring connected to keyword, competitor and reporting tools | Less suited to some highly customised desktop-crawl and extraction workflows. |
| Ahrefs Site Audit | Cloud crawler inside an SEO research platform | Technical monitoring alongside backlink and organic-search research | Best value is realised when the wider Ahrefs platform is already part of the workflow. |
Best Control
Choose Screaming Frog
Best when you need exact crawl configuration, local data, List mode, migration validation, custom extraction or raw technical evidence.
Best Visuals
Choose Sitebulb
Best when clear visualisation, audit hints and accessible stakeholder explanations are central to delivery.
Best Platform Fit
Choose Semrush or Ahrefs
Best when cloud project monitoring should sit beside keyword, competitor, backlink and reporting data in one paid platform.
VERDICT
Final verdict: use Screaming Frog when investigation control matters most
It is especially strong for technical consultants, migrations, QA, URL-list testing and custom extraction. Choose a cloud suite when shared monitoring and connected research are more important, or combine the products when each solves a different stage of the workflow.
Common questions
Screaming Frog SEO Spider FAQs
These answers clarify the practical limits of the crawler and where additional data or professional judgement is still required.
Is Screaming Frog SEO Spider free?
Yes. The free version can crawl up to 500 URLs per crawl and includes many core checks. Advanced configuration, saved crawls, comparison, integrations and other professional features require a paid licence.
What counts towards the 500-URL limit?
Discovered internal HTML pages and eligible resources can consume the crawl allowance according to configuration. A website with fewer than 500 visible pages may still reach the limit because URLs are not the same as pages.
Is Screaming Frog cloud-based?
SEO Spider is installed desktop software for Windows, macOS and Linux. It can connect to cloud APIs and export reports, but the crawl itself runs using the configured machine and storage.
Can it crawl JavaScript websites?
Yes, the licensed version supports JavaScript rendering. Use rendering only where necessary, compare source with rendered output and provide enough time and resources for critical content to appear.
Can Screaming Frog find orphan pages?
Yes, when you connect other URL sources such as XML sitemaps, analytics and Search Console. A link crawl alone cannot discover a page with no crawlable route from the starting scope.
Does it show what Google has indexed?
No crawler can reproduce Google’s complete index. SEO Spider can identify potentially indexable URLs and connect relevant Search Console data, but important cases still need Google-specific verification.
Can it audit very large websites?
Yes, with suitable scope, database storage, hardware and server-friendly speed settings. Start with representative samples and priority sections before assuming every URL must be crawled in one run.
Is Screaming Frog safe to use?
It is a legitimate professional crawler, but any crawler can create unnecessary server load when configured aggressively. Crawl only authorised sites, set a considerate speed and coordinate large production audits.
Does Screaming Frog fix SEO problems?
No. It discovers and organises evidence. A person still needs to validate the issue, understand business impact, choose the correct repair and confirm the result with a recrawl and real performance data.
SEO
Need the crawl turned into an action plan?
Request a Professional SEO Audit →GuestPost.UK can combine Screaming Frog evidence with Search Console, analytics, content, architecture and backlink review, then prioritise fixes by risk, reach and commercial value.
Services
Guest Posting Sites
Free SEO Tools
Academy
Resources
Contact
Your trusted partner for Guest Posting, Link Building and AI-powered SEO. We help businesses grow authority, rankings and organic traffic with ethical, results-driven strategies.
💙 PayPal
💳 VISA
💳 Mastercard
🏦 Bank
🔒 SSL
© 2026. All rights reserved.