Deutsch Deutsch English English

Screaming Frog

What is Screaming Frog?

Screaming Frog is one of the leading SEO crawling tools that performs website analyses at a technical level. The tool simulates the behavior of search engine crawlers and identifies potential SEO issues that can affect visibility in search results.

Core Features of Screaming Frog

1. Website Crawling

Screaming Frog crawls websites systematically and captures:

  • All accessible URLs
  • response codes
  • Meta tags and HTML titles
  • Heading structure (H1-H6)
  • Internal and external links
  • Images and their alt texts

2. Technical SEO Analysis

The tool automatically checks:

  • duplicate content
  • Missing meta descriptions
  • error links
  • redirect flows
  • Canonical tag issues
  • index sitemap validation

3. internal SEO Assessment

Screaming Frog analyzes:

  • Keyword density
  • Title tag lengths
  • Meta description lengths
  • Heading hierarchy
  • Image optimization

Screaming Frog vs. Other SEO Tools

Criterion
Screaming Frog
Ahrefs Site Audit
SEMrush Site Audit
Crawling speed
Very fast
Medium
Medium
Detailed analysis
Very detailed
Detailed
Detailed
Cost
Free (up to 500 URLs)
From €99/month
From €119/month
Offline usage
Yes
No
No
Custom crawling
Yes
Limited
Limited

Installation and Setup

System Requirements

  • Windows, macOS, or Linux
  • Java Runtime Environment (JRE) 8 or higher
  • At least 4 GB RAM for large websites
  • Stable internet connection

Getting Started

  1. Download: Download the tool from the official website
  2. Installation: No installation required - directly executable
  3. Configuration: Adjust crawling settings
  4. Start: Enter URL and start crawling

Crawling Configuration

Basic Settings

  • Crawl mode: Spider (standard) or list mode
  • Crawl limit: Number of URLs to crawl
  • Crawl depth: Maximum click depth
  • User-Agent: Crawler identification

Advanced Options

  • JavaScript rendering: For dynamic content
  • Custom headers: Additional HTTP headers
  • Proxy settings: For anonymous crawling
  • Respect robots.txt: Compliance with robots file

Important Reports and Analyses

1. Internal Links Report

The Internal Links Report shows:

  • Link distribution within the website
  • Orphan pages (pages without internal links)
  • Link equity distribution
  • Internal link anchor texts

2. Images Report

The Images Report analyzes:

  • Missing alt texts
  • Oversized images
  • Image formats and sizes
  • Broken image links

3. Page Titles Report

The Page Titles Report identifies:

  • Duplicate title tags
  • Title tags with inappropriate length
  • Missing title tags
  • Keyword optimization in titles

4. Meta Descriptions Report

The Meta Descriptions Report shows:

  • Duplicate meta descriptions
  • Missing meta descriptions
  • Length issues
  • Keyword integration

Technical SEO Checks

1. HTTP Status Codes

Screaming Frog detects:

  • 404 errors (Not Found)
  • 500 server errors
  • 301/302 redirects
  • Soft 404 errors

2. Redirect Analysis

The tool analyzes:

  • Redirect chains
  • Redirect loops
  • Missing redirects
  • 301 vs. 302 redirects

3. Canonical Tag Analysis

Screaming Frog checks:

  • Self-referencing canonicals
  • Canonical chains
  • Missing canonical links
  • Cross-domain canonicals

On-Page SEO Optimization

1. Content Analysis

The tool evaluates:

  • Content length
  • Keyword density
  • Heading structure
  • Duplicate content

2. URL Structure

Screaming Frog analyzes:

  • URL lengths
  • Parameters in URLs
  • URL hierarchy
  • Descriptive URLs

Export and Reporting

1. CSV Export

  • All crawling data as CSV
  • Filterable and sortable data
  • Integration with Excel or Google Sheets
  • Custom export options

2. sitemap production

  • Automatic sitemap creation
  • Priority and frequency settings
  • Last-modified dates
  • GSC integration

3. Visualization

  • Site architecture diagrams
  • Link map visualization
  • Crawl depth analysis
  • Orphan page identification

Best Practices for Screaming Frog

1. Regular Audits

  • Weekly crawling routines
  • Monthly deep-dive analyses
  • Pre- and post-launch checks
  • Competitor analyses

2. Filters and Segments

  • URL pattern filters
  • Status code filters
  • Content-type segments
  • Create custom filters

3. mass processing

  • Bulk URL export
  • Mass edit functions
  • Batch processing
  • Automated workflows

Integration with Other Tools

1. Google Search Console

  • GSC data import
  • Coverage problem analysis
  • Performance data correlation
  • Indexing status check

2. Google Analytics

  • Traffic data integration
  • Conversion path analysis
  • User journey mapping
  • Performance correlation

3. Ahrefs and SEMrush

  • Backlink data integration
  • Keyword ranking correlation
  • Competitor analysis
  • Domain authority checks

Common Problems and Solutions

1. Crawling Limits

Problem: Tool stops on large websites

Solution:

  • Increase crawl limits
  • Segmented crawling
  • Use list mode
  • Use Pro version

2. JavaScript Rendering

Problem: Dynamic content is not captured

Solution:

  • Enable JavaScript rendering
  • Adjust rendering timeout
  • Headless browser configuration

3. Memory Issues

Problem: Tool becomes slow or crashes

Solution:

  • Increase RAM
  • Reduce crawl limits
  • Regular restarts
  • Use 64-bit version

Screaming Frog SEO Spider Pro Features

1. Advanced Crawling Options

  • Unlimited URL limits
  • Custom extraction
  • Log file analysis
  • Database integration

2. API Integration

  • RESTful API
  • Webhook support
  • Automated crawling
  • Third-party integration

3. Team Features

  • group access
  • Shared configurations
  • Centralized reporting
  • User management

Pricing and Licensing

Free Version

  • Up to 500 URLs
  • Basic crawling features
  • Standard reports
  • Community support

Pro Version

  • Unlimited URLs
  • Advanced features
  • API access
  • Priority support
  • Cost: £149/year

Frequently Asked Questions about Screaming Frog

Question
Answer
What does Screaming Frog actually crawl and capture on a website?
Screaming Frog crawls websites systematically and captures all accessible URLs along with their HTTP status codes, meta tags and title tags, heading structure from H1 to H6, internal and external links, and images including their alt texts. By simulating search engine crawler behavior, it surfaces technical SEO issues that can affect visibility in search results. This makes it useful both for full-site audits and for checking specific technical dimensions after launches or migrations.
What is the difference between the free version and Screaming Frog SEO Spider Pro?
The free version allows crawling of up to 500 URLs with basic crawling features, standard reports, and community support. The Pro version removes that URL limit and adds advanced features such as unlimited crawling, custom extraction, log file analysis, database integration, RESTful API and webhook support, automated crawling, and team capabilities like multi-user access and shared configurations. Pro is listed at £149 per year and includes priority support, which is aimed at larger sites and teams that need automation and deeper workflows.
What system requirements and setup steps are needed to run Screaming Frog?
Screaming Frog runs on Windows, macOS, or Linux and requires a Java Runtime Environment (JRE) 8 or higher. For large websites you should plan for at least 4 GB of RAM and a stable internet connection. Getting started is straightforward: download the tool from the official website, run it without a traditional installation step because it is directly executable, adjust crawling settings, then enter a start URL and begin the crawl.
When should I use Spider mode versus list mode for crawling?
Spider mode is the standard crawl mode: the tool discovers URLs by following links from a seed URL and respects settings such as crawl limit, maximum click depth, and User-Agent. List mode is better when the crawl should stop on large sites or when you already have a defined URL set—for example when segmenting a big site or rechecking a known list. Combined with crawl limits, segmented crawling, and the Pro version for higher capacity, list mode helps avoid incomplete audits caused by free or memory-related crawling limits.
Which Screaming Frog reports are most useful for technical and on-page SEO?
The Internal Links Report shows link distribution, orphan pages without internal links, link equity distribution, and internal anchor texts. The Images Report flags missing alt texts, oversized images, formats and sizes, and broken image links. The Page Titles Report finds duplicate, missing, or poorly lengthed title tags and keyword issues in titles, while the Meta Descriptions Report covers duplicates, missing descriptions, length problems, and keyword integration. Alongside these, technical checks cover HTTP status codes including 404, 500, 301/302 and soft 404s, redirect chains and loops, and canonical issues such as self-referencing tags, chains, missing tags, and cross-domain canonicals.
How does Screaming Frog compare to Ahrefs Site Audit and SEMrush Site Audit?
According to the comparison on this page, Screaming Frog is rated as very fast for crawling and very detailed in analysis, while Ahrefs and SEMrush Site Audit are described as medium speed and detailed. Cost differs as well: Screaming Frog offers a free tier up to 500 URLs, whereas Ahrefs starts from €99 per month and SEMrush from €119 per month for their site audit products in this comparison. Screaming Frog also supports offline usage and custom crawling more flexibly, while the other two tools are listed as not supporting offline usage and offering only limited custom crawling.
What should I do if JavaScript content is missing or the crawler runs out of memory?
If dynamic content is not captured, enable JavaScript rendering, adjust the rendering timeout, and configure the headless browser so rendered pages are included in the crawl. If the tool becomes slow or crashes, increase available RAM, reduce crawl limits, restart regularly, and use the 64-bit version. For very large sites that hit crawling limits, increase the crawl limit, segment the crawl, switch to list mode, or move to the Pro version so the audit can complete without stopping mid-run.