Deutsch Deutsch English English

Robots.txt Syntax - Fundamentals and Best Practices 2025

The robots.txt file is an important technical SEO element that allows website operators to tell search engine crawlers which areas of a website may be crawled and which may not. This file is located in the root directory of a domain and follows a specific syntax format.

Basic Syntax Rules

1. File Format and Location

The robots.txt file must:

  • Be stored in the root directory of the domain (e.g. https://example.com/robots.txt)
  • Be a plain text file
  • Be UTF-8 encoded
  • Use lowercase letters (robots.txt, not Robots.txt)

2. Basic Structure

User-agent: [Crawler-Name]
Disallow: [Path]
Allow: [Path]
Crawl-delay: [Seconds]
Sitemap: [URL]

User-Agent Directives

Targeting Specific Crawlers

User-agent: Googlebot
Disallow: /admin/

User-agent: Bingbot
Disallow: /private/

Targeting All Crawlers

User-agent: *
Disallow: /temp/

Common User-Agents

Crawler
User-Agent
Purpose
Google
Googlebot
Web Crawling
Google Images
Googlebot-Image
Image Indexing
Bing
Bingbot
Web Crawling
Yahoo
Slurp
Web Crawling
Facebook
facebookexternalhit
Link Preview

Disallow and Allow Directives

Using Disallow

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /temp/

Using Allow

User-agent: *
Disallow: /images/
Allow: /images/public/

Wildcards and Pattern Matching

User-agent: *
Disallow: /*.pdf$
Disallow: /temp/*
Disallow: /admin/

Crawl-Delay Directive

Controlling Crawling Speed

User-agent: *
Crawl-delay: 10

Crawler-Specific Delays

User-agent: Googlebot
Crawl-delay: 1

User-agent: Bingbot
Crawl-delay: 5

Sitemap Directive

Specifying XML-Sitemaps

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xml
Sitemap: https://example.com/sitemap-news.xml

Common Syntax Errors

1. Wrong Capitalization

❌ Wrong:

User-Agent: *
DisAllow: /admin/

✅ Correct:

User-agent: *
Disallow: /admin/

2. Missing Colons

❌ Wrong:

User-agent *
Disallow /admin/

✅ Correct:

User-agent: *
Disallow: /admin/

3. Spaces Before Colons

❌ Wrong:

User-agent : *
Disallow : /admin/

✅ Correct:

User-agent: *
Disallow: /admin/

4. Multiple User-Agent Blocks

❌ Wrong:

User-agent: *
Disallow: /admin/

User-agent: *
Disallow: /private/

✅ Correct:

User-agent: *
Disallow: /admin/
Disallow: /private/

Best Practices for Robots.txt

1. Avoid Complete Blocking

❌ Caution with:

User-agent: *
Disallow: /

2. Allow Important Areas

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Allow: /css/
Allow: /js/
Allow: /images/

3. Specify Sitemap URLs

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xml

4. Comments for Documentation

# Main robots.txt for example.com
# Last updated: 2025-01-21

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

# Sitemaps
Sitemap: https://example.com/sitemap.xml

Testing and Validation

1. Google Search Console

  • Use robots.txt tester
  • Check crawling status
  • Identify errors

2. Online Tools

  • Use robots.txt validators
  • Use syntax checkers
  • Test crawling simulation

3. Manual Tests

curl -A "Googlebot" https://example.com/robots.txt

Advanced Configurations

E-Commerce Websites

User-agent: *
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/
Disallow: /search?*
Allow: /products/
Allow: /categories/

Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://example.com/sitemap-categories.xml

Multilingual Websites

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap-de.xml
Sitemap: https://example.com/sitemap-en.xml
Sitemap: https://example.com/sitemap-fr.xml

Development/Staging Environments

User-agent: *
Disallow: /

# Only for internal tests
User-agent: InternalBot
Allow: /

Monitoring and Maintenance

1. Regular Review

  • Monthly syntax validation
  • Analyze crawling logs
  • Check sitemap status

2. Document Changes

  • Use version control
  • Keep change log
  • Inform team

3. Performance Monitoring

  • Monitor crawling frequency
  • Observe server load
  • Optimize crawl budget

Common Problems and Solutions

Problem: Crawlers Ignore Robots.txt

Solution:

  • Fix syntax errors
  • Specify User-Agent correctly
  • Adjust crawl-delay

Problem: Important Pages Not Crawled

Solution:

  • Add Allow directives
  • Review Disallow rules
  • Update sitemap

Problem: Too Many Crawling Requests

Solution:

  • Increase crawl-delay
  • Block unnecessary areas
  • Optimize crawl budget

Robots.txt Checklist

  • ☐ File stored in root directory
  • ☐ UTF-8 encoding used
  • ☐ Syntax correct (capitalization)
  • ☐ Colons after directives
  • ☐ No spaces before colons
  • ☐ Sitemap URLs specified
  • ☐ Important areas allowed
  • ☐ Comments for documentation
  • ☐ Validated with tools
  • ☐ Tested in GSC

Related Topics

Last updated: October 21, 2025