Web Scraping at Robots.txt : Meilleures pratiques

What is robots.txt?

The robots.txt file is a standard used by websites to communicate with web crawlers and bots. It specifies which parts of the site can or cannot be accessed by automated systems. Although primarily designed for search engines, robots.txt also impacts web scraping practices.

Objectif

The primary goal of robots.txt is to instruct web crawlers (like those from search engines) which pages or sections of a website they are allowed to crawl or index. This can help prevent certain content from appearing in search engine results, manage server load, and control the accessibility of private or sensitive information. With it, site administrators control and manage the activities of web crawlers, preventing overloads and protecting sensitive data.

Localisation

The robots.txt file must be placed in the root directory of the website. For instance, it should be accessible via http://www.example.com/robots.txt.

Format

The file consists of simple text and follows a basic structure. It includes directives that specify which user agents (bots) should follow which rules.

Common Directives:

• User-agent

  Defines which web crawler the following rules apply to.
  For example: User-agent: *
  The asterisk (*) is a wildcard that applies to all bots.

• Disallow

  Specifies which paths or pages a crawler should not access.
  For example: Disallow: /private/
  This tells bots not to crawl any URL that starts with /private/.

• Allow

  Overrides a Disallow directive for specific paths.
  For example: Allow: /private/public-page.html
  This permits crawlers to access public-page.html even if /private/ is disallowed.

• Crawl-delay

  Sets a delay between requests to manage the load on the server.
  For example: Crawl-delay: 10

• Sitemap

  Indicates the location of the XML sitemap to help crawlers find and index pages more efficiently.
  For example: Sitemap: http://www.example.com/sitemap.xml

Example of robots.txt File

User-agent: *
Disallow: /private/
Allow: /private/public-page.html
Crawl-delay: 12
Sitemap: http://www.example.com/sitemap.xml

Additional Considerations

1. Some search engines have a size limit for the robots.txt file, usually 500KB. Ensure the file does not exceed this limit.
2. The robots.txt file should use UTF-8 encoding. Using other encodings may prevent correctly parsing.
3. Some crawlers (like Googlebot) support the use of wildcards in Disallow et Allow directives (e.g., * for any characters, $ for the end of a string).

   Disallow: /private/*
   Disallow: /temp/$

4. The robots.txt file is case-sensitive. For example, /Admin/ et /admin/ are different paths.
5. People can use the # symbol to add comments in the file, which are ignored by crawlers but can help administrators understand and maintain the file.

   # Prevent all crawlers from accessing admin pages
   User-agent: *
   Disallow: /admin/

6. Before applying the robots.txt file to a production environment, use tools (such as the robots.txt Tester in Google Search Console) to test the rules and ensure they work as expected.
7. For large websites or those with dynamic content, it might be necessary to dynamically generate the robots.txt file. Ensure the generated file is always valid and includes all necessary rules.
8. Not all crawlers obey the robots.txt file rules, so additional measures (like server firewalls, IP blacklists, etc.) may be necessary to protect sensitive content for malicious crawlers.
9. If you want to prevent search engines from indexing specific pages but allow crawlers to access them to fetch other content, use the noindex meta tag instead of Disallow.

   <meta name="robots" content="noindex">

10. Try to keep the robots.txt file straightforward and avoid overly complex rules. Complex rules can be difficult to maintain and may lead to potential parsing errors.

How robots.txt Affects Web Scraping

1. Guidelines for Crawlers

   The primary function of robots.txt is to provide instructions to web crawlers about which parts of the site should not be accessed. For instance, if a file or directory is disallowed in robots.txt, crawlers are expected to avoid those areas.

2. Respect for robots.txt

   • Ethical Scraping: Many ethical web scrapers and crawlers adhere to the rules specified in robots.txt as a courtesy to site owners and to avoid overloading the server.
   • Legal Considerations: While not legally binding, ignoring robots.txt can sometimes lead to legal issues, especially if the scraping causes damage or breach of terms of service.

3. Disallowed vs. Allowed Paths

   • Disallowed Paths: These are specified using the Disallow directive. For example, Disallow: /private-data/ means that all crawlers should avoid the /private-data/ directory.
   • Allowed Paths: If certain directories or pages are allowed, they can be specified using the Allow directive.

4. User-Agent Specific Rules

   File of robots.txt can specify rules for different crawlers using the User-agent directive.

   For example:

   User-agent: Googlebot
Disallow: /no-google/

   This blocks Googlebot from accessing /no-google/ but allows other crawlers.

5. Server Load

   By following robots.txt guidelines, scrapers reduce the risk of overloading a server, which can happen if too many requests are made too quickly.

6. Not a Security Mechanism

   File of robots.txt is not a security feature. It’s a guideline, not a restriction. It relies on crawlers respecting the rules set out. Malicious scrapers or those programmed to ignore robots.txt can still access disallowed areas.

7. Compliance and Best Practices

   • Respect robots.txt: To avoid potential conflicts and respect website operators, scrapers should adhere to the rules defined in robots.txt.
   • Consider robots.txt Status: Always check robots.txt before scraping a site to ensure compliance with the site’s policies.

Common Misconceptions About robots.txt

1. robots.txt is Legally Binding

   robots.txt is not a legal contract but a protocol for managing crawler access. While it’s crucial for ethical scraping, it does not legally enforce access restrictions.

2. robots.txt Prevents All Scraping

   robots.txt is a guideline for bots and crawlers but does not prevent all forms of scraping. Manual scraping or sophisticated tools may still access restricted areas.

3. robots.txt Secures Sensitive Data

   robots.txt is not a security feature. It’s intended for managing crawler access rather than securing sensitive information.

How to Scrape Pages from Website with robots.txt

1. Preparing for Scraping

Setting up your environment

Install necessary Python libraries:

import requests
from bs4 import BeautifulSoup
import time

Choosing the right tools

• Requests: For making HTTP requests.
• BeautifulSoup: For parsing HTML and XML.
• Scrapy: A comprehensive web scraping framework.
• Sélénium : For interacting with dynamically loaded content.

Assessing the website’s terms of service

Review the website’s terms of service to ensure your actions comply with their policies. Some websites explicitly forbid scraping.

2. Scraping with Caution

Fetching and parsing robots.txt

First, check the robots.txt file to understand the site’s crawling rules:

response = requests.get('https://example.com/robots.txt')
robots_txt = response.text

def parse_robots_txt(robots_txt):
    rules = {}
    user_agent = '*'
    for line in robots_txt.split('\n'):
        if line.startswith('User-agent'):
            user_agent = line.split(':')[1].strip()
        elif line.startswith('Disallow'):
            path = line.split(':')[1].strip()
            rules[user_agent] = rules.get(user_agent, []) + [path]
    return rules

rules = parse_robots_txt(robots_txt)

Identifying allowed and disallowed paths

Determine which paths you can legally and ethically access based on the robots.txt directives:

allowed_paths = [path for path in rules.get('*', []) if not path.startswith('/')]

Handling disallowed paths ethically

If you need data from disallowed paths, or want to scrape website protected by robots.txt, consider the following options:

• Contact the website owner: Request permission to access the data.
• Use alternative methods: Explore APIs or public data sources.

3. Alternative Data Access Methods

APIs and their advantages

Many websites offer APIs that provide structured access to their data. Using APIs is often more reliable and respectful than scraping.

Public data sources

Look for publicly available data that might meet your needs. Government websites, research institutions, and open data platforms are good places to start.

Data sharing agreements

Reach out to the website owner to negotiate data sharing agreements. This can provide access to data while respecting the site’s policies.

4. Advanced Techniques

Scraping dynamically loaded content

Use Selenium or similar tools to scrape content that is loaded dynamically by JavaScript:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://example.com')
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')

Using headless browsers

Headless browsers like Headless Chrome or PhantomJS can interact with web pages without displaying a user interface, making them useful for scraping dynamic content.

Avoiding detection and handling rate limits

Rotate user agents, use proxies, and implement delays between requests to mimic human behavior and avoid being blocked.

OkeyProxy is a powerful proxy provider, supporting automatic rotation of residential IPs with high quality. With ISPs offering over 150M+ IPs worldwide, you can now register and receive a 1GB free trial!

Start Test Excellent Proxies Now!