Python Web Scraping: Beautiful Soup, Scrapy, and Selenium

Web scraping extracts data from websites. Python has the richest scraping ecosystem — Beautiful Soup for parsing HTML, Scrapy for production crawling, Selenium for JavaScript-rendered pages, and Requests for HTTP.

Legality and Ethics

Before scraping any website:

1. Check robots.txt (https://example.com/robots.txt)
2. Review the site’s Terms of Service
3. Use a reasonable request rate (add delays)
4. Identify your scraper with a custom User-Agent
5. Cache responses to be polite

Beautiful Soup: HTML Parsing

Beautiful Soup parses HTML and XML documents into a navigable tree. Use it with requests for straightforward scraping.

from bs4 import BeautifulSoup
import requests

response = requests.get("https://books.toscrape.com/")
soup = BeautifulSoup(response.text, "html.parser")

# Find elements by tag and class
books = soup.find_all("article", class_="product_pod")
for book in books:
    title = book.h3.a["title"]
    price = book.find("p", class_="price_color").text
    rating = book.p["class"][1]
    print(f"{title}: {price} ({rating} stars)")

Navigation Methods

# Find one element
soup.find("h1")
soup.find(id="main-content")
soup.find("div", attrs={"data-product-id": "123"})

# Find all matching elements
soup.find_all("a")                        # all links
soup.find_all("a", href=True)             # links with href
soup.find_all("a", class_="nav-link")     # by class (note: class_, not class)
soup.find_all("li", limit=10)            # first 10

# CSS selectors
soup.select("div.content p")             # descendant
soup.select("div > p")                   # direct child
soup.select(".price_color")              # class
soup.select("#main")                     # id
soup.select("a[href^='https']")          # attribute starts with

# Traversing the tree
tag.parent
tag.children
tag.next_sibling
tag.previous_sibling
tag.find_next_sibling("p")

Extracting Data

# Text content (strips HTML tags)
element.get_text(strip=True)

# Attributes
element["href"]
element.get("href")
element["class"]  # returns list of classes

# Tag name
element.name  # e.g., "div", "a"

Scraping Etiquette and Legal Considerations

Responsible scraping requires more than just technical skill. Always identify your bot with a descriptive User-Agent string that includes a way to contact you. Set reasonable delays between requests — time.sleep(random.uniform(1, 3)) is a good baseline. Implement exponential backoff for rate-limited responses: if you get a 429 status code, wait 60 seconds, then 120, then 240. Some sites use honeypot traps — hidden links or form fields that only bots follow. Detecting and avoiding these requires inspecting the page structure carefully.

Handling CAPTCHAs and IP Blocks

When a site deploys CAPTCHA challenges, your scraper has several options. Use rotating residential proxies to distribute requests across IP addresses. Services like BrightData, ScrapingBee, or ScraperAPI provide proxy rotation as a managed service. For CAPTCHAs specifically, services like 2Captcha and Anti-Captcha can solve them programmatically, but this raises ethical and legal concerns. A better approach is to identify and use the site’s official API if one exists — most major platforms have APIs that provide structured data without scraping.

Scrapy: Production Crawling

Scrapy is a framework for large-scale web scraping — async requests, automatic throttling, data pipelines, and export formats.

Project Setup

pip install scrapy
scrapy startproject bookscraper
cd bookscraper
scrapy genspider books books.toscrape.com

Spider Definition

# spiders/books.py
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        # Extract books from the current page
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css("p.price_color::text").get(),
                "url": response.urljoin(book.css("h3 a::attr(href)").get()),
                "rating": book.css("p.star-rating::attr(class)").get().split()[-1],
            }

        # Follow pagination
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Data Pipelines

# pipelines.py — clean and validate scraped data
class BookPipeline:
    def process_item(self, item, spider):
        # Clean price
        if item.get("price"):
            item["price"] = float(item["price"].replace("£", ""))
        
        # Validate required fields
        if not item.get("title"):
            raise DropItem(f"Missing title in {item}")
        
        return item

Running the Spider

scrapy crawl books -o books.json
scrapy crawl books -o books.csv
scrapy crawl books -o books.jsonlines

Advanced Scrapy Settings

# settings.py
DOWNLOAD_DELAY = 1            # 1 second between requests
CONCURRENT_REQUESTS = 8       # parallel requests
CONCURRENT_REQUESTS_PER_DOMAIN = 4
AUTOTHROTTLE_ENABLED = True   # auto-adjust speed
ROBOTSTXT_OBEY = True
USER_AGENT = "MyScraper/1.0 (+https://example.com/bot)"
ITEM_PIPELINES = {
    "bookscraper.pipelines.BookPipeline": 300,
---

Selenium: Dynamic Content

Selenium controls a real browser — essential for JavaScript-rendered sites that make API calls or require interaction.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com/dynamic-page")

# Wait for JavaScript to render
wait = WebDriverWait(driver, 10)
element = wait.until(
    EC.presence_of_element_located((By.CLASS_NAME, "dynamic-content"))
)

# Interact with the page
driver.find_element(By.ID, "search-box").send_keys("Python")
driver.find_element(By.CSS_SELECTOR, "button.search").click()

# Wait for results
results = wait.until(
    EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".result-item"))
)

for result in results:
    print(result.text)

driver.quit()

Headless Mode

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")

driver = webdriver.Chrome(options=options)

Handling Infinite Scroll

# Scroll to the bottom repeatedly
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

Combining Tools

Different sites need different approaches:

API Detection

Check the Network tab in DevTools. Many “dynamic” sites load data via a hidden JSON API — much easier to scrape directly:

import requests

# Found by inspecting network requests
api_url = "https://example.com/api/v1/products?page=1"
headers = {"X-API-Key": "..."}
data = requests.get(api_url, headers=headers).json()

for product in data["products"]:
    print(product["name"], product["price"])

Anti-Scraping Mitigations

# Rotate User-Agents
headers = {
    "User-Agent": random.choice([
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...",
    ])
---

# Use proxies
proxies = {"http": "http://proxy1:8080", "https": "http://proxy1:8080"}

# Random delays
import time
time.sleep(random.uniform(1, 3))

# Handle cookies and sessions
session = requests.Session()
session.get("https://example.com")  # gets initial cookies

Best Practices

# Cache responses to avoid hammering the server
import requests_cache
requests_cache.install_cache("scrape_cache", expire_after=3600)

# Retry on failure
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retries = Retry(total=3, backoff_factor=1, status_forcelist=[500, 502, 503])
session.mount("http://", HTTPAdapter(max_retries=retries))
session.mount("https://", HTTPAdapter(max_retries=retries))

Summary

Beautiful Soup handles static HTML parsing, Scrapy scales to thousands of pages, and Selenium renders JavaScript. Check robots.txt, respect rate limits, cache aggressively, and detect hidden APIs to avoid unnecessary scraping. The right tool depends on the site’s complexity — start with the simplest approach and escalate only when needed.

Related: Python Requests HTTP | Python Error Handling

Storing Scraped Data in Databases

For large-scale scraping projects, SQLite provides a zero-configuration database that stores scraped results in a single file. Use sqlite3 to create tables matching your data schema, insert rows with parameterized queries to prevent SQL injection, and add unique constraints to avoid duplicate entries. For higher concurrency needs, PostgreSQL or MySQL handle multiple writer processes. Use ORM libraries like SQLAlchemy for programmatic schema management and migration support across database backends.

Data Extraction and Storage Patterns

Extracted data needs organized storage. Define a schema upfront using dataclasses or pydantic models that match your target data structure. Store results in CSV for simple datasets, SQLite for relational data with querying needs, or JSON Lines (one JSON object per line) for streaming large datasets. Add a unique hash to each record to prevent duplicate inserts during re-scrapes. Implement checkpointing — save progress periodically so a failed scrape resumes where it left off rather than starting over.

FAQ

Is web scraping legal?

Web scraping is generally legal if you respect robots.txt, comply with the site’s Terms of Service, do not bypass authentication, and do not collect personal data without consent. Scraping for competitive intelligence is usually OK; scraping copyrighted content for redistribution is not. Consult a lawyer for specific cases.

How do I avoid getting blocked while scraping?

Use polite delays between requests (1-3 seconds), rotate User-Agents, use rotating proxies for large-scale scraping, and respect the site’s rate limits. Cache responses locally to avoid repeat requests. For JavaScript-heavy sites, consider using Playwright over Selenium for better performance.

Which is better: Beautiful Soup or Scrapy?

Beautiful Soup is simpler and great for single-page scraping and prototyping. Scrapy is better for multi-page crawling, production pipelines, and large-scale data extraction. Use Beautiful Soup when you need quick results; use Scrapy when you need a maintainable, scalable scraping system.

How do I scrape websites that require login?

Use requests.Session() to maintain cookies across requests. Send a POST request to the login endpoint with credentials, then use the same session for subsequent requests. For JavaScript-based login flows, use Selenium or Playwright to automate the login process and extract session cookies.

Avoiding Anti-Scraping Countermeasures

Sites detect scrapers through request patterns, headers, and behavior. Rotate user agents and IP addresses using proxy pools or rotating residential proxies. Add random delays between requests to avoid rate limiting. Use session objects to maintain cookies and headers consistently. For JavaScript-heavy sites, headless browsers work but consume more resources. Consider using Playwright over Selenium for modern scraping — it is faster, supports browser contexts for isolation, and handles modern web frameworks better. Always check robots.txt and respect site terms of service.

What is the best approach for scraping paginated content?

For Scrapy, follow the “next page” link using response.follow(next_page, self.parse). For requests + Beautiful Soup, loop through page numbers in the URL pattern. For cursor-based pagination (infinite scroll), track the cursor parameter sent to the API endpoint.

For a comprehensive overview, read our article on Pip Command Not Found.