Python Web Scraping with BeautifulSoup

Fetch a page with requests, pick out the data with BeautifulSoup and save it to CSV without getting blocked.

By Dr. Dev2026-03-097 min read
🕷️

Install

You only need two packages: requests downloads the page and beautifulsoup4 reads the HTML.

pip install requests beautifulsoup4

Fetch a page

Send a normal browser-like User-Agent and always set a timeout so your script never hangs.

import requests

url = "https://example.com/products"
headers = {"User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"}
html = requests.get(url, headers=headers, timeout=15).text

Extract the data

Pass the HTML to BeautifulSoup and select elements with CSS selectors, the same ones you use in the browser DevTools.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
rows = []
for item in soup.select(".product"):
    rows.append({
        "name": item.select_one("h2").get_text(strip=True),
        "price": item.select_one(".price").get_text(strip=True),
    })

Save to CSV

import csv

with open("products.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["name", "price"])
    w.writeheader()
    w.writerows(rows)

Scrape politely

Read the site's robots.txt and terms first, add a short delay between requests with time.sleep(1), and cache pages you already downloaded. Polite scrapers rarely get blocked.

Related articles