Web Scraping with BeautifulSoup
Published by Brayan K
When a website has data you want but no API, web scraping lets you download its HTML and extract exactly what you need. Combined with the requests library you already know, BeautifulSoup turns a tangled page into clean, structured data.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Download the page, then parse it
A web page is HTML: tags such as <h1> and <a>, with attributes such as class="price" or href="...", nested inside one another like a tree; HTML Elements & Attributes covers them. A scraper does two jobs. requests downloads the page, as HTTP Requests & APIs showed, and Beautiful Soup turns its HTML into a tree you can search. Beautiful Soup isn't part of Python: the package is called beautifulsoup4, and you import it as bs4.
pip install beautifulsoup4import requests
from bs4 import BeautifulSoup
resp = requests.get("https://books.toscrape.com/", timeout=10)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
print(soup.title.text)books.toscrape.com is a site made for practising scraping. This site's editor can't download pages, so the rest of the lesson parses HTML held in a string, which is also the easy way to work on a scraper: save one page and try your code on it without asking the site again. The editor loads Beautiful Soup by itself when your code imports it.
Beautiful Soup reads the text you give it
from bs4 import BeautifulSoup
html = """
<html>
<head><title>Laptops | Example Shop</title></head>
<body>
<h1>Laptops</h1>
<p class="intro">3 results</p>
</body>
</html>
"""
soup = BeautifulSoup(html, "html.parser")
print(soup.title.text)
print(soup.h1.text)
print(soup.find("p").text)
# ✅ Expected output:
# Laptops | Example Shop
# Laptops
# 3 resultssoup.title and soup.h1 are the first tags with those names, and .text is the text inside. Every example names its parser, "html.parser", the one that comes with Python. lxml and html5lib are parsers you install separately, and they can build different trees from broken HTML, so always name the one you mean: without lxml installed, BeautifulSoup(html, "lxml") stops with FeatureNotFound, Couldn't find a tree builder with the features you requested: lxml. Hand it the page's address instead of its HTML by mistake, and it parses the address as text: there are no tags in it to find.
find() gives one tag or None; find_all() and select() give a list
from bs4 import BeautifulSoup
html = """
<ul id="menu">
<li class="dish">Coffee <span class="cost">3.50</span></li>
<li class="dish special">Tea <span class="cost">2.75</span></li>
</ul>
"""
soup = BeautifulSoup(html, "html.parser")
print(soup.find("li"))
print(len(soup.find_all("li")))
print(soup.select_one("li.special span").text)
print([s.text for s in soup.select("#menu .cost")])
print(soup.find("h1"))
print(soup.find_all("h1"))
# ✅ Expected output:
# <li class="dish">Coffee <span class="cost">3.50</span></li>
# 2
# 2.75
# ['3.50', '2.75']
# None
# []find() and find_all() take a tag name, and attributes to match. select() and select_one() take a CSS selector, as in CSS Basics & Selectors: #menu .cost is every element with the class cost inside the element whose id is menu. So find_all("div.book") looks for a tag named div.book and finds nothing, where select("div.book") works. When nothing matches, find() and select_one() give None, and the text of None is an error:
from bs4 import BeautifulSoup
soup = BeautifulSoup("<div>hello</div>", "html.parser")
heading = soup.find("h1")
print(heading)
print(heading.text)
# ✅ Expected output:
# None
# ✅ Expected error:
# AttributeError: 'NoneType' object has no attribute 'text'It's the same None as a failed search in Regular Expressions (re), and the same fix: check before you use it.
from bs4 import BeautifulSoup
for html in ["<h1>Deals</h1>", "<div>hello</div>"]:
soup = BeautifulSoup(html, "html.parser")
heading = soup.find("h1")
if heading:
print(heading.text)
else:
print("no h1 on this page")
# ✅ Expected output:
# Deals
# no h1 on this pageThe opposite mistake is asking a list for its text: soup.find_all("li").text stops with an AttributeError saying a ResultSet has no attribute text, and asking whether you meant find(). Loop over the list instead. Regular Expressions ended by saying that real pages are better read with a parser; here is its HTML, parsed:
from bs4 import BeautifulSoup
html = "<b>bold</b> and <i>italic</i>"
soup = BeautifulSoup(html, "html.parser")
print(soup.find_all(["b", "i"]))
print(soup.get_text())
# ✅ Expected output:
# [<b>bold</b>, <i>italic</i>]
# bold and italicfind_all() with a list finds every tag named in it, in page order, and get_text() gives all the text with no tags left to trip over.
.text is everything inside, and it's always a string
from bs4 import BeautifulSoup
html = """
<li class="dish">
Coffee <span class="cost">3.50</span>
</li>
"""
soup = BeautifulSoup(html, "html.parser")
dish = soup.find("li")
print(repr(dish.text))
print(repr(dish.get_text(strip=True)))
print(repr(dish.get_text(" ", strip=True)))
print(dish.string)
print(dish.span.string)
# ✅ Expected output:
# '\n Coffee 3.50\n'
# 'Coffee3.50'
# 'Coffee 3.50'
# None
# 3.50.text keeps the page's line breaks and indentation. get_text(strip=True) strips each piece of text and joins them with nothing, so pass a separator to keep a space. .string is the text only when a tag holds exactly one thing; this list item holds text and a span, so its .string is None. Between tags, too, get_text() adds nothing unless you give it a separator:
from bs4 import BeautifulSoup
soup = BeautifulSoup("<p>Hello</p><p>World</p><address>12 High St<br>Leeds</address>", "html.parser")
print(soup.get_text())
print(soup.get_text(" "))
print(soup.address.get_text(", "))
# ✅ Expected output:
# HelloWorld12 High StLeeds
# Hello World 12 High St Leeds
# 12 High St, LeedsNumbers on a page arrive as text, so they compare and add like text:
from bs4 import BeautifulSoup
html = '<span class="seats">12</span><span class="seats">5</span>'
soup = BeautifulSoup(html, "html.parser")
seats = [s.text for s in soup.find_all("span")]
print(seats, max(seats))
numbers = [int(s.text) for s in soup.find_all("span")]
print(numbers, max(numbers))
# ✅ Expected output:
# ['12', '5'] 5
# [12, 5] 12And a price usually comes with its currency sign:
from bs4 import BeautifulSoup
soup = BeautifulSoup('<p class="gbp">£44.50</p>', "html.parser")
cost = soup.p.text
print(cost)
print(float(cost))
# ✅ Expected output:
# £44.50
# ✅ Expected error:
# ValueError: could not convert string to float: '£44.50'Take the sign off first: float(soup.p.text.replace("£", "")) gives 44.5.
Attributes, and why class is a list
from bs4 import BeautifulSoup
html = '<a href="/catalogue/dune_1/" class="title big">Dune</a> <a class="title">Coming soon</a>'
soup = BeautifulSoup(html, "html.parser")
first, second = soup.find_all("a")
print(first["href"])
print(first["class"])
print(first.attrs)
print(second.get("href"))
print(second.get("href", "no link yet"))
# ✅ Expected output:
# /catalogue/dune_1/
# ['title', 'big']
# {'href': '/catalogue/dune_1/', 'class': ['title', 'big']}
# None
# no link yetA tag's attributes work like a dictionary, as in Dictionaries: square brackets for one that must be there, get() for one that may not. class can hold several names, so it comes back as a list. Square brackets on an attribute the tag doesn't have stop the program:
from bs4 import BeautifulSoup
soup = BeautifulSoup('<a class="title">Coming soon</a>', "html.parser")
link = soup.find("a")
print(link.text)
print(link["href"])
# ✅ Expected output:
# Coming soon
# ✅ Expected error:
# KeyError: 'href'To find tags by class, find_all() takes class_, with an underscore, because class is a Python keyword: find("p", class="cost") is a SyntaxError.
from bs4 import BeautifulSoup
html = '<p class="cost sale">5.00</p><p class="sale cost">6.00</p><p class="cost">7.00</p>'
soup = BeautifulSoup(html, "html.parser")
print(len(soup.find_all("p", class_="cost")))
print(len(soup.find_all("p", class_="cost sale")))
print(len(soup.select("p.cost.sale")))
print([p["class"] for p in soup.find_all("p")])
# ✅ Expected output:
# 3
# 1
# 2
# [['cost', 'sale'], ['sale', 'cost'], ['cost']]class_="cost" matches any tag that has that class among its names, while class_="cost sale" matches only that exact text, in that order. The selector p.cost.sale needs both names, in either order. Attribute names with a dash can't be keyword arguments, so they go in attrs=, or in a selector:
from bs4 import BeautifulSoup
soup = BeautifulSoup('<div data-id="42">Dune</div><div data-id="7">Emma</div>', "html.parser")
print(soup.find("div", attrs={"data-id": "7"}).text)
print(soup.select_one('div[data-id="42"]').text)
# ✅ Expected output:
# Emma
# DuneThe line breaks between tags are children too
from bs4 import BeautifulSoup
html = """
<ul>
<li>Coffee</li>
<li>Tea</li>
</ul>
"""
soup = BeautifulSoup(html, "html.parser")
menu = soup.find("ul")
print(len(menu.contents))
print(repr(menu.contents[0]))
first = menu.find("li")
print(repr(first.next_sibling))
print(first.find_next_sibling("li").text)
print([li.text for li in menu.find_all("li")])
# ✅ Expected output:
# 5
# '\n'
# '\n'
# Tea
# ['Coffee', 'Tea']The list has five children: the two items, and the three line breaks around them. So the .next_sibling of the first item is a line break, not the next item. find_next_sibling("li") and find_all() look only at tags, and skip them.
A link is often only part of an address
An href is often relative to the page it's on. urljoin(), from urllib.parse, makes the full address the way a browser does:
from urllib.parse import urljoin
page = "https://books.toscrape.com/catalogue/page-2.html"
for href in ["page-3.html", "../index.html", "/static/style.css", "https://example.com/x", "?sort=price"]:
print(f"{href:22} {urljoin(page, href)}")
# ✅ Expected output:
# page-3.html https://books.toscrape.com/catalogue/page-3.html
# ../index.html https://books.toscrape.com/index.html
# /static/style.css https://books.toscrape.com/static/style.css
# https://example.com/x https://example.com/x
# ?sort=price https://books.toscrape.com/catalogue/page-2.html?sort=priceGluing the strings yourself gets these wrong:
page = "https://books.toscrape.com/catalogue/page-2.html"
print(page + "../index.html")
print("https://books.toscrape.com/" + "/static/style.css")
# ✅ Expected output:
# https://books.toscrape.com/catalogue/page-2.html../index.html
# https://books.toscrape.com//static/style.cssWith the soup, that gives full links to every book, and the address of the next page:
from urllib.parse import urljoin
from bs4 import BeautifulSoup
page_url = "https://books.toscrape.com/catalogue/page-2.html"
html = """
<h3><a href="dune_1/index.html">Dune</a></h3>
<h3><a href="emma_2/index.html">Emma</a></h3>
<a class="next" href="page-3.html">next</a>
"""
soup = BeautifulSoup(html, "html.parser")
for a in soup.select("h3 a"):
print(a.text, urljoin(page_url, a["href"]))
print("Next page:", urljoin(page_url, soup.select_one("a.next")["href"]))
# ✅ Expected output:
# Dune https://books.toscrape.com/catalogue/dune_1/index.html
# Emma https://books.toscrape.com/catalogue/emma_2/index.html
# Next page: https://books.toscrape.com/catalogue/page-3.htmlA slash at the end changes the result: urljoin("https://example.com/docs", "intro.html") gives https://example.com/intro.html, and with docs/ it gives https://example.com/docs/intro.html.
The page you inspect is not the page you parse
A browser's developer tools show the page after the browser has built it: it adds a <tbody> to every table that has none, and runs JavaScript that can add whole sections. Beautiful Soup sees the HTML the server sent, which View Source shows. A selector copied from the developer tools can find nothing:
from bs4 import BeautifulSoup
html = """
<table id="scores">
<tr><th>Name</th><th>Points</th></tr>
<tr><td>Maya</td><td>120</td></tr>
<tr><td>Leo</td><td>95</td></tr>
</table>
"""
soup = BeautifulSoup(html, "html.parser")
print(soup.select("table > tbody > tr"))
for row in soup.select("#scores tr")[1:]:
print([td.text for td in row.find_all("td")])
# ✅ Expected output:
# []
# ['Maya', '120']
# ['Leo', '95']Slicing off the first row skips the header, whose cells are th, not td.
Scraping politely: robots.txt
Many sites publish a robots.txt file saying which pages programs should leave alone, and sometimes how many seconds to wait between requests. Python's urllib.robotparser reads it. parse() takes the file's lines, so this works without a download:
from urllib.robotparser import RobotFileParser
rules = """
User-agent: *
Disallow: /checkout/
Disallow: /search
Crawl-delay: 2
"""
robots = RobotFileParser()
robots.parse(rules.splitlines())
site = "https://books.example"
for path in ["/catalogue/page-2.html", "/checkout/", "/search?q=dune", "/searching-tips.html"]:
print(path, robots.can_fetch("CourseBot", site + path))
print("Seconds between requests:", robots.crawl_delay("CourseBot"))
# ✅ Expected output:
# /catalogue/page-2.html True
# /checkout/ False
# /search?q=dune False
# /searching-tips.html False
# Seconds between requests: 2Each Disallow is the start of a path, so /search also blocks /searching-tips.html. Up to Python 3.14.4, robotparser follows the original robots.txt rules: the first rule that matches a path decides, and it doesn't understand the * and $ patterns some sites use, so it treats Disallow: /*.pdf$ as blocking nothing. From 3.14.5 it follows the newer standard, RFC 9309, as search engines do: the longest matching rule decides, and * and $ work. A polite scraper checks the rules, says who it is, and waits between pages:
import time
from urllib.parse import urljoin
from urllib.robotparser import RobotFileParser
import requests
from bs4 import BeautifulSoup
AGENT = "CourseBot/1.0 (+https://example.com/my-bot)"
robots = RobotFileParser("https://shop.example/robots.txt")
robots.read()
delay = robots.crawl_delay(AGENT) or 1
url = "https://shop.example/catalogue/page-1.html"
for page in range(3):
if not robots.can_fetch(AGENT, url):
break
resp = requests.get(url, headers={"User-Agent": AGENT}, timeout=10)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
for link in soup.select("h3 a"):
print(link.get_text(strip=True), urljoin(url, link["href"]))
next_link = soup.select_one("li.next a")
if next_link is None:
break
url = urljoin(url, next_link["href"])
time.sleep(delay)robots.read() downloads the rules. Each round checks them, sends a User-Agent that names the program and where to read about it, and stops at three pages or when there's no next link. time.sleep(delay) pauses for that many seconds before the next request. The site's terms of use are its other rules; read them before you scrape it.
Practice: links and prices from a book list
Fill the three blanks with the values in the hint lines, in order, so that the program prints each book's price and full link, and copes with the book that has no link yet.
from urllib.parse import urljoin
from bs4 import BeautifulSoup
page_url = "https://books.toscrape.com/catalogue/page-1.html"
html = """
<article class="book"><h3><a href="dune_1/index.html">Dune</a></h3><p class="gbp">£9.99</p></article>
<article class="book"><h3><a href="emma_2/index.html">Emma</a></h3><p class="gbp">£44.50</p></article>
<article class="book"><h3><a>Coming soon</a></h3><p class="gbp">£12.00</p></article>
"""
soup = BeautifulSoup(html, "html.parser")
for book in soup.___("article.book"):
link = book.find("a")
href = link.___("href")
cost = float(book.find("p", class_="gbp").text.replace("£", ""))
if href:
print(f"{link.text}: £{cost:.2f} {urljoin(page_url, ___)}")
else:
print(f"{link.text}: no page yet")
# 👉 replace ___ with select
# 👉 replace ___ with get
# 👉 replace ___ with href
# ✅ Expected output:
# Dune: £9.99 https://books.toscrape.com/catalogue/dune_1/index.html
# Emma: £44.50 https://books.toscrape.com/catalogue/emma_2/index.html
# Coming soon: no page yetOn your own: a price list from search results
This HTML is one page of a shop's search results. Print each book that has a price, cheapest first, with its price, title and full link, then how many books had no price.
from urllib.parse import urljoin
from bs4 import BeautifulSoup
page_url = "https://shop.example/search?q=python"
html = """
<div class="result"><a href="/p/fluent">Fluent Python</a><span class="cost">£44.50</span></div>
<div class="result"><a href="/p/automate">Automate the Boring Stuff</a><span class="cost">£22.99</span></div>
<div class="result"><a href="/p/crash">Python Crash Course</a></div>
<div class="result"><a href="/p/think">Think Python</a><span class="cost">£9.99</span></div>
"""
soup = BeautifulSoup(html, "html.parser")
# your code herefrom urllib.parse import urljoin
from bs4 import BeautifulSoup
page_url = "https://shop.example/search?q=python"
html = """
<div class="result"><a href="/p/fluent">Fluent Python</a><span class="cost">£44.50</span></div>
<div class="result"><a href="/p/automate">Automate the Boring Stuff</a><span class="cost">£22.99</span></div>
<div class="result"><a href="/p/crash">Python Crash Course</a></div>
<div class="result"><a href="/p/think">Think Python</a><span class="cost">£9.99</span></div>
"""
soup = BeautifulSoup(html, "html.parser")
rows = []
skipped = 0
for card in soup.select("div.result"):
link = card.find("a")
cost_tag = card.find("span", class_="cost")
if cost_tag is None:
skipped += 1
continue
cost = float(cost_tag.text.replace("£", ""))
rows.append((cost, link.text, urljoin(page_url, link["href"])))
for cost, title, url in sorted(rows):
print(f"£{cost:6.2f} {title:27} {url}")
print("No price:", skipped)
# ✅ Expected output:
# £ 9.99 Think Python https://shop.example/p/think
# £ 22.99 Automate the Boring Stuff https://shop.example/p/automate
# £ 44.50 Fluent Python https://shop.example/p/fluent
# No price: 1Each row is a tuple with the price first, so sorted() puts the cheapest first, and the price is a float, so £9.99 comes before £22.99; as text it would come last.
Practice quiz
soup = BeautifulSoup("<div>hello</div>", "html.parser"). What does soup.find("span") give?
- []
- None
- ''
- It stops with an error
Answer: None. find() gives None when nothing matches. find_all() would give an empty list.
What is soup.p.string for "<p>Hi <b>there</b></p>"?
- None
- 'Hi there'
- 'Hi '
- It stops with an error
Answer: None. .string is the text only when a tag holds exactly one thing. This one holds text and a b tag. .text gives 'Hi there'.
What does urljoin("https://example.com/docs", "intro.html") give?
- https://example.com/docs/intro.html
- https://example.com/docsintro.html
- https://example.com/intro.html
- It stops with an error
Answer: https://example.com/intro.html. Without a slash at the end, docs is treated as a page, and intro.html replaces it. https://example.com/docs/ would keep it.
link = BeautifulSoup("<a>Plain</a>", "html.parser").find("a"). What do link.get("href") and then link["href"] do?
- Both give ''
- Both give None
- The first stops with an error
- The first gives None, then the second stops with KeyError: 'href'
Answer: The first gives None, then the second stops with KeyError: 'href'. Attributes work like a dictionary: get() gives None for a missing one, square brackets raise a KeyError.
For "<i>12</i><i>5</i>", what does print(max(i.text for i in soup.find_all("i"))) show?
- 12
- 5
- 17
- It stops with an error
Answer: 5. The text from a page is a string, and "5" comes after "1". Convert with int() first: the largest is then 12.
What is soup.li["class"] for '<li class="dish special">Tea</li>'?
- 'dish special'
- 'dish'
- ['dish', 'special']
- It stops with an error
Answer: ['dish', 'special']. class can hold several names, so Beautiful Soup gives it as a list.
What does soup.get_text() give for "<p>Hello</p><p>World</p>"?
- HelloWorld
- Hello World
- Hello, then World on the next line
- ['Hello', 'World']
Answer: HelloWorld. get_text() joins the pieces of text with nothing between them. get_text(" ") puts a space between.
robots.txt holds Disallow: /search. What does can_fetch("CourseBot", "https://shop.example/searching-tips.html") give?
- True
- None
- It stops with an error
- False
Answer: False. A Disallow rule is a prefix, and /searching-tips.html starts with /search.
Continue this course
- Previous: Reading CSV & Excel Files
- Next: itertools — Iteration power tools — chain, groupby, combinations and more
- Quick reference: Python cheat sheet
Frequently asked questions
Why is data I can see in the browser missing from the soup?
The page builds it with JavaScript after it loads, and Beautiful Soup only sees the HTML the server sent, which View Source shows. That data usually arrives from a JSON address the page calls; find it in the developer tools' Network tab and request it directly, as HTTP Requests & APIs and Working with JSON show.
How do I save what I scraped?
Collect the rows in a list of lists or dictionaries, then write them with the csv module's writer or DictWriter, as Reading CSV & Excel Files shows. File Handling covers writing other kinds of files.
Is scraping a site allowed?
A site's robots.txt and its terms of use say what it allows. robotparser reads the first, as this lesson shows; read the terms yourself, and check the law where you live, which this lesson can't cover.