Web scraping

Definition

In simple words

It's a robot that reads a web page and copies the bits you want into a table, in place of you doing it by hand for hours.

Why it matters

It's a common way to gather data for ML projects when no API exists. It's also full of legal and ethical limits, which anyone building datasets must know.

How it works

1. Look for an official API first.
2. Read the site's terms and robots.txt.
3. Fetch slowly, and identify yourself.
4. Don't collect personal data without a lawful basis.

import requests
from bs4 import BeautifulSoup

html = requests.get(url, timeout=10).text
soup = BeautifulSoup(html, "html.parser")
for link in soup.select("a.title"):
    print(link.text, link["href"])

In code

The standard library can parse HTML too, shown here on a saved page.

from html.parser import HTMLParser

page = """<html><body>
<h2 class="title">Learn AI</h2><p>Price: 0</p>
<h2 class="title">Learn Python</h2><p>Price: 0</p>
</body></html>"""

class Titles(HTMLParser):
    def __init__(self):
        super().__init__()
        self.inside = False
        self.found = []
    def handle_starttag(self, tag, attrs):
        self.inside = tag == "h2" and ("class", "title") in attrs
    def handle_data(self, data):
        if self.inside:
            self.found.append(data)
            self.inside = False

parser = Titles()
parser.feed(page)
print(parser.found)

['Learn AI', 'Learn Python']

Common mistakes

• Hammering a site with fast requests. It can break the site and get you blocked.
• Assuming public means free to use. Copyright and privacy laws still apply.

Check yourself

Summary

• Fetch, parse, and extract.
• Prefer APIs, and go slowly.
• Respect terms, copyright and privacy.