Checkpoint: Scrape a Page Into a List

What this checkpoint is for

This lesson doesn't teach a new Playwright concept. It's a checkpoint: a small, real script that puts Lessons 3 through 7 to work together in one place. If you can read lesson.py top to bottom and predict what each line does before reading its comment, the beginner tier has landed. If any line surprises you, that's useful information, go back to the lesson it came from (noted below) before starting the intermediate tier.

Lesson 6 (clicking and typing) isn't exercised here, this particular scrape doesn't need to interact with anything, just read what's already on the page. It'll come back starting in Lesson 10.

The task

Scrape every book on the books.toscrape.com front page into a Python list of dictionaries, each with a title and a price. This is exactly the shape of a real, small scraping job: load a catalog page, pull structured data off it, and end up with something you could save to a CSV, insert into a database, or hand to an LLM.

The code, piece by piece

def scrape_books(url: str) -> list[dict]:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url)

The function is given a URL rather than hardcoding it inside, a small habit worth building early: it makes the function reusable and, later, easy to wrap as a tool (Lesson 15) without rewriting anything inside it.

        titles = page.locator(".product_pod h3 a")
        prices = page.locator(".product_pod p.price_color")

Two locators built from the same underlying structure, one for the title link inside each book, one for its price. Because both locators walk the page top to bottom, matching in the same order, index i in titles always corresponds to the same book as index i in prices.

        count = titles.count()
        books = []
        for i in range(count):
            title = titles.nth(i).get_attribute("title")
            price = prices.nth(i).text_content()
            books.append({"title": title, "price": price})

The Lesson 7 pattern: count the matches, loop by index, narrow each locator to one element with .nth(i), read what you need, and append a dictionary to a growing list. .get_attribute("title") is used instead of .text_content() for the title specifically because this site truncates long titles visually but keeps the full text in the title="" attribute, the same trick from Lesson 5.

        browser.close()
        return books

Close the browser before returning, the function is responsible for cleaning up everything it opened, exactly like every earlier lesson's with block did.