Reading Content Off a Page

From one thing to many things

Lessons 4 and 5 read one value at a time: a title, a price. Real scraping usually means the opposite: loop over every matching element on a page and pull structured data out of each one. This lesson does exactly that against quotes.toscrape.com, ten quotes on the page, each with a text, an author, and a list of tags, collected into a plain Python list of dictionaries.

That list-of-dictionaries shape is deliberate. It's the same shape you'd get back from a database query or a JSON API, which makes it easy to save, filter, or hand off to something else later, exactly the shape Lesson 13, extracting structured data into typed Pydantic models, builds on.

.text_content() versus .inner_html()

Two different ways to read what's inside an element, and they answer different questions:

• .text_content() returns the visible text, with any HTML tags stripped out. If an element contains <span>Hello <b>world</b></span>, this returns "Hello world". This is almost always what you want when scraping data meant to be read, prices, names, quotes.
• .inner_html() returns the raw markup inside the element, "Hello <b>world</b>" in the example above. Less commonly what you want for data extraction, but genuinely useful for two things: debugging (seeing exactly what structure you're matching against, when a selector isn't behaving the way you expected) and cases where you deliberately need to preserve formatting rather than strip it.

Looping over multiple matches

A locator that matches several elements doesn't behave like a Python list you can directly iterate with for x in locator. Instead, the standard pattern is: get the count, then index into the locator with .nth(i) for each position.

for i in range(quote_elements.count()):
    quote = quote_elements.nth(i)

range(quote_elements.count()) produces 0, 1, 2, ... up to (but not including) the count, one index for each match, which .nth(i) then uses to grab that specific element.

The plural read: .all_text_contents()

Where .text_content() reads one element's text, .all_text_contents() reads every matching element's text at once and returns them as a list of strings. Here, each quote has multiple <a class="tag"> elements underneath it, and the goal is all of them, not just the first, so .all_text_contents() is the natural fit, no manual loop needed.

The code, piece by piece

quote_elements = page.locator(".quote")

One locator, matching all ten <div class="quote"> blocks on the page.

for i in range(quote_elements.count()):
    quote = quote_elements.nth(i)
    text = quote.locator(".text").text_content()
    author = quote.locator(".author").text_content()
    tags = quote.locator(".tag").all_text_contents()
    quotes.append({"text": text, "author": author, "tags": tags})

For each quote, narrow to that one element, then read three things out of it: the quote text, the author's name, and every tag, into one dictionary appended to a growing list.

first_quote_html = quote_elements.first.locator(".text").inner_html()

The raw markup version of the first quote's text, shown purely so you can compare it against the stripped .text_content() version above.