Where we left off

Lesson 3's flaw: raw term frequency treats every word equally, so a question built out of common words ("what," "do," "it," "about") barely discriminates between documents at all, whichever document is longest or uses those common words most often tends to win, regardless of relevance. TF-IDF fixes exactly this, by weighting each word's contribution by how rare it is across the whole corpus.

The code, piece by piece

def inverse_document_frequency(term, documents) -> float:
    doc_count = sum(1 for doc in documents if term in doc)
    return math.log(len(documents) / doc_count)

For each term, count how many of the documents contain it at least once. A term that shows up in every document tells you nothing about which one is relevant, log(6 / 6) = 0, it contributes nothing to any score. A term in just one document out of six is a real signal, log(6 / 1) ≈ 1.79, and gets weighted up accordingly. The log keeps this from scaling linearly, going from "rare" to "extremely rare" still matters, just with diminishing returns.

score += term_frequency * idf

Lesson 3's raw count, now multiplied by that rarity weight before being added up. A word that appears often in this document and rarely across the corpus drives the score the most, exactly the combination that flags "this document is specifically about that."

Checkpoint

  • TF-IDF: term frequency, multiplied by inverse document frequency, downweighting words that appear everywhere, upweighting words that appear almost nowhere.
  • IDF: log(total documents / documents containing this term) - a word in every document contributes zero; a word in one document out of many contributes a lot.
  • This is still sparse retrieval, still zero embedding calls, just a smarter way to count.

If anything here still feels unclear, ask before moving to Lesson 5.