Where we left off

Every lesson so far rebuilt both indexes, dense and sparse, from scratch on every run, embedding every document over again even though nothing changed. naive_rag Lesson 13 fixed this for a single vector store; this lesson does the same thing for both halves of a hybrid index at once.

The code, piece by piece

def build_store() -> dict:
    return {
        "names": names,
        "token_lists": [tokenize(text) for text in texts],
        "doc_vectors": embed_texts(texts),
    }

One dict, both indexes. token_lists costs nothing to rebuild, it's just tokenizing text, no API call involved. doc_vectors is the expensive part, one embedding call per document, and the only reason this lesson exists: there's no benefit to persisting the sparse half on its own, but persisting both together means one save/load pair covers the whole hybrid index instead of two separate mechanisms.

def save_store(store: dict, path: Path) -> None:
    path.write_text(json.dumps(store))

Same trick as naive_rag Lesson 13: a dict of lists of floats and strings is exactly what JSON already represents, no custom serialization needed for either index.

Checkpoint

  • Persisting a hybrid index means persisting both halves together, the sparse index is cheap to rebuild on its own, but bundling it with the dense index means one file, one load, one hybrid-ready store.
  • The dense half is where persistence actually pays off, embedding API calls are the expensive, worth-skipping part.
  • Delete store.json to force a full rebuild, the same escape hatch naive_rag Lesson 13 used.

If anything here still feels unclear, ask before moving to Lesson 14.