Chunking with HybridChunker

Why not just split the Markdown string every N characters

A naive chunker that slices export_to_markdown()'s output every N characters doesn't know where a table row ends or a section begins, it will happily cut a table in half or split a sentence across two chunks. HybridChunker, from docling_core.transforms.chunker, starts from the DoclingDocument's own structure instead of the rendered string: it splits on real section and table boundaries first, then respects a tokenizer's max token count as a hard limit, merging small adjacent pieces under the same heading where they still fit.

Chunking a document

from docling_core.transforms.chunker.hybrid_chunker import HybridChunker

chunker = HybridChunker()
chunks = list(chunker.chunk(dl_doc=doc))

Each chunk carries .text (the chunk's own content) and .meta, which includes .headings, the section heading(s) that chunk falls under. That's the same outline Lesson 5 pulled out by hand with a list comprehension, HybridChunker tracks it automatically, per chunk, so you always know which part of the document a retrieved chunk came from.

contextualize()

contextualized = chunker.contextualize(chunk=chunk)

A chunk's raw .text for a table row might read like "North, Product Line = Hardware. North, Q2 Revenue = $412,000...", correct, but thin on its own once it's embedded and sitting in a vector store next to thousands of other chunks. contextualize() prepends the chunk's heading(s), so the string that actually gets embedded reads "Revenue by Region\nNorth, Product Line = Hardware...", carrying enough context to be found and understood on its own. This is the string you should hand to an embedding model, not the bare .text.

Notice the table becomes its own chunk, and its rows are serialized as key = value pairs rather than a pipe table, that serialization is exactly what makes a table row searchable by an embedding model, "North, Growth = +20.9%" is meaningful text on its own in a way a raw pipe-table row fragment would not be.