Chunk Size and Overlap

Where we left off

Lesson 4 split notes.txt on blank lines, a boundary that happened to exist in that specific file. Most real documents don't hand you such a convenient boundary, so this lesson introduces the more general technique: fixed-size chunking, splitting text every N characters regardless of where sentences or paragraphs fall, and the problem that creates.

The problem: a chunk boundary doesn't know what a sentence is

Cutting text every fixed number of characters will, sooner or later, cut directly through the middle of a word or a sentence. When that happens, the fact or detail that spanned the cut is now split across two chunks, and neither chunk alone contains the complete idea. If retrieval only returns one of those two chunks (which is likely, since half a fact is also half as similar to a relevant question), part of the answer is simply missing from what generation ever sees.

Overlap is the fix: instead of starting each new chunk exactly where the last one ended, start it a little earlier, re-including the tail end of the previous chunk. Whatever got cut off at the end of one chunk now has a decent chance of appearing whole, at least once, somewhere.

The code, piece by piece

def fixed_size_chunks(text: str, size: int, overlap: int) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start += size - overlap
    return chunks

start += size - overlap is the whole idea in one line. With overlap=0, each chunk starts exactly size characters after the last one, no repetition, this is what Lesson 4's paragraph split effectively did too, just at paragraph boundaries instead of a fixed character count. With overlap=30, each new chunk starts 30 characters earlier than that, so the last 30 characters of one chunk reappear as the first 30 characters of the next.