Node Parsers and Chunking Strategies

Splitting is a strategy, not a single fixed step

Lesson 2 used SentenceSplitter as if it were the only way to turn a Document into Nodes, and by default it is what Settings uses under the hood whenever an Index builds itself. But LlamaIndex ships several node parsers under llama_index.core.node_parser, each trading off differently between simplicity, speed, and how much they respect the source text's structure. This lesson contrasts the two simplest: the SentenceSplitter you already know, and TokenTextSplitter, a plainer alternative with no sentence awareness at all.

Both are pure local computation, same as Lesson 2, no LLM or embedding calls involved, chunking is just string manipulation.

Other parsers that exist but aren't covered here

llama_index.core.node_parser also ships SentenceWindowNodeParser (splits into single sentences but attaches a window of surrounding sentences as metadata, for retrieving a small chunk while synthesizing with more context) and SemanticSplitterNodeParser (uses embedding similarity between sentences to decide where topic boundaries fall, rather than a fixed token count). Both are worth knowing exist; this lesson sticks to the two parsers that need no LLM or embedding calls, to stay cheap and to keep the comparison to boundary strategy alone.

The code, piece by piece

sentence_splitter = SentenceSplitter(chunk_size=60, chunk_overlap=10)
sentence_nodes = sentence_splitter.get_nodes_from_documents([remote_work_doc])

token_splitter = TokenTextSplitter(chunk_size=60, chunk_overlap=10)
token_nodes = token_splitter.get_nodes_from_documents([remote_work_doc])

Same chunk_size/chunk_overlap budget for both, on purpose, so any difference in node count or boundaries comes from the splitting *strategy*, not from a different size setting. chunk_size=60 (tokens) is deliberately small here, small enough that a ~1000-character document actually splits into more than a dozen Nodes, making the difference in boundaries visible.

sentence_chunk_8 = sentence_nodes[8].text.strip()
token_chunk_11 = token_nodes[11].text.strip()

Two chunks that happen to cover the same region of the source text (the "for tax and legal reasons." sentence), picked by inspecting both node lists. SentenceSplitter stops its chunk exactly at that sentence's period; TokenTextSplitter's corresponding chunk runs straight through the period into the start of the next sentence, because it has no notion of "sentence" at all, only a token count and a separator (a space, by default) to split on.