Streaming Responses

The problem with waiting

Every call so far has been blocking: your program pauses on ollama.chat() until the model has finished writing its entire answer, then everything appears on screen at once. For a short answer that's fine; for a long one, a real user is left staring at nothing for however many seconds generation takes, even though the model already has the first sentence ready long before the last one.

Streaming fixes that by handing you each piece of the answer as soon as it exists, instead of making you wait for all of it.

The code, piece by piece

stream = ollama.chat(
    model="llama3.2",
    messages=[...],
    stream=True,
)

The single new argument, stream=True, changes what ollama.chat() returns: instead of one complete ChatResponse, you get an iterator you can loop over, one that yields a new small ChatResponse "chunk" every time the model has generated a bit more text.

for chunk in stream:
    print(chunk.message.content, end="", flush=True)

Each chunk has the same shape as the single response from Lesson 4, just with .message.content holding a fragment (often a word or two) instead of the whole answer. end="" stops print from adding its own newline after every fragment, and flush=True forces Python to show it on screen immediately rather than buffering it, both needed to get a genuinely live, word-by-word typing effect in the terminal.

if chunk.done:
    break

The stream's last chunk has done=True and empty content, it's a signal that generation has finished, not more text to print. Looping over the iterator until it naturally ends works too; checking .done explicitly here just makes that final signal visible in the code.

This is the same idea as .stream() on a LangChain model in the langchain course, applied to a local model instead of Gemini.