Concurrent Requests

One server, several requests

Every lesson so far has sent one request, waited for the reply, then moved on. A real application, especially one serving more than one user, often needs to handle several requests to the same local Ollama server at once. This lesson compares sending requests one at a time against sending them concurrently, from several threads.

Why threads, not `asyncio`, for this

ollama.chat() is a blocking call: your program pauses on the network round trip to localhost:11434 until a reply comes back. That's exactly the situation Python threads handle well, even with the Global Interpreter Lock, a thread that's blocked waiting on network I/O isn't holding up other threads from making progress. (The ollama package also ships an AsyncClient for asyncio-based code, if your program is already structured that way; threads are used here because they need no other changes to code you've already written in this course.)

The code, piece by piece

for i in range(n):
    ask(i)
sequential_total = time.time() - start

Baseline: each call fully completes before the next one starts. Total time is roughly the sum of every individual call's time.

with concurrent.futures.ThreadPoolExecutor(max_workers=n) as executor:
    results = list(executor.map(ask, range(n)))

ThreadPoolExecutor runs ask() for all n values at once, each in its own thread, all hitting the same local Ollama server simultaneously. executor.map() blocks until every call finishes and returns results in the original order, regardless of which thread happened to finish first.

What Ollama does with multiple requests at once

Ollama can serve multiple requests concurrently, up to a limit controlled by the OLLAMA_NUM_PARALLEL environment variable (server-side, not something this lesson's code sets), and bounded by how much VRAM/RAM is available to hold the model's active state for each simultaneous request. Requests beyond that limit queue and wait their turn rather than failing outright. This is why concurrent requests here don't take four times as long as one request. They genuinely overlap. But it's also not free: pushing far more concurrent load than your hardware and OLLAMA_NUM_PARALLEL can actually support will eventually just shift the wait from your code to Ollama's queue.