Batching and Async

The problem with a loop of .invoke() calls

Every call to .invoke() since Lesson 1 has been one question, one answer, one round trip over the network. If you need answers to three different questions, the obvious approach is a loop:

for thing in things:
    chain.invoke({"thing": thing})

This works, but it's slow in a specific way: each .invoke() waits for its network round trip to fully finish before the next one even starts. Three questions means three round trips, back to back, with no overlap.

.batch(): many inputs, run concurrently

chain.batch([{"thing": thing} for thing in things])

.batch() takes a list of inputs and lets LangChain start all of their network requests at (roughly) the same time, instead of one after another. The requests overlap while they're waiting on the network, rather than queuing up. Run this lesson and compare the two printed times, chain.batch() finishing in roughly half the time (or less) of the sequential loop for the same three questions is the concurrency actually paying off.

.ainvoke() and asyncio: the same idea, written explicitly

await asyncio.gather(*(chain.ainvoke({"thing": thing}) for thing in things))

.ainvoke() is the async version of .invoke(). If you haven't used Python's async/await before: it's a way of saying "start this, but don't block everything else while waiting for it to finish." asyncio.gather(...) starts several async calls at once and waits for all of them together, similar in spirit to what .batch() does, but written with Python's own async tools instead of a LangChain-provided shortcut.

In practice, .batch() is simpler to reach for; .ainvoke() matters more once your own program is already async (for example, a web server handling many users at once).

Why three DIFFERENT sets of questions?

Look closely at the code: sequential, batched, and async each use a different set of three things (sky/grass/banana vs. fire truck/snow/ eggplant vs. school bus/coal/flamingo), not the same three questions three times. This is deliberate.

Google's Gemini API automatically caches recently-seen identical prompts, if all three phases asked about "the sky," the second and third phases might come back suspiciously fast for a reason that has nothing to do with batching or async, just because the answer was already cached from the first phase. Using distinct questions for each phase keeps the comparison honest: any speed difference you see is really about concurrency, not caching.