Every query so far has been a black box

query_engine.query(question) has been a one-line call for 21 lessons: a Response comes back, and that's all you see. How many LLM calls did that actually make? How many tokens did they use? What order did retrieval, embedding, and synthesis happen in? None of that has been visible, until now.

llama_index.core.callbacks answers those questions without changing a single line of query code. A callback handler observes every internal event (LLM calls, embedding calls, node parsing, retrieval, synthesis) as it happens, and you attach handlers globally via Settings.callback_manager, the same "set once on Settings, everything built afterward picks it up automatically" pattern Lesson 3 introduced for Settings.llm and Settings.embed_model.

Two handlers, two different views of the same events

This lesson attaches both at once:

HandlerWhat it captures
TokenCountingHandlerCumulative token counts: embedding tokens, LLM prompt tokens, LLM completion tokens
LlamaDebugHandlerAn ordered trace of every event type that fired, with timing

TokenCountingHandler answers "how much did this cost," LlamaDebugHandler answers "what actually happened, in what order." Both attach to the same CallbackManager, which fans every event out to every handler registered on it, they don't interfere with each other.

The code, piece by piece

token_counter = TokenCountingHandler()
llama_debug = LlamaDebugHandler(print_trace_on_end=False)
Settings.callback_manager = CallbackManager([token_counter, llama_debug])

print_trace_on_end=False disables LlamaDebugHandler's default behavior (auto-printing a trace tree every time a top-level operation finishes), since this lesson prints its own summary instead, using get_event_pairs() directly for a cleaner one-time view.

token_counter.total_embedding_token_count
token_counter.prompt_llm_token_count
token_counter.completion_llm_token_count
token_counter.total_llm_token_count

All four counters are cumulative for as long as the handler stays attached, they don't reset between calls. Note embedding tokens jump twice in this lesson's output: once when the index is built (embedding the 3 fixture documents), again during the query (embedding the question itself).

llama_debug.get_event_pairs()

Returns every recorded start/end event pair, in chronological order. Each pair's [0].event_type names what kind of operation it was (EMBEDDING, RETRIEVE, SYNTHESIZE, LLM, and more).

Checkpoint

  • Settings.callback_manager: a global hook point, set once, every Index and QueryEngine built afterward reports events to it automatically, no query code changes needed.
  • TokenCountingHandler: cumulative embedding and LLM token counts, the "how much did this cost" view.
  • LlamaDebugHandler: an ordered event trace with timing, the "what actually happened" view; get_event_pairs() reads it back programmatically.
  • Both handlers attach to the same CallbackManager and observe the same underlying events without interfering with each other.

If anything here still feels unclear, ask before moving to Lesson 23.