Evaluating Retrieval

Trust, but verify

Every RAG lesson up to this point has trusted the QueryEngine: retrieve some Nodes, ask the LLM to synthesize an answer, print it, done. Nothing checked whether that answer was actually any good. llama_index.core.evaluation is a small toolkit of LLM-as-judge evaluators built for exactly that gap, automated checks you can run against a RAG pipeline the same way you'd run unit tests against ordinary code.

This lesson uses two of them, asking two genuinely different questions about the same query/response/retrieved-context triple:

A RAG pipeline can fail either check independently: retrieval can pull the right Nodes but the LLM still invents an unsupported detail (faithfulness fails, relevancy might still pass), or retrieval can miss entirely and the LLM answers from its own training data instead (faithfulness might still pass on that fabricated answer, relevancy fails because the context doesn't address the query).

Both evaluators are themselves LLM calls

FaithfulnessEvaluator and RelevancyEvaluator default to Settings.llm as the judge if no llm= is passed in, same one-LLM, multiple-jobs pattern used everywhere else in this course. This means every evaluation costs at least one extra LLM call on top of the query itself, worth knowing before running this against a large batch of test questions in a real project.

The code, piece by piece

faithfulness_evaluator = FaithfulnessEvaluator()
relevancy_evaluator = RelevancyEvaluator()

Both default to Settings.llm as judge. You can pass a different, often cheaper or more capable, llm= explicitly if you want the judge to be a different model than the one answering questions.

response = query_engine.query(question)
faithfulness_result = faithfulness_evaluator.evaluate_response(query=question, response=response)

evaluate_response() takes the query string and the QueryEngine's own Response object directly, it pulls the retrieved context straight off response.source_nodes internally, no need to pass contexts by hand.

faithfulness_result.passing  # bool
faithfulness_result.score    # float, 1.0 or 0.0 for these two evaluators

EvaluationResult carries .passing (a bool) and .score (a float). For FaithfulnessEvaluator and RelevancyEvaluator specifically, the underlying judge is a binary YES/NO prompt, so score is always 1.0 or 0.0, other evaluators in this module (e.g. CorrectnessEvaluator, not used here) return a continuous score instead.