Prompt Engineering Lifecycle

What we're building

The full loop, in one script: run an experiment against a deliberately naive first-draft prompt, spot its weakness, push a fixed prompt to the hub, pull it back, and re-run the identical experiment. Two experiments, rag-lifecycle-v1 and rag-lifecycle-v2, same dataset and evaluator, different prompt.

What this reveals

Lessons 9-12 each ran one experiment in isolation. Lesson 13 introduced the Prompt Hub, but only round-tripped one version. Real prompt work is a loop: notice a weakness in the current prompt's scores, change the wording, re-run the same evaluation, see if it actually helped, repeat. This lesson makes that loop concrete with a mistake that's genuinely common when building a first RAG prompt: naive_prompt never references {context} at all, so the model answers each question from whatever it already knows, completely ignoring what retrieve() found.

Because keyword_overlap (Lesson 10) and the dataset (Lesson 8) stay fixed across both runs, the prompt is the only variable, so the score difference between v1 and v2 is attributable to the wording change, not to a different test.

make_target() exists so that swapping which prompt is used doesn't require duplicating the evaluation code, one function builds a target closure around whatever prompt it's given, called twice with two different prompts.

The code, piece by piece

naive_prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer in one short sentence."),
    ("human", "Question: {question}"),
])
baseline = evaluate(make_target(naive_prompt), data=DATASET_NAME, evaluators=[keyword_overlap], experiment_prefix="rag-lifecycle-v1")

The bug: no {context} placeholder anywhere in this prompt, so retrieval happens (inside make_target) but its result is thrown away. Same evaluate() call as Lesson 10 otherwise.

grounded_prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer using only the given context, in one short sentence."),
    ("human", "Context:\n{context}\n\nQuestion: {question}"),
])
client.push_prompt(PROMPT_NAME, object=grounded_prompt)

The fix: a {context} placeholder, and an instruction to actually use it. Pushing under the same PROMPT_NAME creates a new version rather than a new prompt.

v2_prompt = client.pull_prompt(PROMPT_NAME)
v2 = evaluate(make_target(v2_prompt), ..., experiment_prefix="rag-lifecycle-v2")

Pulling again (rather than reusing the local grounded_prompt variable) proves the loop works the way it would in a real team: someone else could have made this edit directly in the hub's UI, and pulling is how your evaluation code picks it up. In one real run of this lesson, v1 averaged 0.38 and v2 averaged 0.64 keyword_overlap, your numbers won't match exactly, Gemini's wording varies run to run, but the gap should be consistently large.