Generation with the Original Image

Where we left off

Lesson 7 retrieved an image's caption. This lesson asks: is the caption good enough to generate the final answer from, or should Gemini look at the original image again? The answer this course takes is deliberate: re-attach the original image at generation time, not just its caption, whenever an image is the best match. Retrieval runs on captions (Lesson 7, text-only search across everything); only generation, for the one or two records that actually got retrieved, pays the cost of a second look at the real pixels.

Why not just answer from the caption alone?

Because a caption is a lossy, one-time summary, written before anyone knew which exact question would be asked of it. CAPTION_PROMPT (Lesson 4) tries to be thorough, but it cannot anticipate every possible follow-up question a real image might get asked. A caption written for general retrieval says "8 Nm, printed in red"; if a question asks something the caption's author didn't think to describe, "is the torque spec printed on a sticker or engraved directly?", the caption alone cannot answer it, but the original image, looked at again with that specific question in mind, can. This is the same principle as naive_rag's citation-and-grounding lessons, applied to images: answer from the actual source, not from a summary of it, whenever the actual source is available.

The code, piece by piece

def generate_answer(query: str, retrieved: list[dict]) -> str:
    parts: list[types.Part | str] = []
    for record in retrieved:
        if record["image_path"] is not None:
            image_bytes = record["image_path"].read_bytes()
            parts.append(types.Part.from_bytes(data=image_bytes, mime_type="image/png"))
        else:
            parts.append(record["text"])

For each retrieved record, check image_path (Lesson 6's field, used for exactly this): if it's set, read the original file again and attach it as an image Part, the caption is discarded at this point, its job (being retrievable) is already done. If it's None, this record is a text chunk, and its text goes into the prompt exactly like every prior course in this series.

parts.append(f"Question: {query}
Answer using only the information above.")
response = client.models.generate_content(model=CHAT_MODEL, contents=parts)

One generate_content call, contents mixing however many images and text chunks got retrieved, plus the question, same multimodal contents shape Lesson 3 introduced, now built dynamically from retrieval's output instead of hardcoded to one file.