← Learning

Multimodal RAG

A linear, one-concept-per-lesson path through Multimodal RAG, retrieval-augmented generation extended to cover images as well as text, built from scratch with no framework, from your first multimodal generate_content call to a complete FastAPI service that returns cited answers grounded in both text and images. 26 lessons · 3 tiers.

Prerequisites: Completion of Naive RAG, or already comfortable with embeddings, cosine similarity, and top-k retrieval - this course reuses that pipeline unchanged for text and adds one new idea, captioning-then-embed, for images.

Lessons use Google's Gemini free tier (gemini-embedding-001 for embeddings, gemini-3.5-flash-lite for chat and captioning, the latter via native multimodal image input), the same models as this site's other RAG courses. No Docker and no database through most of the course; the Advanced tier graduates the mixed store to a local ChromaDB instance, still no server or account required.

Built directly on google-genai, Gemini's native multimodal SDK - the captioning-then-embed step this course adds runs entirely through its image-input API.

Course source

Every lesson's README and lesson.py for this course live in the ai-agent-engineering repo.

View on GitHub →
Multimodal RAG

Beginner

Captioning-then-embed end to end: describe an image with Gemini, embed the caption, retrieve it, and re-attach the real image at generation time.

  1. 01What Is Multimodal RAGGitHub
  2. 02The Blind Spot of Text-Only RetrievalGitHub
  3. 03Sending an Image to GeminiGitHub
  4. 04Captioning an ImageGitHub
  5. 05Embedding Image CaptionsGitHub
  6. 06A Mixed Vector StoreGitHub
  7. 07Retrieval Across ModalitiesGitHub
  8. 08Generation with the Original ImageGitHub
  9. 09Checkpoint: CLI Q&A Over Notes and ImagesGitHub

Intermediate

Multiple images per document, explicit modality metadata, persistence, balanced retrieval across modalities, and a minimal text-vs-image evaluation.

  1. 10Captioning Multiple Images per DocumentGitHub
  2. 11Modality MetadataGitHub
  3. 12Persisting Captions and EmbeddingsGitHub
  4. 13Balancing k Across ModalitiesGitHub
  5. 14Prompting for Modality-Cited AnswersGitHub
  6. 15Failure Modes of Captioning and MatchingGitHub
  7. 16Minimal Evaluation: Text vs. Image QuestionsGitHub
  8. 17Extracting Images from a PDFGitHub
  9. 18Checkpoint: Notes-and-Diagrams Search AssistantGitHub

Advanced

Graduating the mixed store to ChromaDB, wrapping it as a FastAPI service that serves the original image back, and a capstone.

  1. 19Where Per-Query Captioning Gets ExpensiveGitHub
  2. 20Introducing ChromaDB for the Mixed StoreGitHub
  3. 21Metadata Filtering by ModalityGitHub
  4. 22Refactoring into ingest() and ask()GitHub
  5. 23Wrapping It as a ServiceGitHub
  6. 24Serving the Original Image BackGitHub
  7. 25Capstone: A Complete Multimodal RAG ServiceGitHub
  8. 26Where Multimodal RAG Hits a WallGitHub
Start at Lesson 1

Contact

Book a Call