Images and LLM Captioning

Why a plain image converts to almost nothing

MarkItDown converts formats it can extract structured content from. An image file is just pixels, there's no text layer, no document structure, nothing to parse the way a docx or xlsx has. Without help, the image converter can only report what's available without actually understanding the picture: EXIF metadata, if the file has any. Most screenshots and simple photos, like this course's fixture, don't, so the result is close to an empty string.

llm_client and llm_model: describing the picture

MarkItDown(...) accepts two extra keyword arguments, llm_client and llm_model. When set, the image converter sends the image to that model and asks it to describe what's in it, then uses the model's description as the Markdown output. This turns "a file with no text layer" into "a file with a text description an LLM downstream can actually search and reason over."

md = MarkItDown(llm_client=client, llm_model="gemini-3.5-flash-lite")

Using Gemini through an OpenAI-shaped client

MarkItDown's llm_client argument expects an object shaped like OpenAI's client (specifically, something with .chat.completions.create(...)). This project uses Gemini, not OpenAI, everywhere else, but Google exposes an OpenAI-compatibility endpoint that speaks the exact same request/response shape. So instead of authenticating against api.openai.com, this lesson builds an OpenAI client pointed at Google's endpoint:

client = OpenAI(
    api_key=os.environ["GOOGLE_API_KEY"],
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
)

No OpenAI account, no second API key, openai here is just the shape of the client, not the provider actually answering the request.

The code, piece by piece

result_plain = MarkItDown().convert(IMAGE_PATH)

The baseline: no LLM client, nothing to work with beyond metadata.

md_llm = MarkItDown(llm_client=client, llm_model="gemini-3.5-flash-lite")
result_llm = md_llm.convert(IMAGE_PATH)

Same .convert() call, same file, the only difference is the MarkItDown instance now has an LLM client attached. This makes one real network call to Gemini, keep that in mind before re-running this lesson repeatedly against a constrained quota.