VLM Image Captions

Caption every extracted image with a vision-language model to add alt-text, feed into retrieval pipelines, or describe diagrams and charts for downstream LLMs. See the CaptioningConfig reference for all options.

Feature gate

Requires the captioning Cargo feature. Included in full. Requires liter-llm and a vision-capable provider.

When to Use

• You need alt-text for accessibility-compliant exports
• You need searchable text descriptions per image to feed into a retrieval pipeline alongside the document body
• You need diagrams, charts, or photos described for LLM downstream consumption

When Not to Use

• You only need OCR’d text from images — use OCR for text extraction from images
• You’re processing high-volume batches where API spend is a concern — captioning calls an LLM per image
• Images are mostly decorative or structural elements

Configuration

Python

Python

import asyncio
from xberg import ExtractInput, extract, ExtractionConfig, CaptioningConfig, LlmConfig

async def main() -> None:
    config = ExtractionConfig(
        captioning=CaptioningConfig(
            llm=LlmConfig(model="openai/gpt-4o-mini"),
            min_image_area=0,
        ),
    )
    result = await extract(ExtractInput(uri="report.pdf"), config)
    for image in result.results[0].images or []:
        if image.caption:
            print(image.caption)

asyncio.run(main())

TypeScript

TypeScript

import { extract } from "@xberg-io/xberg";

const output = await extract({
    kind: "uri",
    uri: "report.pdf",
}, {
    captioning: {
        llm: { model: "openai/gpt-4o-mini" },
        minImageArea: 1000,
    },
});

for (const image of output.results[0].images ?? []) {
    if (image.caption) {
        console.log(image.caption);
    }
}

Rust

Rust

use xberg::{extract, ExtractionConfig, ExtractInput, CaptioningConfig, LlmConfig};

let config = ExtractionConfig {
    captioning: Some(CaptioningConfig {
        llm: LlmConfig {
            model: "openai/gpt-4o-mini".to_string(),
            ..Default::default()
        },
        prompt: None,
        min_image_area: 1000,
    }),
    ..Default::default()
};
let output = extract(ExtractInput::from_uri("report.pdf"), &config).await?;
for image in &output.results[0].images {
    if let Some(caption) = &image.caption {
        println!("{caption}");
    }
}

TOML

xberg.toml

[captioning]
min_image_area = 1000

[captioning.llm]
model = "openai/gpt-4o-mini"

Custom Prompt

Override the built-in caption prompt:

Python

Python

from xberg import ExtractionConfig, CaptioningConfig, LlmConfig

config = ExtractionConfig(
    captioning=CaptioningConfig(
        llm=LlmConfig(model="openai/gpt-4o-mini"),
        prompt="Describe this figure in one sentence suitable for alt-text.",
        min_image_area=4000,
    ),
)

The prompt is sent alongside each image as a single VLM request. The model sees the image plus the prompt; the response becomes the caption verbatim.

Filtering Small Images

min_image_area is in pixels (width × height). Icons, bullets, and decorative glyphs below the threshold are skipped — their caption field stays None. The default 1000 excludes 32×32 icons but admits typical inline figures. Raise the threshold to skip thumbnails; lower it to caption everything.

Output Shape

{
  "images": [
    {
      "image_kind": "diagram",
      "page_number": 3,
      "caption": "A flowchart showing the data ingestion pipeline: source → cleaner → indexer → retrieval API.",
      "bounding_box": { "x0": 72.0, "y0": 144.0, "x1": 540.0, "y1": 456.0 }
    },
    {
      "image_kind": "icon",
      "caption": null
    }
  ]
}

bounding_box uses PDF coordinates (x0=left, y0=bottom, x1=right, y1=top) and is only populated for PDF-extracted images when the extractor reports position data; it is omitted otherwise. page_number is a sibling field on the image, not part of the box.

Supported Providers

Any vision-capable liter-llm provider works (see the VLM OCR provider table). For batch captioning, gpt-4o-mini, claude-3-5-haiku, and google/gemini-2.0-flash are typically the cheapest options.

API-key precedence chain matches LLM Integration:

1. CaptioningConfig.llm.api_key
2. XBERG_LLM_API_KEY
3. Per-provider env var

Local engines (Ollama, LM Studio with a VLM, vLLM) need no key.

Related

• LLM Integration — provider matrix, local engines, VLM OCR
• OCR — text-from-image extraction
• Configuration Reference