Skip to content
Xberg

Extraction Basics

Extract text, metadata, and structure from 100 file formats — PDFs, Office documents, images, email, HTML, archives, and more. Single files or batches, from local paths or in-memory bytes, with per-document configuration overrides and built-in content filtering.

See the Configuration Reference for all extraction settings and the Supported Formats reference for format-specific options.

Entry Points

Section titled “Entry Points”

Two extraction functions are the public entry points:

Function Input model Purpose
extract ExtractInput Extract one URI or in-memory byte payload
extract_batch ExtractInput[] Extract multiple URI and byte inputs

ExtractInput uses kind = "uri" for local paths, file:// URIs, and HTTP(S) URLs. Use kind = "bytes" for in-memory payloads. extract and extract_batch return an ExtractionResult envelope with results, errors, summary, and optional crawl metadata.

Beyond local paths and bytes, HTTP(S) URIs are fetched and can be crawled — see UrlExtractionConfig and CrawlConfig. Embedded images and their preprocessing are controlled by ImageExtractionConfig.

Extract One Input

Section titled “Extract One Input”
extract_one.py
from xberg import ExtractInput, extract


output = await extract(ExtractInput(kind="uri", uri="document.pdf"))
print(output.results[0].content)

Extract from Bytes

Section titled “Extract from Bytes”

When content is already loaded in memory, pass bytes through ExtractInput with an explicit MIME type.

extract_from_bytes.py
from xberg import ExtractInput, extract


with open("document.pdf", "rb") as file:
    data = file.read()


output = await extract(
    ExtractInput(
        kind="bytes",
        bytes=data,
        mime_type="application/pdf",
        filename="document.pdf",
    )
)

Batch Processing

Section titled “Batch Processing”

extract_batch accepts a list of ExtractInput values. Mix URI and byte inputs in one request when a pipeline receives documents from multiple sources.

extract_batch.py
from xberg import ExtractInput, extract_batch


inputs = [
    ExtractInput(kind="uri", uri="report.pdf"),
    ExtractInput(kind="uri", uri="scan.tiff", mime_type="image/tiff"),
]


output = await extract_batch(inputs)
for result in output.results:
    print(result.content[:200])

Per-Input Configuration

Section titled “Per-Input Configuration”

When a batch contains a mix of document types that need different settings, attach per-input overrides to ExtractInput while sharing a common batch config.

mixed_batch.py
from xberg import (
    ExtractionConfig,
    ExtractInput,
    FileExtractionConfig,
    extract_batch,
)


config = ExtractionConfig(output_format="markdown")


inputs = [
    ExtractInput(kind="uri", uri="report.pdf"),
    ExtractInput(
        kind="uri",
        uri="scan.tiff",
        config=FileExtractionConfig(force_ocr=True),
    ),
    ExtractInput(
        kind="uri",
        uri="notes.html",
        config=FileExtractionConfig(output_format="plain"),
    ),
]


output = await extract_batch(inputs, config)

Fields set to None in FileExtractionConfig inherit the batch default. Batch-level concerns like max_concurrent_extractions, use_cache, and security_limits cannot be overridden per input. See the Configuration Reference for the full list of overridable fields.

Archive and XML Bomb Protections

Section titled “Archive and XML Bomb Protections”

Every archive-bearing format (.zip, .docx, .pptx, .xlsx, .odt, .ods, .odp, .epub, .hwpx, and the iWork formats .pages/.key/.numbers) and every XML-bearing format is checked against SecurityLimits before or during parsing. These checks exist to stop a hostile input from exhausting memory or CPU rather than to validate document correctness — a legitimate large document should never come close to the defaults below.

Limit Defends against Default
max_archive_size ZIP bombs — declared uncompressed size accumulated across entries 500 MiB
max_compression_ratio ZIP bombs — a single entry or the archive total expanding beyond a sane ratio 100:1
max_files_in_archive Archives with an unreasonable number of member files 10,000
max_nesting_depth Deeply nested containers (nested archives, iWork protobuf messages) 1,024 levels
max_xml_depth Deeply nested XML elements 1,024 levels
max_entity_length Billion-laughs-class attacks — a single XML entity/attribute/token expanding to hundreds of MB 1 MiB
max_content_size Aggregate text growth across a whole document (catches long-tail expansion that max_entity_length alone would miss) 100 MB
max_iterations Infinite or near-infinite parser loops (XML token loop, HTML tokenizer, JSON parser) 10,000,000
max_table_cells Documents claiming an unreasonable number of table cells (CSV, XLSX, HTML tables) 100,000

When both max_nesting_depth and max_xml_depth apply to the same parse, the tighter of the two wins — lowering either one alone is enough to clamp nesting.

Archive-specific checks (max_archive_size, max_compression_ratio, max_files_in_archive) are enforced by ZipBombValidator against the ZIP central directory before any entry is decompressed, so an oversized or over-compressed archive is rejected without ever running the decompressor.

Configuring limits

Section titled “Configuring limits”

SecurityLimits is set on ExtractionConfig.security_limits and applies to the whole extraction (it cannot be overridden per file in a batch).

security_limits.py
from xberg import ExtractInput, ExtractionConfig, SecurityLimits, extract


config = ExtractionConfig(
    security_limits=SecurityLimits(
        max_archive_size=100 * 1024 * 1024,  # 100 MiB
        max_files_in_archive=1_000,
    ),
)


output = await extract(ExtractInput(kind="uri", uri="archive.zip"), config=config)

Any field left unset falls back to the SecurityLimits::default() value shown in the table above.

PHP note: SecurityLimits is constructible and readable in PHP, but ExtractionConfig does not accept a SecurityLimits argument in its constructor — the getSecurityLimits() getter always reflects the defaults. PHP callers cannot currently tighten or loosen these limits; if you need non-default limits from PHP, extract via another binding or the CLI.

What happens when a limit is hit

Section titled “What happens when a limit is hit”

All of these checks raise the same error family:

  • Rust — XbergError::Security { message, source }, where source is a boxed SecurityError (ZipBombDetected, ArchiveTooLarge, TooManyFiles, NestingTooDeep, ContentTooLarge, EntityTooLong, TooManyIterations, XmlDepthExceeded, TooManyCells, or UnreadableEntry for an archive entry whose header could not be read at all).
  • Python — a xberg.SecurityError exception (subclass of xberg.XbergError).
  • Other bindings — an error/result whose kind/name is "Security" (or the language’s equivalent typed error), carrying the same human-readable message.

The extraction is aborted for that input; it does not silently truncate or return partial content for a security violation the way a format-parsing warning would.

PDF Reading Order Repair

Section titled “PDF Reading Order Repair”

Native PDF text extraction reads glyphs in the order they were written into the content stream, not the order a human reads the page. For multi-column layouts (academic papers, magazines, dense reports) that order can jump between columns mid-sentence. Xberg can repair this by projecting text spans onto layout-detected regions, grouping them into columns, and re-emitting them top-to-bottom within each column, left-to-right across columns.

This repair is off by default. Enable it with ExtractionConfig.pdf_options.reading_order = true (PdfConfig::reading_order in Rust, default false).

pdf_reading_order.py
from xberg import ExtractInput, ExtractionConfig, PdfOptions, extract


config = ExtractionConfig(
    pdf_options=PdfOptions(reading_order=True),
)


output = await extract(
    ExtractInput(kind="uri", uri="two_column_paper.pdf"),
    config=config,
)

Requirements: this option only takes effect when the layout-detection feature is compiled in and layout hints were produced for the page (the page must have a layout model pass run over it). Without layout hints, the flag is a no-op and extraction falls back silently to native text order.

What it fixes: the flowing body text of multi-column pages — the case where native extraction interleaves column A and column B mid-paragraph.

What it does not fix: reading order only reorders text spans, not table cells. A page containing a rotated or scrambled table is unaffected by this option — that class of problem lives in the table extraction path, not the span-reordering pass, and setting reading_order will not repair it.

Cost: reordering only runs when layout hints are already available, so it adds span-projection and column-detection work on top of an existing layout-detection pass rather than triggering a new one on its own. There is no published benchmark number for the added latency; measure it against your own corpus before enabling it in a latency-sensitive pipeline.

Content Filtering

Section titled “Content Filtering”

Xberg strips running headers, footers, watermarks, and cross-page repeating text by default so downstream RAG and LLM pipelines see clean body content. ContentFilterConfig lets you opt back in when those regions carry useful text.

By default headers, footers, and watermarks are stripped and cross-page repeating text is deduplicated; see ContentFilterConfig for field-level defaults and per-format behavior.

keep_headers_footers.py
from xberg import (
    ContentFilterConfig,
    ExtractionConfig,
    ExtractInput,
    extract,
)


config = ExtractionConfig(
    content_filter=ContentFilterConfig(
        include_headers=True,
        include_footers=True,
    ),
)


output = await extract(
    ExtractInput(kind="uri", uri="contract.pdf"),
    config=config,
)

When a layout-detection model is active, it can independently classify regions as page headers or footers and strip them per page. Setting include_headers=True / include_footers=True also disables that per-page stripping. See the reference page for the full field semantics and per-format behavior.

Supported Formats

Section titled “Supported Formats”

Xberg supports 100 file formats across 120 file extensions in 8 categories:

Category Extensions Notes
PDF .pdf Native text + OCR for scanned pages
Images .png, .jpg, .jpeg, .tiff, .bmp, .webp, .heic, .heif, .avif OCR backend; HEIC/HEIF/AVIF need heic feature + libheif
Office .docx, .pptx, .xlsx, .odt, .ods, .odp Modern + OpenDocument via native parsers
Legacy Office .doc, .ppt, .wpd, .wp, .wp5, .wp6 Native OLE/CFB parsing; WordPerfect via libwpd
Email .eml, .msg Full support including attachments
Web .html, .htm Converted to Markdown with metadata
Text .md, .txt, .xml, .json, .yaml, .toml, .csv Direct extraction
Archives .zip, .tar, .tar.gz, .tar.bz2, .7z Recursive extraction

Image metadata and EXIF

Section titled “Image metadata and EXIF”

For every supported image format — JPEG, PNG, TIFF, WebP, BMP, GIF, JPEG 2000, HEIC, HEIF, AVIF — Xberg returns an ImageMetadata block on metadata.format containing:

  • width / height in pixels
  • format — uppercase format tag (e.g. JPEG, PNG, HEIF)
  • exif — a key/value map of EXIF tags

EXIF extraction is powered by the pure-Rust nom-exif integration and covers camera identity (Make, Model, LensModel, LensSpecification, Software), timestamps (DateTimeOriginal, CreateDate, OffsetTime, SubSecTime), full exposure parameters (ExposureTime, FNumber, ISO, ApertureValue, ShutterSpeedValue, ExposureProgram, ExposureMode, MeteringMode, Flash, SceneCaptureType), the complete GPS block (GPSLatitude, GPSLongitude, GPSAltitude, GPSTimeStamp, GPSDateStamp, GPSSpeed, GPSImgDirection, GPSMapDatum, GPSProcessingMethod), color space, thumbnail offsets, and provenance fields (Copyright, ImageDescription, ImageUniqueID).

EXIF works on every target, including wasm-target and android-target, because nom-exif is pure Rust. HEIC / HEIF / AVIF pixel decoding requires the heic Cargo feature and the system libheif library, and is therefore native-only — see the installation guide.

When the heic feature is enabled, HEIC / HEIF / AVIF inputs are decoded to RGBA via libheif, re-encoded as PNG, and then flow through the standard OCR / layout pipeline. EXIF is read from the original HEIC bytes before the PNG re-encode so no metadata is lost.

Page Tracking

Section titled “Page Tracking”

Xberg can track page boundaries and extract per-page content. Page tracking availability depends on the format:

  • PDF — Full byte-accurate page tracking with O(1) lookup
  • PPTX — Slide boundary tracking (each slide = one page)
  • DOCX — Best-effort detection using explicit <w:br type="page"/> tags
  • Other formats — No page tracking

Enable page extraction with PageConfig:

page_tracking.py
config = ExtractionConfig(
    pages=PageConfig(
        insert_page_markers=True,
        marker_format="\n\n<!-- PAGE {page_num} -->\n\n"
    )
)

Page markers like <!-- PAGE 1 --> are inserted at boundaries in the content field — useful for LLMs that need to understand document layout. When both page tracking and chunking are enabled, chunks automatically include first_page and last_page metadata.

See PageConfig Reference for all options and Chunking for chunk-to-page mapping examples.

Code File Extraction

Section titled “Code File Extraction”

Source code files (.py, .rs, .ts, .go, etc.) go through tree-sitter and produce a ProcessResult on ExtractedDocument.code_intelligence (structure, imports/exports, symbols, docstrings, diagnostics, semantic chunks). Code files bypass text chunking — TSLP’s function/class-aware CodeChunks map directly to Xberg Chunks with semantic chunk_type and heading context.

See Code Intelligence for usage and TreeSitterProcessConfig for fields.

PDF Page Rendering

Section titled “PDF Page Rendering”

Render individual PDF pages as PNG images. Unlike the extraction pipeline (which parses text, tables, metadata), this API produces raw pixel data for thumbnails, vision model input, or custom OCR pipelines. It is exposed as pure-Rust functions on the core crate.

Functions

Section titled “Functions”
Function Purpose
render_pdf_page_to_png Render one zero-based page index to PNG bytes at a given DPI
pdf_page_count Read the page count without rasterizing, to drive a render loop over pages

Render a single page, or count first and loop to process every page without holding all images in memory:

render_pdf_pages.rs
use xberg::{pdf_page_count, render_pdf_page_to_png};


let pdf_bytes = std::fs::read("document.pdf")?;


// Render one specific page (zero-based) at 300 DPI, no password.
let png = render_pdf_page_to_png(&pdf_bytes, 0, Some(300), None)?;
std::fs::write("page-0.png", &png)?;


// Or count pages and render each in turn.
let count = pdf_page_count(&pdf_bytes, None)?;
for page_index in 0..count {
    let png = render_pdf_page_to_png(&pdf_bytes, page_index, Some(150), None)?;
    std::fs::write(format!("page-{page_index}.png"), &png)?;
}

dpi defaults to 150 when passed None. password unlocks encrypted PDFs.

DPI Configuration

Section titled “DPI Configuration”
DPI Pixel size (US Letter) Use case
72 612 x 792 Thumbnails, quick previews
150 (default) 1275 x 1650 General-purpose, screen display
300 2550 x 3300 OCR input, print quality

Tip: Use 300 DPI when rendering pages for OCR or vision models. The default 150 DPI may reduce recognition accuracy on small text.

MIME Type Detection

Section titled “MIME Type Detection”

When extracting from bytes, ExtractInput requires an explicit MIME type since there’s no file extension to infer it from. For file paths, auto-detection from the extension is automatic.

Example: Override MIME Type

Section titled “Example: Override MIME Type”
Python
from xberg import ExtractInput, extract


# File without extension — provide MIME type explicitly
result = await extract(
    ExtractInput(
        kind="uri",
        uri="document_copy",
        mime_type="application/pdf",
    ),
    config=config,
)

Error Handling

Section titled “Error Handling”

All extraction functions raise typed exceptions on failure. Catch specific exceptions to handle different failure modes:

Python
import asyncio
from xberg import ExtractInput, extract, ExtractionConfig
from xberg import (
    XbergError,
    ParsingError,
    OcrError,
    ValidationError,
)


async def main() -> None:
    try:
        result = await extract(ExtractInput(uri="document.pdf"), ExtractionConfig())
        print(f"Extracted {len(result.results[0].content)} characters")
    except FileNotFoundError as e:
        print(f"File not found: {e}")
    except ParsingError as e:
        print(f"Failed to parse document: {e}")
    except OcrError as e:
        print(f"OCR processing failed: {e}")
    except XbergError as e:
        print(f"Extraction error: {e}")


    try:
        config: ExtractionConfig = ExtractionConfig()
        pdf_bytes: bytes = b"%PDF-1.4\n"
        result = await extract(
            ExtractInput(kind="bytes", bytes=pdf_bytes, mime_type="application/pdf", filename="document.pdf"),
            config,
        )
        print(f"Extracted: {result.results[0].content[:100]}")
    except ValidationError as e:
        print(f"Invalid configuration: {e}")
    except OcrError as e:
        print(f"OCR failed: {e}")
    except XbergError as e:
        print(f"Extraction failed: {e}")


asyncio.run(main())

Next Steps

Section titled “Next Steps”