Development Workflow

Everything you need to build, test, and debug Xberg locally. This guide assumes you’ve already followed the Contributing Guide to fork and clone the repository.

The Task Runner

Xberg uses mise for its toolchain and for all build and test workflows (mise run <task>; mise tasks lists them). One command to bootstrap everything:

mise run setup

That installs all toolchains and dependencies. Safe to re-run anytime — it’s idempotent.

Core Tasks

Top-level tasks operate on the Rust core:

mise run build                # Build the Rust core (debug)
mise run test                 # Run Rust core tests
mise run check                # Lint + format check across the whole repo

The Rust namespace exposes the finer-grained variants:

mise run rust:build:dev       # Debug build (faster compile, no optimizations)
mise run rust:build release   # Release build (slow compile, fast binary)
mise run rust:test            # Run all Rust tests
mise run rust:test:ci         # Same tests, with CI diagnostics (tessdata setup)

Language bindings are generated and compiled by Alef, so they do not each get a <lang>:build/<lang>:test pair. Build and test them through the aggregate tasks below, or via the per-language e2e and native-test tasks (see End-to-end Test Suites).

Bulk Operations

mise run alef:build           # Build every binding (via Alef)
mise run test:bindings        # Run binding test suites only
mise run check                # Lint + format check across the whole repo

Testing Locally

Rust

The core lives in crates/xberg/. Most changes start here.

mise run rust:test

cargo test -p xberg test_pdf_extraction -- --nocapture

RUST_LOG=debug cargo test -p xberg test_name -- --nocapture

Bindings

Bindings are generated and compiled by Alef. Build them all, then run their tests:

mise run alef:build           # Compile every binding
mise run test:bindings        # Run every binding test suite

Each mise run <lang>:e2e task (alef test --e2e --lang <lang>) runs that binding’s own package tests ([crates.test.<lang>] in alef.toml) before its e2e suite — mise run python:e2e, mise run swift:e2e, and so on (see End-to-end Test Suites).

The RUST_LOG env var propagates into every binding — the Rust core logs through the host process’s stderr:

RUST_LOG=debug mise run python:e2e

Working with Alef Bindings

Every language binding — the crates under crates/xberg-*, the packages under packages/, the READMEs, and the e2e suites — is generated by Alef from the Rust source and alef.toml. Do not hand-edit generated output; it is overwritten on the next regeneration.

To change a binding, edit the Rust source, README templates, fixtures, or alef.toml, then regenerate:

alef all --clean --skip-compile --skip-snippet-validation  # Regenerate all Alef-managed output, no build
mise run alef:build           # Compile the bindings
mise run version:sync         # Sync the version from Cargo.toml to every manifest
mise run alef:verify          # Check that generated output is up to date

alef all regenerates and formats (via poly); --skip-compile keeps it from compiling. Run mise run dart:format:context first when the Dart output changes, so dart format reads the pinned language version. Commit the generator inputs and regenerated output together in one change.

End-to-end Test Suites

End-to-end tests guarantee that every language binding produces identical results for the same document. They live in e2e/ as shared fixtures — test inputs paired with expected outputs.

Run end-to-end Tests

Each language runs its suite with mise run <lang>:e2e:

Language	Directory	Run with
Python	`e2e/python/`	`mise run python:e2e`
TypeScript / Node.js	`e2e/node/`	`mise run node:e2e`
Rust	`e2e/rust/`	`mise run rust:e2e`
Go	`e2e/go/`	`mise run go:e2e`
Java	`e2e/java/`	`mise run java:e2e`
.NET	`e2e/csharp/`	`mise run csharp:e2e`
Ruby	`e2e/ruby/`	`mise run ruby:e2e`
PHP	`e2e/php/`	`mise run php:e2e`
Elixir	`e2e/elixir/`	`mise run elixir:e2e`
Swift	`e2e/swift/`	`mise run swift:e2e`
Zig	`e2e/zig/`	`mise run zig:e2e`
Dart	`e2e/dart/`	`mise run dart:e2e`
Kotlin / Android	`e2e/kotlin_android/`	`mise run kotlin-android:e2e`
WebAssembly	`e2e/wasm/`	`mise run wasm:e2e`

Regenerate end-to-end Tests

E2E suites are generated from shared fixtures by Alef. Generation is repo-wide, not per-language:

mise run dart:format:context && alef e2e generate --strict  # Regenerate every suite from fixtures
mise run e2e:verify           # Regenerate and fail on uncommitted drift (the CI check)

Benchmarking

Measure extraction performance with the benchmark harness in tools/benchmark-harness/. Use it to track regressions, compare against alternatives, and identify bottlenecks with flamegraphs.

Quick Start

FRAMEWORK=xberg MODE=single-file mise run benchmark:run
FRAMEWORK=xberg MODE=batch mise run benchmark:run

Common Modes

Mode	What it measures
`single-file`	Latency — one file at a time
`batch`	Throughput — multiple files in parallel

With Profiling

The profiling.yaml workflow (manual dispatch) generates flamegraphs of the pipeline benchmark. mise run rust:build:profiling builds the same profiling profile (release plus debug symbols) locally.

View live benchmark results at https://xberg.io/benchmarks.

Linting and Pre-commit

mise run lint               # Lint all code via poly
mise run format             # Format all code via poly
mise run check              # Format + lint check, no modifications (CI-equivalent)

poly handles Python (ruff), JS/TS (oxc), TOML, and Markdown directly, so those languages have no dedicated lint task.

The repository uses pre-commit hooks that enforce conventional commit messages, code formatting, and linter rules. If a commit is rejected, the hook output tells you exactly what to fix.

Working with Documentation

Docs are an Astro Starlight site under docs-site/. Building the documentation site requires Node.js 22.23.4 or newer; CI uses Node.js 24.

Building Locally

mise run docs:build         # pnpm install + astro build
mise run docs:serve         # pnpm install + astro dev (live reload)

How Snippets Work

Documentation examples have two source paths:

Prefer fixture-backed examples for public APIs shared across languages. Add or update the JSON fixture under fixtures/, run mise run dart:format:context && alef e2e generate --strict, and render its generated snippets with ApiSnippetGroup. The generated files under docs-site/src/snippets-generated/ are Alef output; never edit them directly.
Use maintained snippets under docs-site/src/snippets/ only for examples that do not map to an E2E fixture, such as deployment, server, or language-specific workflows. Import these Markdown files into the relevant .mdx page as Astro components.

A fixture-backed example renders every available language without one import per binding:

import ApiSnippetGroup from "../../../components/ApiSnippetGroup.astro";

<ApiSnippetGroup topic="batch/extract_batch_uri_basic" />

The topic is the generated path after the language directory, without the .md extension. For example, docs-site/src/snippets-generated/rust/batch/extract_batch_uri_basic.md maps to batch/extract_batch_uri_basic.

Maintained snippets remain organized by language and capability:

docs-site/src/snippets/
├── python/           # Python examples
│   ├── api/          #   extract, extract_batch, etc.
│   ├── config/       #   ExtractionConfig, OcrConfig, etc.
│   ├── ocr/          #   OCR backends
│   ├── plugins/      #   Plugin implementations
│   ├── mcp/          #   MCP server and client
│   └── utils/        #   Embeddings, chunking, errors
├── rust/             # Rust examples (same layout)
├── typescript/       # TypeScript examples
├── go/, java/, csharp/, ruby/
├── docker/           # Docker commands
├── api_server/       # Server startup examples
└── cli/              # CLI usage

When you change a user-facing API, update its fixture and regenerate. When no fixture-backed example is appropriate, update the maintained snippet instead. Use mise run docs:snippets:gaps to find unreferenced snippets or missing language variants, and mise run docs:snippets:all to audit them and syntax-check both generated and maintained snippets.

Debugging

Rust Panics

RUST_BACKTRACE=1 cargo test -p xberg test_name
RUST_BACKTRACE=full cargo test -p xberg test_name

Python FFI Problems

When something goes wrong in the Rust core during a Python call, the failure surfaces as a typed exception from xberg — the base XbergError plus specific subclasses (OcrError, ParseError, ConfigError, and so on). Catch it and inspect the message, and set RUST_LOG to trace the core:

import asyncio
import os

os.environ["RUST_LOG"] = "debug"

from xberg import ExtractionConfig, XbergError, extract


async def main() -> None:
    try:
        await extract("broken.pdf", ExtractionConfig())
    except XbergError as error:
        print(f"Extraction failed: {error}")


asyncio.run(main())

Verbose Logging

Crank up the log level to see what the Rust core is doing:

RUST_LOG=debug mise run python:e2e
RUST_LOG=trace mise run rust:test

CI/CD

CI is not a single pipeline — it is split into per-domain workflows under .github/workflows/, each gated by its own path filters so a change only triggers the workflows it touches. Editing Rust core fires ci-rust; touching docs fires ci-docs; changing a binding fires ci-e2e and, for mobile, ci-mobile.

Workflow	Scope
`ci-lint.yaml`	Format + lint validation via poly, plus commit/PR checks
`ci-rust.yaml`	Rust core build and tests (Linux x86_64/arm64 and macOS)
`ci-e2e.yaml`	Cross-language binding build and e2e suites
`ci-mobile.yaml`	Android + iOS binding checks
`ci-docker.yaml`	Docker build and smoke tests
`ci-docs.yaml`	Documentation build and validation
`ci-gpu.yaml`	GPU-backed jobs (manual dispatch)

Running CI Checks Locally

Before pushing, run the same checks CI runs:

mise run check              # Format + lint (matches ci-lint)
mise run rust:test:ci       # Rust tests with CI diagnostics
mise run e2e:verify         # Generated e2e suites are current
mise run <lang>:e2e         # One e2e suite, as each ci-e2e leg runs it

Other Workflows

Workflow	When it runs	What it does
`ci-docs.yaml`	Changes to `docs-site/**` or `alef.toml`	Builds, validates, and deploys documentation
`validate-pr.yml`	Every PR	Conventional-commit and PR checks
`benchmarks.yaml`	Manual trigger	Runs the full benchmark suite
`profiling.yaml`	Manual trigger	Generates flamegraphs
`publish.yaml`	Release events	Publishes packages to registries
`publish-docker.yaml`	Tags and releases	Builds and pushes Docker images

Performance

Xberg’s core is written in Rust, which enables zero-copy memory handling, SIMD acceleration, and true multi-core parallelism — all at compile time with no garbage collection.

Why Rust Matters

Native compilation: LLVM optimizes code ahead of time (inlining, vectorization, dead code elimination)
Zero-copy strings: Slicing uses borrowed references, not heap allocations
SIMD acceleration: Whitespace detection and character classification run 15-37x faster than scalar operations
No GIL: True multi-core parallelism across all CPU cores
Deterministic memory: Drop semantics free memory instantly, no GC pauses

Key Optimizations

Batch processing: 6-10x faster than sequential extraction through work-stealing scheduler
Caching: 85%+ hit rates for repeated files (SQLite-backed, automatic invalidation)
Streaming: Large files processed in 4KB chunks, constant memory regardless of file size
Lazy initialization: Expensive subsystems (Tokio, plugins) initialized on first use only

Benchmarking Your Workload

Measure with your actual files using the benchmark harness (see Benchmarking section for full instructions). For detailed analysis and live benchmark results, visit https://xberg.io/benchmarks.