Development Workflow

Everything you need to build, test, and debug Xberg locally. This guide assumes you’ve already followed the Contributing Guide to fork and clone the repository.

The Task Runner

Xberg uses Task for all build and test workflows. One command to bootstrap everything:

task setup

That installs all toolchains and dependencies. Safe to re-run anytime — it’s idempotent.

Core Tasks

Top-level tasks operate on the Rust core:

task build                # Build the Rust core (debug)
task build:release        # Build the Rust core (release)
task test                 # Run Rust core tests
task check                # Lint + format check across the whole repo

The Rust namespace exposes the finer-grained variants:

task rust:build:dev       # Debug build (faster compile, no optimizations)
task rust:build:release   # Release build (slow compile, fast binary)
task rust:test            # Run all Rust tests
task rust:test:ci         # Same tests, with CI diagnostics (tessdata setup)
task rust:test:quick      # Fast unit tests only

Language bindings are generated and compiled by Alef, so they do not each get a <lang>:build/<lang>:test pair. Build and test them through the aggregate tasks below, or via the per-language e2e and native-test tasks (see End-to-end Test Suites).

Bulk Operations

task build:all            # Build core + every binding
task build:bindings       # Build every binding (via Alef)
task test:all             # Run all tests (core + bindings)
task test:bindings        # Run binding test suites only
task test:cov             # All tests with coverage
task check                # Lint + format check across the whole repo

Testing Locally

Rust

The core lives in crates/xberg/. Most changes start here.

task rust:test

cargo test -p xberg test_pdf_extraction -- --nocapture

RUST_LOG=debug cargo test -p xberg test_name -- --nocapture

Bindings

Bindings are generated and compiled by Alef. Build them all, then run their tests:

task build:bindings       # Compile every binding
task test:bindings        # Run every binding test suite

A few native bindings expose their own unit-test task:

task swift:test
task zig:test
task dart:test
task kotlin-android:test

For all other languages the cross-language check is the e2e suite — task python:e2e, task node:e2e, task go:e2e, and so on (see End-to-end Test Suites).

The RUST_LOG env var propagates into every binding — the Rust core logs through the host process’s stderr:

RUST_LOG=debug task python:e2e

Testing the live browser demo

The demo at docs-site/public/demo.html loads @xberg-io/xberg-wasm from a CDN. To test local changes against it, use:

task demo:dev

This builds the Wasm binary and TypeScript dist, patches the demo with local URLs, and starts two servers:

Server	URL	Role
Docs	`http://localhost:8001`	Serves the patched `demo-dev.html`
Assets	`http://localhost:9000`	Serves the local Wasm package

Open http://localhost:8001/demo-dev.html — no manual edits needed. The patched file (docs-site/public/demo-dev.html) is gitignored and regenerated on every run. The two different ports reproduce the cross-origin setup the CDN creates in production.

To skip the slow Rust build when you’ve only changed TypeScript:

SKIP_WASM_BUILD=1 task demo:dev

Working with Alef Bindings

Every language binding — the crates under crates/xberg-*, the packages under packages/, the READMEs, and the e2e suites — is generated by Alef from the Rust source and alef.toml. Do not hand-edit generated output; it is overwritten on the next regeneration.

To change a binding, edit the Rust source, README templates, fixtures, or alef.toml, then regenerate:

task alef:generate        # Regenerate all Alef-managed output (alef all --clean, formatting via poly, no build)
task alef:build           # Compile the bindings (same as build:bindings)
task alef:sync            # Sync the version from Cargo.toml to every manifest
task alef:verify          # Check that generated output is up to date

task alef:generate regenerates and formats (via poly) without compiling. Commit the generator inputs and regenerated output together in one change.

End-to-end Test Suites

End-to-end tests guarantee that every language binding produces identical results for the same document. They live in e2e/ as shared fixtures — test inputs paired with expected outputs.

Run end-to-end Tests

Each language runs its suite with task <lang>:e2e:

Language	Directory	Run with
Python	`e2e/python/`	`task python:e2e`
TypeScript / Node.js	`e2e/node/`	`task node:e2e`
Rust	`e2e/rust/`	`task rust:e2e`
Go	`e2e/go/`	`task go:e2e`
Java	`e2e/java/`	`task java:e2e`
.NET	`e2e/csharp/`	`task csharp:e2e`
Ruby	`e2e/ruby/`	`task ruby:e2e`
PHP	`e2e/php/`	`task php:e2e`
Elixir	`e2e/elixir/`	`task elixir:e2e`
Swift	`e2e/swift/`	`task swift:e2e`
Zig	`e2e/zig/`	`task zig:e2e`
Dart	`e2e/dart/`	`task dart:e2e`
Kotlin / Android	`e2e/kotlin_android/`	`task kotlin-android:e2e`
WebAssembly	`e2e/wasm/`	`task wasm:e2e`

Or run one suite by name with task e2e:lang E2E_LANG=python.

Regenerate end-to-end Tests

E2E suites are generated from shared fixtures by Alef. Generation is repo-wide, not per-language. The canonical tasks are:

task e2e:generate         # Regenerate all suites from fixtures
task e2e:build            # Build the bindings the suites link against
task e2e:test             # Run every suite
task e2e:all              # Generate, build, and run in one pass

Verify the checked-in suites are current with task e2e:verify.

Benchmarking

Measure extraction performance with the benchmark harness in tools/benchmark-harness/. Use it to track regressions, compare against alternatives, and identify bottlenecks with flamegraphs.

Quick Start

task benchmark:run FRAMEWORK=xberg MODE=single-file
task benchmark:run FRAMEWORK=xberg MODE=batch

Common Modes

Mode	What it measures
`single-file`	Latency — one file at a time
`batch`	Throughput — multiple files in parallel

With Profiling

Generate flamegraphs to see where time is spent:

task benchmark:profile

This builds with the profiling profile (release plus debug symbols) and runs the pipeline benchmark. Results appear under flamegraphs/<sha>/ as interactive SVGs.

View live benchmark results at https://xberg.io/benchmarks.

Linting and Pre-commit

task lint               # Lint all code via poly
task format             # Format all code via poly
task check              # Format + lint check, no modifications (CI-equivalent)

poly handles Python (ruff), JS/TS (oxc), TOML, and Markdown directly, so those languages have no dedicated lint task. Compiled bindings expose a native linter:

task rust:lint          # cargo fmt + clippy (auto-fix); rust:lint:check for check-only
task go:lint            # golangci-lint
task ruby:lint          # rubocop + steep
task php:lint           # mago
task csharp:lint        # dotnet format
task swift:lint         # swift format lint
task zig:lint           # zig fmt --check

The repository uses pre-commit hooks that enforce conventional commit messages, code formatting, and linter rules. If a commit is rejected, the hook output tells you exactly what to fix.

Working with Documentation

Docs are an Astro Starlight site under docs-site/.

Building Locally

task docs:build         # pnpm install + astro build
task docs:serve         # pnpm install + astro dev (live reload)

How Snippets Work

Code examples in the docs aren’t inline — they’re pulled from docs-site/src/snippets/ and imported into each .mdx page as Astro components (import { Content as Snip_... } from "../../../snippets/.../file.md", rendered inside <TabItem>). This keeps examples testable and reusable across pages.

docs-site/src/snippets/
├── python/           # Python examples
│   ├── api/          #   extract, extract_batch, etc.
│   ├── config/       #   ExtractionConfig, OcrConfig, etc.
│   ├── ocr/          #   OCR backends
│   ├── plugins/      #   Plugin implementations
│   ├── mcp/          #   MCP server and client
│   └── utils/        #   Embeddings, chunking, errors
├── rust/             # Rust examples (same layout)
├── typescript/       # TypeScript examples
├── go/, java/, csharp/, ruby/
├── docker/           # Docker commands
├── api_server/       # Server startup examples
└── cli/              # CLI usage

When you change a user-facing API, update the matching snippet. When you add a new feature, create a snippet and import it from the relevant doc page. Use task docs:snippets:gaps to find unreferenced snippets or missing language variants, and task docs:snippets:validate to syntax-check them.

Debugging

Rust Panics

RUST_BACKTRACE=1 cargo test -p xberg test_name
RUST_BACKTRACE=full cargo test -p xberg test_name

Python FFI Problems

When something goes wrong in the Rust core during a Python call, the failure surfaces as a typed exception from xberg — the base XbergError plus specific subclasses (OcrError, ParseError, ConfigError, and so on). Catch it and inspect the message, and set RUST_LOG to trace the core:

import asyncio
import os

os.environ["RUST_LOG"] = "debug"

from xberg import ExtractionConfig, XbergError, extract


async def main() -> None:
    try:
        await extract("broken.pdf", ExtractionConfig())
    except XbergError as error:
        print(f"Extraction failed: {error}")


asyncio.run(main())

Verbose Logging

Crank up the log level to see what the Rust core is doing:

RUST_LOG=debug task python:e2e
RUST_LOG=trace task rust:test

CI/CD

CI is not a single pipeline — it is split into per-domain workflows under .github/workflows/, each gated by its own path filters so a change only triggers the workflows it touches. Editing Rust core fires ci-rust; touching docs fires ci-docs; changing a binding fires ci-e2e and, for mobile, ci-mobile.

Workflow	Scope
`ci-lint.yaml`	Format + lint validation via poly, plus commit/PR checks
`ci-rust.yaml`	Rust core build and tests (Linux, macOS, Windows)
`ci-e2e.yaml`	Cross-language binding build and e2e suites
`ci-mobile.yaml`	Android + iOS binding checks
`ci-docker.yaml`	Docker build and smoke tests
`ci-docs.yaml`	Documentation build and validation
`ci-gpu.yaml`	GPU-backed jobs (manual dispatch)

Running CI Checks Locally

Before pushing, run the same checks CI runs:

task check              # Format + lint (matches ci-lint)
task rust:test:ci       # Rust tests with CI diagnostics
task e2e:all            # Generate, build, and run every e2e suite
task test:cov           # All tests with coverage

Other Workflows

Workflow	When it runs	What it does
`ci-docs.yaml`	Changes to `docs/` or `zensical.toml`	Builds and validates documentation
`validate-pr.yml`	Every PR	Conventional-commit and PR checks
`benchmarks.yaml`	Manual trigger	Runs the full benchmark suite
`profiling.yaml`	Manual trigger	Generates flamegraphs
`publish.yaml`	Release events	Publishes packages to registries
`publish-docker.yaml`	Tags and releases	Builds and pushes Docker images

Performance

Xberg’s core is written in Rust, which enables zero-copy memory handling, SIMD acceleration, and true multi-core parallelism — all at compile time with no garbage collection.

Why Rust Matters

Native compilation: LLVM optimizes code ahead of time (inlining, vectorization, dead code elimination)
Zero-copy strings: Slicing uses borrowed references, not heap allocations
SIMD acceleration: Whitespace detection and character classification run 15-37x faster than scalar operations
No GIL: True multi-core parallelism across all CPU cores
Deterministic memory: Drop semantics free memory instantly, no GC pauses

Key Optimizations

Batch processing: 6-10x faster than sequential extraction through work-stealing scheduler
Caching: 85%+ hit rates for repeated files (SQLite-backed, automatic invalidation)
Streaming: Large files processed in 4KB chunks, constant memory regardless of file size
Lazy initialization: Expensive subsystems (Tokio, plugins) initialized on first use only

Benchmarking Your Workload

Measure with your actual files using the benchmark harness (see Benchmarking section for full instructions). For detailed analysis and live benchmark results, visit https://xberg.io/benchmarks.