Skip to content
Xberg

Redaction & Anonymisation

Strip PII from extracted documents before they leave your system—emails, phone numbers, credit cards, names, organizations, and custom patterns you define. Redacted results stay locally processed with no network calls; an audit trail preserves what was redacted and where.

When to Use

Section titled “When to Use”
  • You ship extracted content to a service that should never see PII.
  • You need a deterministic, local pattern engine (no network calls) for regex-detectable PII.
  • You need tenant-specific tokens (employee IDs, project codenames, internal product names) removed alongside built-in categories.

When Not to Use

Section titled “When Not to Use”
  • You need to keep PII in the result for downstream NER or analytics. Run NER first and store the entities; redact in a second pass.
  • You need to redact free-form names and your build doesn’t include redaction-ml. The pattern engine cannot find arbitrary names — it covers only regex-detectable categories.

Configuration

Section titled “Configuration”
Python
import asyncio
from xberg import ExtractInput, extract, ExtractionConfig, RedactionConfig


async def main() -> None:
    config = ExtractionConfig(
        redaction=RedactionConfig(
            categories=["email", "phone", "ssn", "credit_card", "iban"],
            strategy="mask",
        ),
    )
    result = await extract(ExtractInput(uri="contract.pdf"), config)
    print(result.results[0].content)
    print(f"Redacted {result.results[0].redaction_report.total_redacted} spans")


asyncio.run(main())

PII Categories

Section titled “PII Categories”
PiiCategory Detection Notes
Email Pattern RFC-5322-ish.
Phone Pattern E.164 + national formats.
Ssn Pattern US SSN with 000/666/9xx exclusions.
CreditCard Pattern 13–19 digits + Luhn check.
PostalCode Pattern Multi-locale.
IpAddress Pattern IPv4 + IPv6.
Iban Pattern ISO country code + length + checksum.
SwiftBic Pattern See “Known limitations” — current regex over-matches plain English words.
DateOfBirth — Not yet implemented. Accepted in config but never fires — see “Known limitations”.
Person NER Requires RedactionConfig.ner = Some(NerConfig).
Organization NER Same.
Location NER Same.
Custom(label) User-supplied custom_terms or custom_patterns.

Strategies

Section titled “Strategies”
RedactionStrategy Output Use when
Mask (default) [REDACTED] You only need PII gone.
Hash [HASH:<16hex>] — SHA-256 truncated to 16 hex chars, wrapped You need equality joins downstream without recovering the source.
TokenReplace [PERSON_1], [PERSON_2], … per category You need to preserve co-reference inside the document.
Drop empty string You need the span gone with no marker.

User-Supplied Terms and Patterns

Section titled “User-Supplied Terms and Patterns”

The most-used surface in production. Pass literal strings or regex patterns the caller already knows are sensitive.

Python
from xberg import (
    ExtractionConfig, RedactionConfig, RedactionTerm, RedactionPattern,
)


config = ExtractionConfig(
    redaction=RedactionConfig(
        strategy="token_replace",
        custom_terms=[
            RedactionTerm(label="Project", value="Project Polaris", case_sensitive=False),
            RedactionTerm(label="Employee", value="EMP-7421", case_sensitive=True),
        ],
        custom_patterns=[
            RedactionPattern(label="InternalId", pattern=r"INT-\d{6}", case_sensitive=False),
        ],
    ),
)

RedactionTerm.value is regex-escaped before matching — pass literal text without escaping. RedactionPattern.pattern uses the Rust regex crate dialect (no look-around). Case-insensitive by default; set case_sensitive = true for exact-byte match. Patterns are validated at config-construction time via RedactionConfig::validate().

User hits always surface as PiiCategory::Custom(label) and are retained even when RedactionConfig.categories filters out the built-in detectors.

Pairing with NER

Section titled “Pairing with NER”

To redact names, organisations, and locations, attach a NerConfig:

Python
from xberg import (
    ExtractionConfig, RedactionConfig, NerConfig, LlmConfig,
)


config = ExtractionConfig(
    redaction=RedactionConfig(
        categories=["person", "organization", "location", "email"],
        strategy="token_replace",
        ner=NerConfig(
            backend="llm",
            llm=LlmConfig(model="openai/gpt-4o-mini"),
        ),
    ),
)

Choose the NER backend per the NER guide. The LLM backend can call provider-hosted models; the xberg-gliner ONNX backend runs locally with exported artifacts from xberg-io/gliner-models. Private or non-public artifacts require Hugging Face credentials supported by hf-hub.

Output Shape

Section titled “Output Shape”
{
  "content": "Contact [REDACTED] at [REDACTED]. Reference [PROJECT_1].",
  "redaction_report": {
    "total_redacted": 3,
    "findings": [
      { "start": 8, "end": 24, "category": "person", "strategy": "mask", "replacement_token": "[REDACTED]" },
      { "start": 28, "end": 50, "category": "email", "strategy": "mask", "replacement_token": "[REDACTED]" },
      { "start": 61, "end": 75, "category": { "custom": "Project" }, "strategy": "token_replace", "replacement_token": "[PROJECT_1]" }
    ]
  }
}

Offsets refer to the ORIGINAL pre-redaction content. Use them only for audit-trail reconstruction — the original bytes are gone by the time the result reaches the caller.

Data Handling

Section titled “Data Handling”

The redaction post-processor:

  • Runs locally. The pattern engine makes no network calls.
  • Drops the original text. Only redaction_report carries spans back to the original — and only as numeric offsets, never as the original characters.
  • Adjusts chunk byte ranges in place when preserve_offsets = true (default). Set false to keep chunk offsets pointing at the original document.

The NER backend, when enabled, follows whichever backend you configure — see NER for the network-call surface of ner-llm.

Redaction blast radius

Section titled “Redaction blast radius”

Redaction does not stop at content. It rewrites every text-bearing field on the result in place, so a caller cannot recover PII by reading a secondary surface the primary content no longer exposes. Beyond content, the following are masked with the same strategy and category set:

  • formatted_content, and per-chunk content
  • summary text, and translation content plus its formatted markup
  • NER entities text and page-classification labels
  • tables — cell values and markdown — including per-page tables
  • Per-page content, elements, and ocr_elements text
  • djot_content plain text
  • images — captions, descriptions, and nested OCR sub-documents (recursively)
  • uris — URL and label (so an always-on Email match cannot leak via a mailto: link)
  • annotations (PDF comment text)
  • form_fields — name, value, default value, and tooltip
  • extracted_keywords (with keywords-yake / keywords-rake)
  • metadata — title, subject, authors, keywords
  • structured_output, and code_intelligence (with tree-sitter) — string values in the JSON tree; object keys are left alone

The secondary surfaces are re-scanned by the pattern engine (and custom terms/patterns), not the NER backend — NER runs once, over content. redaction_report.total_redacted and findings count only the content pass; secondary surfaces are still rewritten but do not add to the report totals.

Rehydration (Reversible Redaction)

Section titled “Rehydration (Reversible Redaction)”

TokenReplace is the only strategy that is reversible. Mask, Hash, and Drop destroy the original value permanently — there is nothing to rehydrate. When you need to redact for a downstream consumer but retain the ability for an authorized party to recover the original text, run redaction with redact_capturing_rehydration_map instead of redact:

use xberg::text::redaction::{redact_capturing_rehydration_map, encrypt_map, decrypt_map};


// `config` must use RedactionStrategy::TokenReplace for the categories you want to recover later.
let rehydration_map = redact_capturing_rehydration_map(&mut result, &config).await?;


// The map is plaintext token -> original text in memory. Encrypt before it touches disk or a queue.
let passphrase = std::env::var("XBERG_REHYDRATION_PASSPHRASE")?;
let encrypted_blob: Vec<u8> = encrypt_map(&rehydration_map, &passphrase)?;
// Persist `encrypted_blob` yourself — xberg does not write it anywhere.

redact_capturing_rehydration_map (crates/xberg/src/text/redaction/engine.rs) runs the same pass as redact, then returns a RehydrationMap (type RehydrationMap = HashMap<String, String>, token to original text) built from every TokenReplace allocation. It contains only TokenReplace hits — a config that uses Mask or Hash for a category produces no recoverable entry for that category, by design.

What the encryption actually does

Section titled “What the encryption actually does”

encrypt_map and decrypt_map (crates/xberg/src/text/redaction/rehydration.rs) implement a specific, auditable wire format, not a black box:

  • The map is JSON-serialized, then encrypted with AES-256-GCM.
  • The encryption key is derived from your passphrase with scrypt (N = 2^14, r = 8, p = 1) and a random 16-byte salt generated per call.
  • The output is XPII\x01 (5-byte magic) + 16-byte salt + 12-byte nonce + 16-byte GCM tag + ciphertext. Every call to encrypt_map produces different bytes for the same map, because the salt and nonce are freshly generated each time.
  • decrypt_map fails closed: a wrong passphrase or corrupted blob returns an error, never partial or garbage plaintext.

This is reversible encryption, not anonymisation. Anyone who holds the passphrase can recover every original value in the map. That is the intended behavior — rehydration exists precisely so an authorized key holder can reverse a redaction — but it means:

  • The redacted data is not erased. It is encrypted and stored under a key. If your obligation is to make the data unrecoverable, TokenReplace + encrypt_map does not satisfy that; use Mask, Hash, or Drop instead, which have no rehydration path at all.
  • Whoever controls the passphrase controls the data. Treat the passphrase (and the encrypted blob together) as being in the same compliance scope as the original PII — key custody is data custody. Store the passphrase in a secrets manager or environment variable, never in the same location as the encrypted map, and never in source, logs, or tickets.
  • Losing the passphrase makes the redaction permanent and irreversible. There is no recovery path — derive_key only produces the correct key from the exact original passphrase and salt. Losing the passphrase is equivalent to having used Mask.
  • xberg persists nothing on your behalf. redact_capturing_rehydration_map, encrypt_map, and decrypt_map all operate purely in memory and return values to the caller. Choosing where the encrypted blob lives, how the passphrase is distributed, and how both are rotated is entirely your responsibility.

Finding and forgetting a subject

Section titled “Finding and forgetting a subject”

Once you’ve decrypted a map, find_subject and forget_subject (crates/xberg/src/text/redaction/rehydration.rs) let you locate or remove a specific person’s entries by token or by a case-insensitive substring match on the original value:

use xberg::text::redaction::{decrypt_map, encrypt_map, find_subject, forget_subject};


let passphrase = std::env::var("XBERG_REHYDRATION_PASSPHRASE")?;
let mut map = decrypt_map(&encrypted_blob, &passphrase)?;


// Look up every entry for a subject, by name or exact token (e.g. "[EMAIL_3]").
let hits = find_subject(&map, "Jane Doe");


// Remove every matching entry and get back what was removed, for your own audit log.
let removed = forget_subject(&mut map, "Jane Doe");


// forget_subject only mutates the in-memory map — you must re-encrypt and overwrite
// your stored blob, or the erasure has no effect.
let updated_blob = encrypt_map(&map, &passphrase)?;

forget_subject is idempotent: calling it again with the same query after the matching entries are already gone returns an empty result rather than erroring or matching everything. An empty query matches nothing (not “everything”) to avoid a blank erase request wiping the whole map.

This mechanically deletes matching key/value pairs from the decrypted map before you re-encrypt it. It does not, by itself, touch the original document, any copies of result.content you may have kept, backups of the previously-encrypted blob, or log lines that captured the plaintext before redaction ran. Whether deleting these entries satisfies an erasure request in your jurisdiction is your determination to make — forget_subject gives you the mechanism, not the compliance conclusion.

Known Limitations

Section titled “Known Limitations”
  • SWIFT/BIC over-matches plain English words. The current regex ([A-Z]{4}[A-Z]{2}[A-Z0-9]{2}(?:[A-Z0-9]{3})?) accepts arbitrary 8/11-letter all-caps tokens after the engine uppercases the input. Until a country-allowlist lands, scope RedactionConfig.categories to the subset you actually need rather than redacting everything.
  • PERSON / ORGANIZATION / LOCATION require NER. Without RedactionConfig.ner, those categories are silently skipped.
  • DateOfBirth is not yet implemented. The pattern engine has no DOB detector — the category is accepted in config for forward compatibility but never produces a match. Do not rely on it to remove dates of birth.
Section titled “Related”