Reranking

Rerank candidate documents by joint relevance scoring. After vector retrieval returns top-K candidates, rerank to surface the most relevant documents for LLM context.

How it works

Vector similarity uses bi-encoders: the query and each document are embedded independently, then compared by dot product or cosine. This is fast and parallel — ideal for first-pass retrieval over millions of documents — but the query and document never see each other during encoding.

Reranking uses cross-encoders: each (query, document) pair is scored together by a transformer, so the two attend to each other across every layer. That yields far more accurate relevance scores, at the cost of one forward pass per candidate.

Use reranking as the second pass in a retrieval pipeline: retrieve a candidate set cheaply (top-100 via vector search or BM25), rerank it with a cross-encoder, then pass the top-k into your LLM context. This keeps the recall of vector search while sharpening the precision of what reaches the model.

Quick example

Use the fast preset to rerank three documents against a query.

Python

The Python binding has no standalone rerank/rerank_sync entry point — only reranker-backend registration (register_reranker_backend, list_reranker_backends, unregister_reranker_backend) and the RerankerConfig/RerankerModelType types are exposed. To run the dispatcher shown below, call it from Rust (or a language whose binding does expose it):

from xberg import RerankerConfig, RerankerModelType

# NOTE: The Python binding has no standalone reranking entry point — RerankerConfig
# and RerankerModelType are exposed so you can build config for a plugin backend, but
# there is no rerank()/rerank_sync() function to call it with. See the "In-process
# plugin backend" section below to register your own model instead.
config = RerankerConfig(
    model=RerankerModelType(type="preset", name="fast"),
    top_k=2,
)

TypeScript

// NOTE: The Node/TypeScript binding has no standalone rerank()/rerankSync() entry
// point — only reranker-backend registration (registerRerankerBackend,
// listRerankerBackends, unregisterRerankerBackend) and the RerankerConfig type are
// exposed. See the "In-process plugin backend" section below to register your own
// model instead of calling a dispatcher directly.
import type { RerankerConfig } from "@xberg-io/xberg";

const config: RerankerConfig = {
  model: { type: "preset", name: "fast" },
  top_k: 2,
};

Rust

use xberg::{rerank, RerankerConfig, RerankerModelType};

let query = "How to train a dog".to_string();
let documents = vec![
    "Dog training requires patience and consistency.".to_string(),
    "Cats are independent animals that prefer to play alone.".to_string(),
    "Bird care includes proper cage setup and regular cleaning.".to_string(),
];

let config = RerankerConfig {
    model: RerankerModelType::Preset { name: "fast".to_string() },
    top_k: Some(2),
    ..Default::default()
};

let results = rerank(query, documents, &config)?;
for r in results {
    println!("#{}: {:.3} — {}", r.index, r.score, r.document);
}
# Ok::<(), xberg::XbergError>(())

Go

// NOTE: The Go binding has no standalone Rerank() entry point — only reranker-backend
// registration (RegisterRerankerBackend, ListRerankerBackends, UnregisterRerankerBackend)
// and the RerankerConfig type are exposed. See the "In-process plugin backend" section
// below to register your own model instead of calling a dispatcher directly.
import "github.com/xberg-io/xberg"

config := xberg.RerankerConfig{
    Model: &xberg.RerankerModelTypePreset{Name: "fast"},
    TopK:  xberg.Ptr(uint(2)),
}

Java

// NOTE: The Java binding has no standalone Xberg.rerank() entry point — only
// reranker-backend registration (Xberg.registerRerankerBackend, listRerankerBackends,
// unregisterRerankerBackend) and the RerankerConfig type are exposed. See the
// "In-process plugin backend" section below to register your own model instead of
// calling a dispatcher directly.
import io.xberg.*;

RerankerConfig config = new RerankerConfig.Builder()
    .model(new RerankerModelType.Preset("fast"))
    .topK(2L)
    .build();

Picking a preset

Preset	When to use
fast	Latency-critical retrieval, English-only. ~37M parameters, 8192-token context.
balanced	Production English RAG. ~150M parameters (ModernBERT-based), ~8000-token context, the recommended default.
quality	Complex queries where accuracy matters more than latency. 568M parameters, 100+ languages, 8192-token context.
multilingual	International documents needing top accuracy. Same underlying 568M-parameter model as quality, 100+ languages, 8192-token context.

All four download lazily from HuggingFace on first use and cache under ~/.cache/xberg/rerankers/.

Custom HuggingFace cross-encoder

To use any ONNX cross-encoder from HuggingFace, point the Custom variant at its repository ID. The repo must contain an onnx/model.onnx file.

# NOTE: The Python binding has no standalone rerank()/rerank_sync() entry point —
# RerankerConfig/RerankerModelType are exposed to build config, but there is no
# function to run the dispatcher shown here directly from Python.
from xberg import RerankerConfig, RerankerModelType

config = RerankerConfig(
    model=RerankerModelType(
        type="custom",
        model_id="cross-encoder/ms-marco-MiniLM-L-12-v2",
        max_length=512,
    ),
)

LLM rerank via liter-llm

For provider-hosted rerankers, use the Llm variant with a liter-llm model identifier. The model string must include the provider prefix (cohere/, jina/, voyage/).

# NOTE: The Python binding has no standalone rerank()/rerank_sync() entry point —
# RerankerConfig/RerankerModelType/LlmConfig are exposed to build config, but there
# is no function to run the dispatcher shown here directly from Python.
import os
from xberg import RerankerConfig, RerankerModelType, LlmConfig

config = RerankerConfig(
    model=RerankerModelType(
        type="llm",
        llm=LlmConfig(
            model="cohere/rerank-english-v3.0",
            api_key=os.environ["COHERE_API_KEY"],
        ),
    ),
    top_k=5,
)

Set COHERE_API_KEY (or JINA_API_KEY, VOYAGE_API_KEY) in the environment. The Llm variant requires the liter-llm Cargo feature.

In-process plugin backend

To wrap a model you already load — sentence-transformers, llama-cpp-python, a tuned ONNX session — implement the RerankerBackend protocol and register it once at startup.

The protocol returns raw scores in input order. The dispatcher handles sorting and top_k truncation; the plugin must not sort.

Scores can be any sequence of floats — a NumPy array or a plain list. The bridge reads both, so a model that already returns an array needs no conversion.

# NOTE: register_reranker_backend() is real and registers your model in xberg's
# Rust-side registry, but the Python binding exposes no rerank()/rerank_sync() to
# invoke it afterward — invocation must come from a caller that does have a rerank
# entry point (currently only the Rust crate).
from collections.abc import Iterable

from xberg import register_reranker_backend, RerankerConfig, RerankerModelType

class MyReranker:
    def __init__(self):
        from sentence_transformers import CrossEncoder
        self._model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

    def name(self) -> str:
        return "my-reranker"

    def version(self) -> str:
        return "1.0.0"

    def initialize(self) -> None:
        pass

    def shutdown(self) -> None:
        pass

    async def rerank(self, query: str, documents: list[str]) -> Iterable[float]:
        return self._model.predict([(query, doc) for doc in documents])   # ndarray
        # return [0.82, 0.13]   # a plain list works the same


register_reranker_backend(MyReranker())

config = RerankerConfig(model=RerankerModelType(type="plugin", name="my-reranker"))

The Plugin variant loads no ONNX Runtime model, but the rerank dispatch is still gated behind the reranker feature. On no-ORT targets — WebAssembly and the Android x86_64 emulator — rerank and rerank_async compile to stubs that return an error, so the Plugin variant is not available there.

Async execution and Tokio runtimes

rerank_async is the async counterpart to rerank. For the Preset and Custom (ONNX) paths it offloads the blocking inference to Tokio’s blocking thread pool via spawn_blocking, keeping the async executor free; for the Llm and Plugin backends it awaits the backend directly.

The synchronous rerank entry point drives Llm and Plugin backends by blocking on their async work. It cannot run inside a current-thread Tokio runtime — doing so returns an error telling you to call rerank_async or build a multi-thread runtime. The Preset and Custom (ONNX) paths carry no such restriction and run on any runtime or none.

Performance notes

• batch_size controls how many (query, document) pairs share a forward pass. The default of 32 is a good fit for CPU; raise to 64-128 on GPU.
• top_k truncates the response after scoring — it does not reduce inference cost. Always score the full candidate set, then pick.
• Sigmoid normalization is applied automatically to local-model logits so scores fall in [0, 1]. LLM rerankers return provider-native scores unchanged.
• First-call latency is dominated by model download. Warm the cache during application startup, not on the first user request.

See also

• Retrieval — build the candidate set this reranker sharpens
• Retrieval Modes — where reranking sits relative to dense, sparse and late-interaction