Reranking

Rerank candidate documents by joint relevance scoring. After vector retrieval returns top-K candidates, rerank to surface the most relevant documents for LLM context.

How it works

Vector similarity uses bi-encoders: the query and each document are embedded independently, then compared by dot product or cosine. This is fast and parallel — ideal for first-pass retrieval over millions of documents — but the query and document never see each other during encoding.

Reranking uses cross-encoders: each (query, document) pair is scored together by a transformer, so the two attend to each other across every layer. That yields far more accurate relevance scores, at the cost of one forward pass per candidate.

Use reranking as the second pass in a retrieval pipeline: retrieve a candidate set cheaply (top-100 via vector search or BM25), rerank it with a cross-encoder, then pass the top-k into your LLM context. This keeps the recall of vector search while sharpening the precision of what reaches the model.

Quick example

Use the fast preset to rerank three documents against a query.

Python

from xberg import rerank_sync, RerankerConfig, RerankerModelType

query = "How to train a dog"
documents = [
    "Dog training requires patience and consistency.",
    "Cats are independent animals that prefer to play alone.",
    "Bird care includes proper cage setup and regular cleaning.",
]

config = RerankerConfig(
    model=RerankerModelType(type="preset", name="fast"),
    top_k=2,
)

results = rerank_sync(query, documents, config)
for result in results:
    print(f"#{result.index}: {result.score:.3f} — {result.document}")

TypeScript

import { rerankSync, RerankerConfig } from "@xberg-io/xberg";

const config: RerankerConfig = {
  model: { type: "preset", name: "fast" },
  top_k: 2,
};

const results = rerankSync(
  "How to train a dog",
  [
    "Dog training requires patience and consistency.",
    "Cats are independent animals that prefer to play alone.",
    "Bird care includes proper cage setup and regular cleaning.",
  ],
  config,
);

for (const r of results) {
  console.log(`#${r.index}: ${r.score.toFixed(3)} — ${r.document}`);
}

Rust

use xberg::{rerank, RerankerConfig, RerankerModelType};

let query = "How to train a dog".to_string();
let documents = vec![
    "Dog training requires patience and consistency.".to_string(),
    "Cats are independent animals that prefer to play alone.".to_string(),
    "Bird care includes proper cage setup and regular cleaning.".to_string(),
];

let config = RerankerConfig {
    model: RerankerModelType::Preset { name: "fast".to_string() },
    top_k: Some(2),
    ..Default::default()
};

let results = rerank(query, documents, &config)?;
for r in results {
    println!("#{}: {:.3} — {}", r.index, r.score, r.document);
}
# Ok::<(), xberg::XbergError>(())

Go

import "github.com/xberg-io/xberg"

config := xberg.RerankerConfig{
    Model: &xberg.RerankerModelTypePreset{Name: "fast"},
    TopK:  xberg.Ptr(uint(2)),
}
results, err := xberg.Rerank(
    "How to train a dog",
    []string{
        "Dog training requires patience and consistency.",
        "Cats are independent animals that prefer to play alone.",
        "Bird care includes proper cage setup and regular cleaning.",
    },
    &config,
)
if err != nil {
    log.Fatal(err)
}
for _, r := range results {
    fmt.Printf("#%d: %.3f — %s\n", r.Index, r.Score, r.Document)
}

Java

import io.xberg.*;

RerankerConfig config = new RerankerConfig.Builder()
    .model(new RerankerModelType.Preset("fast"))
    .topK(2L)
    .build();

var results = Xberg.rerank(
    "How to train a dog",
    java.util.List.of(
        "Dog training requires patience and consistency.",
        "Cats are independent animals that prefer to play alone.",
        "Bird care includes proper cage setup and regular cleaning."
    ),
    config
);

for (var r : results) {
    System.out.printf("#%d: %.3f — %s%n", r.getIndex(), r.getScore(), r.getDocument());
}

Picking a preset

Preset	When to use
fast	Latency-critical retrieval, English-only. ~37M parameters, 8192-token context.
balanced	Production English RAG. ~150M parameters (ModernBERT-based), ~8000-token context, the recommended default.
quality	Complex queries where accuracy matters more than latency. 568M parameters, 100+ languages, 8192-token context.
multilingual	International documents needing top accuracy. Same underlying 568M-parameter model as quality, 100+ languages, 8192-token context.

All four download lazily from HuggingFace on first use and cache under ~/.cache/xberg/rerankers/.

Custom HuggingFace cross-encoder

To use any ONNX cross-encoder from HuggingFace, point the Custom variant at its repository ID. The repo must contain an onnx/model.onnx file.

from xberg import rerank_sync, RerankerConfig, RerankerModelType

config = RerankerConfig(
    model=RerankerModelType(
        type="custom",
        model_id="cross-encoder/ms-marco-MiniLM-L-12-v2",
        max_length=512,
    ),
)

results = rerank_sync("query text", ["doc1", "doc2"], config)

LLM rerank via liter-llm

For provider-hosted rerankers, use the Llm variant with a liter-llm model identifier. The model string must include the provider prefix (cohere/, jina/, voyage/).

import os
from xberg import rerank_sync, RerankerConfig, RerankerModelType, LlmConfig

config = RerankerConfig(
    model=RerankerModelType(
        type="llm",
        llm=LlmConfig(
            model="cohere/rerank-english-v3.0",
            api_key=os.environ["COHERE_API_KEY"],
        ),
    ),
    top_k=5,
)

results = rerank_sync("query text", documents, config)

Set COHERE_API_KEY (or JINA_API_KEY, VOYAGE_API_KEY) in the environment. The Llm variant requires the liter-llm Cargo feature.

In-process plugin backend

To wrap a model you already load — sentence-transformers, llama-cpp-python, a tuned ONNX session — implement the RerankerBackend protocol and register it once at startup.

The protocol returns raw scores in input order. The dispatcher handles sorting and top_k truncation; the plugin must not sort.

Scores can be any sequence of floats — a NumPy array or a plain list. The bridge reads both, so a model that already returns an array needs no conversion.

from collections.abc import Iterable

from xberg import register_reranker_backend, rerank_sync, RerankerConfig, RerankerModelType

class MyReranker:
    def __init__(self):
        from sentence_transformers import CrossEncoder
        self._model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

    def name(self) -> str:
        return "my-reranker"

    def version(self) -> str:
        return "1.0.0"

    def initialize(self) -> None:
        pass

    def shutdown(self) -> None:
        pass

    async def rerank(self, query: str, documents: list[str]) -> Iterable[float]:
        return self._model.predict([(query, doc) for doc in documents])   # ndarray
        # return [0.82, 0.13]   # a plain list works the same


register_reranker_backend(MyReranker())

config = RerankerConfig(model=RerankerModelType(type="plugin", name="my-reranker"))
results = rerank_sync("query text", ["doc1", "doc2"], config)

The Plugin variant loads no ONNX Runtime model, but the rerank dispatch is still gated behind the reranker feature. On no-ORT targets — WebAssembly and the Android x86_64 emulator — rerank and rerank_async compile to stubs that return an error, so the Plugin variant is not available there.

Async execution and Tokio runtimes

rerank_async is the async counterpart to rerank. For the Preset and Custom (ONNX) paths it offloads the blocking inference to Tokio’s blocking thread pool via spawn_blocking, keeping the async executor free; for the Llm and Plugin backends it awaits the backend directly.

The synchronous rerank entry point drives Llm and Plugin backends by blocking on their async work. It cannot run inside a current-thread Tokio runtime — doing so returns an error telling you to call rerank_async or build a multi-thread runtime. The Preset and Custom (ONNX) paths carry no such restriction and run on any runtime or none.

Performance notes

• batch_size controls how many (query, document) pairs share a forward pass. The default of 32 is a good fit for CPU; raise to 64-128 on GPU.
• top_k truncates the response after scoring — it does not reduce inference cost. Always score the full candidate set, then pick.
• Sigmoid normalization is applied automatically to local-model logits so scores fall in [0, 1]. LLM rerankers return provider-native scores unchanged.
• First-call latency is dominated by model download. Warm the cache during application startup, not on the first user request.

See also

• Retrieval — build the candidate set this reranker sharpens
• Retrieval Modes — where reranking sits relative to dense, sparse and late-interaction