Code Intelligence

Xberg integrates tree-sitter-language-pack (TSLP) to parse source code files. When you extract a source code file, Xberg detects the programming language, parses it with tree-sitter, and emits the source as content split at semantic boundaries – functions, classes, and modules – rather than fixed line counts.

Language support covers 371 programming languages via tree-sitter grammars. See the TSLP documentation for the full language list.

Not available in the WASM build

Code intelligence is excluded from the WebAssembly build (@xberg-io/xberg-wasm). The statically linked tree-sitter grammar pack for all 371 languages pushes the browser .wasm binary well past the 50 MB per-file limit imposed by public CDNs such as jsDelivr. Source files still extract as plain text in WASM, but they are not parsed or split at semantic boundaries. Use a native binding (Rust, Python, Node, Go, etc.) when you need code intelligence.

See the TreeSitterConfig reference for all configuration options.

What You Get

Extracting a source code file produces:

• content – the source text, split into code segments at tree-sitter chunk boundaries and rendered in the configured output format. When chunking is enabled, each segment is preceded by a heading naming its enclosing function, class, or module. With chunking disabled (the default), the whole source is emitted as a single code block.
• metadata.format – tagged as code (format_type: "code"), carrying the structural chunks and, when data extraction is enabled, the hierarchical data tree.
• code_intelligence – tree-sitter’s full analysis as an opaque JSON object: language, metrics, structure, imports, exports, comments, docstrings, symbols, diagnostics, chunks, and the data tree. It is populated for source files when the tree-sitter feature is enabled, and null otherwise.

code_intelligence is the serialized tree_sitter_language_pack::ProcessResult. It is deliberately untyped so every binding (Go, Java, C#, …) can read it as a raw JSON object; deserialize it against TSLP’s own schema if you need typed access.

Getting Started

Code extraction is enabled by default when the tree-sitter feature flag is active. Extract a source code file and read content:

Rust

basic.rs

use xberg::{extract, ExtractInput, ExtractionConfig};

let config = ExtractionConfig::default();
let output = extract(ExtractInput::from_uri("app.py"), &config).await?;
let result = &output.results[0];

// The rendered source is in the content field.
println!("{}", result.content);

// metadata.format is tagged as Code and carries the structural chunks.
if let Some(xberg::types::FormatMetadata::Code(code)) = &result.metadata.format {
    println!("Detected a source code file with {} chunks", code.chunks.len());
}

// The full tree-sitter analysis is an opaque JSON value.
if let Some(intelligence) = &result.code_intelligence {
    println!("{intelligence}");
}

Python

basic.py

import json

import xberg

config = xberg.ExtractionConfig()
output = await xberg.extract(xberg.ExtractInput(kind="uri", uri="app.py"), config=config)
result = output.results[0]

# The rendered source is in the content field.
print(result.content)

# metadata["format"] is tagged as code and carries the structural chunks.
fmt = result.metadata.get("format")
if fmt and fmt.get("format_type") == "code":
    print(f"Detected a source code file with {len(fmt['chunks'])} chunks")

# The full tree-sitter analysis is a JSON string in the Python binding.
if result.code_intelligence is not None:
    print(json.loads(result.code_intelligence)["symbols"])

TypeScript

basic.ts

import { ExtractInputKind, extract } from "@xberg-io/xberg";

const output = await extract({
  kind: ExtractInputKind.Uri,
  uri: "app.ts",
});
const result = output.results[0];

// The rendered source is in the content field.
console.log(result.content);

// metadata.format is tagged "code" and carries the structural chunks.
const fmt = result.metadata?.format;
if (fmt?.formatType === "code") {
  console.log(`Detected a source code file with ${fmt.chunks.length} chunks`);
}

// The full tree-sitter analysis is an unknown-typed JSON object.
if (result.codeIntelligence) {
  console.log(result.codeIntelligence);
}

Go

basic.go

result, err := xberg.Extract("app.py", nil)
if err != nil {
    log.Fatal(err)
}

fmt.Println(result.Content)
// result.Metadata.Format is tagged "code" for source files, and
// result.CodeIntelligence carries the full tree-sitter analysis as raw JSON.

Configuration

Use TreeSitterConfig to control tree-sitter processing. Set enabled: false to skip code intelligence entirely. chunk_max_size controls where the source is split into segments; when unset (the default), the whole file is emitted as a single block.

Which fields each toggle affects

enabled and chunk_max_size shape content and metadata.format. The structure, imports, exports, comments, docstrings, symbols, and diagnostics toggles are passed through to tree-sitter and decide which sections of code_intelligence are populated; they do not change content.

Rust

config.rs

use xberg::{ExtractionConfig, TreeSitterConfig, TreeSitterProcessConfig};

let config = ExtractionConfig {
    tree_sitter: Some(TreeSitterConfig {
        process: TreeSitterProcessConfig {
            chunk_max_size: Some(4096),  // split source at chunk boundaries
            ..Default::default()
        },
        ..Default::default()
    }),
    ..Default::default()
};

Python

config.py

import xberg

config = xberg.ExtractionConfig(
    tree_sitter={
        "process": {
            "chunk_max_size": 4096,
        }
    }
)

TypeScript

config.ts

import { ExtractionConfig } from "@xberg-io/xberg";

const config: ExtractionConfig = {
  treeSitter: {
    process: {
      chunkMaxSize: 4096,
    },
  },
};

TOML

xberg.toml

[tree_sitter.process]
chunk_max_size = 4096

Configuration Fields

See TreeSitterConfig and TreeSitterProcessConfig for all fields.

Chunked Content

With chunk_max_size set, tree-sitter splits the source at function, class, and module boundaries and Xberg emits each chunk as a separate code segment in content, preceded by a heading naming its enclosing scope:

chunked.py

import xberg

config = xberg.ExtractionConfig(
    tree_sitter={"process": {"chunk_max_size": 2048}}
)

output = await xberg.extract(
    xberg.ExtractInput(kind="uri", uri="large_module.py"), config=config
)
result = output.results[0]

# Chunk boundaries are reflected in the layout of result.content.
print(result.content)

Structured per-chunk data is also available without parsing content: each entry of metadata.format.chunks carries the chunk text, its context_path (enclosing scopes), the tree-sitter node_types, and the byte_start/byte_end offsets into the source. For general-purpose text chunking across all formats – for example, in a RAG pipeline – use Xberg’s chunking pipeline instead.

Language Detection

Xberg detects the programming language in two ways:

1. File extension (fast path) – when using extract, the extension is matched against 248 known language extensions
2. Shebang line (fallback) – when the extension is missing or ambiguous, the first line is checked for #!/usr/bin/env python, #!/bin/bash, and so on.

If neither method identifies the language, extraction returns an UnsupportedFormat error. The detected language drives parsing and is reported in code_intelligence.

Language Support

Tree-sitter-language-pack supports 371 programming languages, including Python, Rust, TypeScript, JavaScript, Go, Java, C/C++, Ruby, PHP, C#, Swift, Kotlin, and Elixir. For the full list, see the TSLP language reference.

Related Documentation

• Configuration Reference – TreeSitterConfig and TreeSitterProcessConfig fields
• Chunking Guide – programmatic chunking with offsets and metadata
• tree-sitter-language-pack documentation – Full language support reference