Agent-ModernColBERT-no-prefix

Tl;Dr

A few weeks ago, we evaluatedReason-ModernColBERT, a 150M late-interaction model trained on ReasonIR data nearly solved BrowseComp-Plus, reaching 87.56% accuracy with GPT-5 (a +7.59 absolute jump over the previous SOTA) while topping recall and calibration error, while not being trained for agentic retrieval at all (and being one year old). We now present Agent-ModernColBERT, a model specifically fine-tuned for agentic retrieval using the AgentIR dataset released alongside AgentIR. You can find the training boilerplate here. This very lightweight fine-tuning adds increase the performance of Reason-ModernColBERT by another 10%, which allows, when exposing the get_document function and the GPT-OSS-120B model, to beat the original GPT-5 + Qwen3-8B runs, while using a retriever model 54× smaller and an open source LLM.

How it works

Before issuing a query, deep research agents generate explicit reasoning traces describing what they're looking for and why. Conventional retrievers throw all of that away. The AgentIR paper introduced Reasoning-Aware Retrieval: instead of discarding those reasoning traces, concatenate them to the query you send to the retriever. To enable this, the authors generated trajectories from a deep-research agent and released them (the Tevatron/AgentIR-data dataset), then trained AgentIR-4B, a 4B dense model that beats much larger baselines including ReasonIR-8B and rerank pipelines.

BrowseComp-Plus Results

The get_document column indicates whether the LLM was exposed to the possibility of using the get_document function that allows, in addition to the usual search function that returns a snippet of the top-5 documents for a given query (the beginning of the document), to request the full content of a document using its id. It allows the model to get more information when needed, but can also add noise to the process if the model reads full documents that are irrelevant, and thus require a precise retrieval signal.

A few things worth highlighting:

• With the get_document function exposed, Agent-ModernColBERT + GPT-OSS-120B reaches 72.53, higher than the original GPT-5 + Qwen3-8B setup (which doesn't expose get_document).

• Agent-ModernColBERT is competitive with the single-vector AgentIR-4B despite being 26× smaller and trained on the exact same data, direct evidence that late interaction is the right inductive bias for reasoning-trace-augmented queries.

• We use the same prompt as AgentIR to keep the comparison clean (down to a wrongly-escaped line break, for full reproducibility). The prompt has a small but positive effect, as discussed in the next section.

Query format (⚠️ important)

The main Agent-ModernColBERT was trained on the AgentIR query format, which prepends an instruction and concatenates the agent's reasoning trace with the search query. Documents are encoded as plain text, no prefix. This version of the model is the one trained without the instruction prefix, which yields slighlty lower performance but does not require the prefix:

Helper:

def format_query(reasoning: str | None, search_query: str) -> str:
    r = reasoning if reasoning else "Empty"
    return f"Reasoning: {r}\n\nQuery: {search_query}"

Then encode normally with is_query=True for queries and is_query=False for documents:

from pylate import models

model = models.ColBERT(model_name_or_path="lightonai/Agent-ModernColBERT")

queries = [
    format_query(
        reasoning="The user wants a code-level explanation of PLAID, so passages "
                  "with implementation details should be prioritized.",
        search_query="What does the PLAID centroid scoring step do?",
    ),
]
documents = [
    "PLAID first scores documents against quantized centroids of the ColBERT "
    "