Agent-ModernColBERT

Tl;Dr

A few weeks ago, we evaluated Reason-ModernColBERT, a 150M late-interaction model trained on ReasonIR data nearly solved BrowseComp-Plus, reaching 87.56% accuracy with GPT-5 (a +7.59 absolute jump over the previous SOTA) while topping recall and calibration error, while not being trained for agentic retrieval at all (and being one year old). We now present Agent-ModernColBERT, a model specifically fine-tuned for agentic retrieval using the AgentIR dataset released alongside AgentIR. You can find the training boilerplate here. This very lightweight fine-tuning adds increase the performance of Reason-ModernColBERT by another 10%, which allows, when exposing the get_document function and the GPT-OSS-120B model, to beat the original GPT-5 + Qwen3-8B runs, while using a retriever model 54× smaller and an open source LLM.

How it works

Before issuing a query, deep research agents generate explicit reasoning traces describing what they're looking for and why. Conventional retrievers throw all of that away. The AgentIR paper introduced Reasoning-Aware Retrieval: instead of discarding those reasoning traces, concatenate them to the query you send to the retriever. To enable this, the authors generated trajectories from a deep-research agent and released them (the Tevatron/AgentIR-data dataset), then trained AgentIR-4B, a 4B dense model that beats much larger baselines including ReasonIR-8B and rerank pipelines.

BrowseComp-Plus Results

The get_document column indicates whether the LLM was exposed to the possibility of using the get_document function that allows, in addition to the usual search function that returns a snippet of the top-5 documents for a given query (the beginning of the document), to request the full content of a document using its id. It allows the model to get more information when needed, but can also add noise to the process if the model reads full documents that are irrelevant, and thus require a precise retrieval signal.

A few things worth highlighting:

• With the get_document function exposed, Agent-ModernColBERT + GPT-OSS-120B reaches 72.53, higher than the original GPT-5 + Qwen3-8B setup (which doesn't expose get_document).

• Agent-ModernColBERT is competitive with the single-vector AgentIR-4B despite being 26× smaller and trained on the exact same data, direct evidence that late interaction is the right inductive bias for reasoning-trace-augmented queries.

• We use the same prompt as AgentIR to keep the comparison clean (down to a wrongly-escaped line break, for full reproducibility). The prompt has a small but positive effect, as discussed in the next section.

Query format (⚠️ important)

Agent-ModernColBERT was trained on the AgentIR query format, which prepends an instruction and concatenates the agent's reasoning trace with the search query. Documents are encoded as plain text, no prefix.

The exact query layout is:

A few non-obvious details worth flagging:

• The instruct prefix itself ends with a literal \nQuery: (a backslash followed by n, 2 characters — not a newline). This is inherited byte-for-byte from the AgentIR training script and the model expects it.
• As a consequence, the word Query: appears twice in every query — once at the end of the prefix, once as a real label before the search query.
• If you don't have a reasoning trace, pass the literal string "Empty" (this matches the AgentIR agent loop).

Helper:

INSTRUCT_PREFIX = (
    "Instruct: Given a user's reasoning followed by a web search query, "
    "retrieve relevant passages that answer the query while incorporating the user's reasoning"
    "\\nQuery:"  # literal backslash-n, NOT a newline — keep as-is
)

def format_query(reasoning: str | None, search_query: str) -> str:
    r = reasoning if reasoning else "Empty"
    return f"{INSTR