LateOn

📚 Collection | 📝 Blog

About the LateOn / DenseOn Family

State-of-the-art retrieval is increasingly dominated by closed models, either hidden behind APIs or trained on undisclosed data. This blocks reproducibility, prevents study of possible data leakage, and gatekeeps progress to a handful of private labs. We thus decided to gather and curate a large amount of data and explore various mixtures. We release all the data used in our explorations:

• Gathered pre-training data, 1.4B query-documents pairs alongside annotations used for non-destructive filtering (structural filtering, deduplication, cross-encoder pair relevancy)
• Best pre-training mixture found with already applied filters
• Fine-tuning datasets with query, positive and 2048 mined documents alongside their scores for 1.88M samples.

Based on our findings, we trained LateOn (multi-vector/ColBERT) and DenseOn (single vector/dense) models on a proprietary Apache 2.0-compatible training dataset and release those models as well. Both are built on the ModernBERT backbone at 149M parameters, a size we believe sits at the sweet spot: large enough to handle real-world queries and documents, small enough to serve at high throughput in latency-sensitive production systems. For more information, please read our blogpost.

LateOn

LateOn is a ColBERT (multi-vector) retrieval model built on ModernBERT (149M parameters), trained by LightOn using PyLate. This is the supervised version, fine-tuned with hard-negative contrastive training. The unsupervised version can be found here

Notably it:

• Beats every existing ColBERT model, including those 4× its size (Jina ColBERT v2 at 559M, Arctic Embed L v2 at 568M).
• Holds up under decontamination: when training-overlap samples are stripped from the BEIR corpora, LateOn climbs to 60.36 nDCG@10 on the 12-dataset decontaminated split — first place overall.
• Uses fully open data for both pre-training and fine-tuning, with all signals released as metadata so you can rebuild, extend, or replace any filter.

This release was focused on contrastive data exploration. We did not run a knowledge distillation phase nor use asymmetric prompts (both shown to help in our ColBERT-Zero study). We believe even stronger results are within reach by adding these — stay tuned.

LateOn achieves 57.22 average NDCG@10 on BEIR (15 datasets) and 60.36 on decontaminated BEIR (12 datasets), leading all ColBERT models. See our blog post for full results and analysis.

Alongside LateOn, we also trained DenseOn, a dense (single vector) variant, trained on the same setup. This variant is easier and cheaper to use, albeit with slightly weaker results (achieving a score of 56.20 on BEIR, it is still stronger than any dense base-sized model). It may also suffer from some limitations compared to LateOn, such as in terms of generalisation and long context.

Results

BEIR (15 datasets, NDCG@10)

LateOn achieves 57.22 average NDCG@10, becoming the first ColBERT (and sub 150M parameter) model to break the 57 mark on BEIR by surpassing the previous best ColBERT model (ColBERT-Zero at 55.32) by almost two points and exceeding GTE-ModernColBERT-v1 by two point and a half, despite sharing the same backbone. It is worth noting that we achieve this performance only with contrastive data and without t