mLateOn-unsupervised

📚 Collection | 📝 Multilingual Blog | 📝 English Blog | 📝 Paper

🎯 TL;DR: The intermediate multilingual ColBERT checkpoint produced by Stage 1 only (unsupervised contrastive pre-training) of the mLateOn pipeline, trained on a multilingual dataset with 2.8B query–document pairs across nine languages (including 25% cross-lingual pairs). Released as a strong starting point for your own supervised fine-tuning, knowledge distillation, or downstream adaptation.

About the mDenseOn / mLateOn Family

With DenseOn and LateOn, we demonstrated that an open, carefully curated data recipe can match closed-data retrieval models on English. mDenseOn and mLateOn extend this recipe to multilingual, long-context, and code retrieval.

Rather than independently collecting multilingual corpora from scratch (which would be expensive, uneven across languages, and hard to curate at the same quality), we applied the translate-train approach: machine-translating our validated English data into eight target languages (French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, and Arabic) and adding cross-lingual pairs for cross-lingual alignment.

For more information, please read our multilingual models blog post, our English models blog post and our paper.

mLateOn-unsupervised

mLateOn-unsupervised is the output of the first stage of the mLateOn training pipeline. It has been pre-trained on a large, filtered multilingual corpus of approximately 2.8B query–document pairs across nine languages (including 25% cross-lingual pairs) using in-batch contrastive learning, but has not yet been fine-tuned with mined hard negatives.

For most production use cases, you should use the fully-trained mLateOn instead. This unsupervised checkpoint is intended for:

• Researchers studying what each pipeline stage contributes in a multilingual setting
• Practitioners who want to fine-tune on their own domain-specific or language-specific data
• Distillation experiments where you want to start from a strong but un-aligned multilingual base
• Anyone running their own ablations on hard-negative mining strategies or translate-train recipes

Related Checkpoints

Model Description

• Model Type: PyLate ColBERT model
• Base model: mmBERT-base
• Query/Document Length: 8,192 tokens
• Output Dimensionality: 128 tokens
• Similarity Function: MaxSim
• Languages: English, French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, Arabic
• License: Apache 2.0

Model Sources

• Documentation: PyLate Documentation
• Repository: PyLate on GitHub
• Hugging Face: PyLate models on Hugging Face

Full Model Architecture

ColBERT(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 1536, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'use_residual': True})
  (2): Dense({'in_features': 1536, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'use_residual': True})
  (3): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'use_residual': False})
)

Usage

First install the PyLate library:

pip install -U pylate

Retrieval

Use this model with PyLate to index and retrieve documents. The index uses FastPlaid for efficient similarity search.

Indexing documents

Load the ColBERT model and initialize the PLAID index, then encode and index your documents:

from pylate import indexes, models, retrieve

# Step 1: Load the ColBERT model
model = models.ColBERT(
    model_name_or_path="lightonai/mLateOn-unsupervised",
)

# Step 2: Initialize the PLAID index
index = indexes.PLAID(
    index_folder="pylate-index",
    ind