LateOn-regularized

About

What LateOn-regularized is

[](https://lighton.ai) [](https://www.linkedin.com/company/lighton/) [](https://x.com/LightOnIO) 📚 Collection | 📝 Blog Overview LateOn-regularized is a ColBERT (multi-vector) retrieval model built on the LateOn methology, trained by LightOn using PyLate. It applies a novel Straight-Through Estimator (STE) regularization during supervised fine-tuning that makes the model compatible with efficient ANN candidate-generation methods like MUVERA and SMVE, while fully preserving PLAID retrieval performance. This model is the recommended choice when you plan to use MUVERA or SMVE for candidate generation in your ColBERT retrieval pipeline. If you only use PLAID indexes, LateOn is equivalent and equally recommended. For full details on the problem, methodology, and analysis, please read our blog post. Why This Model Exists Recent ColBERT models (including LateOn) produce highly anisotropic token embeddings: nearly all vectors cluster into a narrow cone with mean pairwise cosine similarity as high as 0.95. While this does not affect PLAID-based retrieval, it breaks ANN methods like MUVERA and SMVE that rely on random projections to separate embeddings. When all vectors point in roughly the same direction, random hyperplanes and rotations cannot distinguish them, making candidate generation essentially random. Mean-centering the embeddings removes the dominant shared direction and helps significantly, but isn't enough on its own as the residual geometry still doesn't project cleanly. STE-based regularization solves this by directly optimizing the model to produce discriminative representations after MUVERA/SMVE projection. During training, we build the true projected codes in the forward pass and compute a contrastive loss on them, then use the Straight-Through Estimator to pass gradients through the non-differentiable discretization steps (SimHash bucketing in MUVERA, top-k sparsification in SMVE). The total loss is a weighted combination of the standard MaxSim contrastive loss and the projected-representation contrastive loss. Surprisingly, the regularization does not make the space more isotropic. Instead, it teaches the model to concentrate discriminative information i

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How embedding models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys lateon-regularized for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (lateon-regularized below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/embeddings \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"lateon-regularized","input":"text to embed"}'

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More embedding models