modernbert-embed-large

ModernBERT-embed-large is an embedding model trained from ModernBERT-large, bringing the new advances of ModernBERT to embeddings!

Indeed, ModernBERT is a base model trained for Masked Language Modeling and can not directly be used to perform tasks such as retrieval without further fine-tuning.

ModernBERT-embed-large is fine-tuned on the Nomic Embed weakly-supervised and supervised datasets and also supports Matryoshka Representation Learning dimensions of 256 to reduce memory with minimal performance loss.

Performance

Usage

You can use these models directly with the latest transformers release and requires installing transformers>=4.48.0:

pip install transformers>=4.48.0

Reminder, this model is trained similarly to Nomic Embed and REQUIRES prefixes to be added to the input. For more information, see the instructions in Nomic Embed.

Most use cases, adding search_query: to the query and search_document: to the documents will be sufficient.

Sentence Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("lightonai/modernbert-embed-large")

query_embeddings = model.encode([
    "search_query: What is TSNE?",
    "search_query: Who is Laurens van der Maaten?",
])
doc_embeddings = model.encode([
    "search_document: TSNE is a dimensionality reduction algorithm created by Laurens van Der Maaten",
])
print(query_embeddings.shape, doc_embeddings.shape)
# (2, 1024) (1, 1024)

similarities = model.similarity(query_embeddings, doc_embeddings)
print(similarities)
# tensor([[0.6518],
#         [0.4237]])

In Sentence Transformers, you can truncate embeddings to a smaller dimension by using the truncate_dim parameter when loading the SentenceTransformer model.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("lightonai/modernbert-embed-large", truncate_dim=256)

query_embeddings = model.encode([
    "search_query: What is TSNE?",
    "search_query: Who is Laurens van der Maaten?",
])
doc_embeddings = model.encode([
    "search_document: TSNE is a dimensionality reduction algorithm created by Laurens van Der Maaten",
])
print(query_embeddings.shape, doc_embeddings.shape)
# (2, 256) (1, 256)

similarities = model.similarity(query_embeddings, doc_embeddings)
print(similarities)
# tensor([[0.6835],
#         [0.3982]])

Note the small differences compared to the full 1024-dimensional similarities.

Transformers

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = (
        attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    )
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(
        input_mask_expanded.sum(1), min=1e-9
    )

queries = ["search_query: What is TSNE?", "search_query: Who is Laurens van der Maaten?"]
documents = ["search_document: TSNE is a dimensionality reduction algorithm created by Laurens van Der Maaten"]

tokenizer = AutoTokenizer.from_pretrained("lightonai/modernbert-embed-large")
model = AutoModel.from_pretrained("lightonai/modernbert-embed-large")

encoded_queries = tokenizer(queries, padding=True, truncation=True, return_tensors="pt")
encoded_documents = tokenizer(documents, padding=True, truncation=True, return_tensors="pt")

with torch.no_grad():
    queries_outputs = model(**encoded_queries)
    documents_outputs = model(**encoded_documents)

query_embeddings = mean_pooling(queries_outputs, encoded_queries["attention_mask"])
query_embeddings = F.normalize(query_embeddings, p=2, dim=1)
doc_embeddings = mean_pooling(documents_outputs, encoded_documents["attention_mask"])
doc_embeddings = F.normalize(doc_embeddings, p=2, dim=1)
print(query_embeddings.shape, doc_embeddings.shape)
# torch.Size([2, 1024]) torch.Size([1, 1024])

similarities = query_embeddings @ doc_embeddings.T
print(similarities)
# tensor([[0.6518],
#         [0.4237]])

In transformers, you can truncate embeddings to a smaller dimension by slicing the mean pooled embeddings, prior to normalization.

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

def mean_poo