USER2

Performance

To evaluate the model, we measure quality on the MTEB-rus benchmark. Additionally, to measure long-context retrieval, we run Russian subset of MultiLongDocRetrieval (MLDR) task.

MTEB-rus

MLDR-rus

We compare only model with context length of 8192.

Matryoshka

To evaluate MRL capabilities, we also use MTEB-rus, applying dimensionality cropping to the embeddings to match the selected size.

Usage

Prefixes

This model is trained similarly to Nomic Embed and expects task-specific prefixes to be added to the input. The choice of prefix depends on the specific task. We follow a few general guidelines when selecting a prefix:

• "classification: " is the default and most universal prefix, often performing well across a variety of tasks.
• "clustering: " is recommended for clustering applications: group texts into clusters, discover shared topics, or remove semantic duplicates.
• "search_query: " and "search_document: " are intended for retrieval and reranking tasks. Also, in some classification tasks, especially with shorter texts, "search_query" shows superior performance to other prefixes. On the other hand, "search_document" can be beneficial for long-context sentence similarity tasks.

However, we encourage users to experiment with different prefixes, as certain domains may benefit from specific ones.

Sentence Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("deepvk/USER2-base")

query_embeddings = model.encode(["Когда был спущен на воду первый миноносец «Спокойный»?"], prompt_name="search_query")
document_embeddings = model.encode(["Спокойный (эсминец)\nЗачислен в списки ВМФ СССР 19 августа 1952 года."], prompt_name="search_document")

similarities = model.similarity(query_embeddings, document_embeddings)

To truncate the embedding dimension, simply pass the new value to the model initialization:

model = SentenceTransformer("deepvk/USER2-base", truncate_dim=128)

This model was trained with dimensions [32, 64, 128, 256, 384, 512, 768], so it’s recommended to use one of these for best performance.

Transformers

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = (
        attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    )
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(
        input_mask_expanded.sum(1), min=1e-9
    )

queries = ["search_query: Когда был спущен на воду первый миноносец «Спокойный»?"]
document