BidirLM-Embedding

(*) Evaluated at max_seq_length=512, matching the paper. The architecture supports much longer context (see the LongEmbed table below); on the MTEB leaderboard the shared LEMBPasskeyRetrieval task is scored at long context, making the leaderboard MTEB(Multilingual, v2) ~0.4 higher than this table.

LongEmbed (Long-Context Retrieval)

Mean nDCG@10 over the 6 LongEmbed retrieval tasks. Each model is reported at the context length (8k or 32k) that maximizes its average; the architecture supports the base model's full context.

Note: Extending the evaluation context from 8k to 32k helped the Gemma-based models (270M, 1B) but not the Qwen-based models (0.6B, 1.7B), which scored best at 8k.

Supported Tasks

General embeddings (via Sentence Transformers): retrieval, semantic similarity (STS), clustering, classification, pair classification, reranking, bitext mining, multilabel classification

Downstream fine-tuning (via Transformers): sequence classification (e.g. MNLI, XNLI, PAWS-X, MathShepherd), token classification (e.g. PAN-X, POS), information retrieval (e.g. MIRACL, CodeSearchNet), sequence regression (e.g. Seahorse)

Usage

Sentence Transformers

Use Sentence Transformers to compute embeddings for any text representation task.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BidirLM/BidirLM-1B-Embedding", trust_remote_code=True)

queries = [
    "What is the capital of France?",
    "How does photosynthesis work?",
]
documents = [
    "Paris is the capital and largest city of France, situated on the river Seine.",
    "Photosynthesis is the process by which plants convert sunlight, water, and CO2 into glucose and oxygen.",
]

query_embeddings = model.encode(queries)
document_embeddings = model.encode(documents)

similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)

Fine-tuning for Downstream Tasks

BidirLM can be directly fine-tuned for downstream tasks:

from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoModelForTokenClassification

tokenizer = AutoTokenizer.from_pretrained("BidirLM/BidirLM-1B-Embedding", trust_remote_code=True)

# Sequence classification (e.g., NLI: entailment, neutral, contradiction)
seq_model = AutoModelForSequenceClassification.from_pretrained(
    "BidirLM/BidirLM-1B-Embedding",
    trust_remote_code=True,
    num_labels=3,
)

# Token classification (e.g., NER)
tok_model = AutoModelForTokenClassification.from_pretrained(
    "BidirLM/BidirLM-1B-Embedding",
    trust_remote_code=True,
    num_labels=7,
)

# Fine-tune with HuggingFace Trainer

Evaluation

Please follow the mteb repository on how to reproduce our scores. The evaluation prompts used for each task are also available at mteb_v2_eval_prompts.json.

Supported Languages

Multilingual support across over 140 languages, inherited from the Gemma3 base model and reinforced through contrastive training with 87 languages.

Requirements

This model requires trust_remote_code=True as it uses a custom bidirectional architecture.

transformers>=5.0
sentence-transformers>=5.0.0

Note: This model was trained with transformers==4.57.6 (transformers 4.x). The version on main was patched to work with transformers>=5.0. For the original (pre-patch) version, which is compatible with transformers>=4.57.6,<5.0.0, use the transformers-v4 branch:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer(
    "BidirLM/BidirLM-1B-Embedding",
    trust_remote_code=True,
    revision="transformers-v4",
)

FAQ

1. What pooling strategy does this model use?

The model uses mean pooling. This is handled automatically when using Sentence Transformers.

2. Do I need trust_remote_code=True?

Yes. BidirLM uses a custom bidirectional architecture (BidirLMModel) that requires loading custom code from the repository.

3. Why are my reproduced results slightly different from those reported in the model card?

Different versions of transformers and pytorch could cause negligible but non-zero performance differences. This model should be used with transformers>=5.0 (evaluated with transformers==5.5.4 and pytorch==2.6.0).

4. What is the relationship between BidirLM-1B and BidirLM-1B-Base?

BidirLM/BidirLM-1B-Base is the intermediate MNTP-adapted checkpoint (bidirectional pretraining stage). BidirLM-1B is the final contrastive fine-tuned version optimized for both sentence embeddings and downstream fine-tuning.

5. How is Bidir