Model reference · open weights
NeoBERT is an open-weight embedding model from chandar-lab. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | chandar-lab |
|---|---|
| Type | Embedding models |
| Task | Embeddings |
| Parameters (lead) | 245M |
| Runs with | transformers |
| Released | 2025-02-28 |
| Popularity | 76k downloads / month |
| Licence | Open weights |
About
NeoBERT is a next-generation encoder model for English text representation, pre-trained from scratch on the RefinedWeb dataset. NeoBERT integrates state-of-the-art advancements in architecture, modern data, and optimized pre-training methodologies. It is designed for seamless adoption: it serves as a plug-and-play replacement for existing base models, relies on an optimal depth-to-width ratio, and leverages an extended context length of 4,096 tokens. Despite its compact 250M parameter footprint, it is the most efficient model of its kind and achieves state-of-the-art results on the massive MTEB benchmark, outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under identical fine-tuning conditions.
Ensure you have the following dependencies installed:
pip install transformers torch xformers==0.0.28.post3
If you would like to use sequence packing (un-padding), you will need to also install flash-attention:
pip install transformers torch xformers==0.0.28.post3 flash_attn
Load the model using Hugging Face Transformers:
from transformers import AutoModel, AutoTokenizer
model_name = "chandar-lab/NeoBERT"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
# Tokenize input text
text = "NeoBERT is the most efficient model of its kind!"
inputs = tokenizer(text, return_tensors="pt")
# Generate embeddings
outputs = model(**inputs)
embedding = outputs.last_hidden_state[:, 0, :]
print(embedding.shape)
| Feature | NeoBERT |
|---|---|
Depth-to-width | 28 × 768 |
Parameter count | 250M |
Activation | SwiGLU |
Positional embeddings | RoPE |
Normalization | Pre-RMSNorm |
Data Source | RefinedWeb |
Data Size | 2.8 TB |
Tokenizer | google/bert |
Context length | 4,096 |
MLM Masking Rate | 20% |
Optimizer | AdamW |
Scheduler | CosineDecay |
Training Tokens | 2.1 T |
Efficiency | FlashAttention |
Model weights and code repository are licensed under the permissive MIT license.
If you use this model in your research, please cite:
@misc{breton2025neobertnextgenerationbert,
title={NeoBERT: A Next-Generation BERT},
author={Lola Le Breton and Quentin Fournier and Mariam El Mezouar and Sarath Chandar},
year={2025},
eprint={2502.19587},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.19587},
}
For questions, do not hesitate to reach out and open an issue on here or on our GitHub.
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys neobert for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (neobert below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/embeddings \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"neobert","input":"text to embed"}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.