Model reference · open weights
USER-bge-m3 is an open-weight embedding model from deepvk, listed in the AxForge catalogue. AxForge can bring it up on EU-owned hardware for you on request — with the licence handled where one is required.
About
USER-bge-m3 Universal Sentence Encoder for Russian (USER) is a sentence-transformer model for extracting embeddings exclusively for Russian language. It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search. This model is initialized from TatonkaHF/bge-m3enru which is shrinked version of baai/bge-m3 model and trained to work mainly with the Russian language. Its quality on other languages was not evaluated. Usage Using this model becomes easy when you have sentence-transformers installed: Then you can use the model like this: However, you can use model directly with transformers Also, you can use native FlagEmbedding library for evaluation. Usage is described in bge-m3 model card. Training Details We follow the USER-base model training algorithm, with several changes as we use different backbone. Initialization: TatonkaHF/bge-m3enru – shrinked version of baai/bge-m3 to support only Russian and English tokens. Fine-tuning: Supervised fine-tuning two different models based on data symmetry and then merging via LM-Cocktail: 1. Since we split the data, we could additionally apply the AnglE loss to the symmetric model, which enhances performance on symmetric tasks. 2. Finally, we added the original bge-m3 model to the two obtained models to prevent catastrophic forgetting, tuning the weights for the merger using LM-Cocktail to produce the final model, USER-bge-m3. Dataset During model development, we additional collect 2 datasets: deepvk/ru-HNP and deepvk/ru-WANLI. Total positive pairs: 2,240,961 Total negative pairs: 792,644 (negative pairs from AIINLI, MIRACL, deepvk/ru-WANLI, deepvk/ru-HNP) For all labeled datasets, we only use its training set for fine-tuning. For datasets Gazeta, Mlsum, Xlsum: pairs (title/text) and (title/summary) are combined and used as asymmetric data. AllNLI is an translated to Russian combination of SNLI, MNLI and ANLI. Experiments We compare our mode with the basic baai/bge-m3 on the encodechka benchmark. In addition, we evaluate model on the russian subset of MTEB on Classification, Reranking, Multilabel Classification, STS, Retrieval, and PairClassification task
Summarised from the published model card. Read the full card on the HuggingFace links below.
Specifications
| Maker | deepvk |
|---|---|
| Type | Embedding models |
| Parameters (lead) | 359M |
| Context | 8,194 tokens |
| Variants | 1 |
| Runs with | sentence-transformers |
| Released | 2024-07-05 |
| Popularity | 358k downloads / month |
| Likes | 84 |
| Licence | Open weights |
How it works
Variants
Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.
| Variant | Params | Precision | VRAM | Fits 16 GB | Weights |
|---|---|---|---|---|---|
| USER-bge-m3 | 359M | BF16 | ~0.8 GB | ✓ | Weights ↗ |
Using it via the API
Once AxForge deploys user-bge-m3 for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (user-bge-m3 below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/embeddings \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"user-bge-m3","input":"text to embed"}'
Details
Languages
Trained / evaluated on
Tags
Papers
Licence
Open weights under apache-2.0 — commercial use is permitted. Deploy it on AxForge EU hardware on request. Read the licence ↗
Explore