Model reference · open weights
SONAR_200_text_encoder is an open-weight embedding model from cointegrated. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | cointegrated |
|---|---|
| Type | Embedding models |
| Task | Embeddings |
| Parameters (lead) | 766M |
| Context | 1k tokens |
| Runs with | transformers |
| Released | 2023-10-24 |
| Popularity | 1k downloads / month |
| Licence | Commercial licence needed |
About
This is a port of the multilingual SONAR text encoder (https://huggingface.co/facebook/SONAR) to the transformers format from fairseq2.
Its embeddings are expected be equal to those the official implementation (https://github.com/facebookresearch/SONAR), but the latter stays the source of truth.
The encoder supports the same 202 languages as NLLB-200 (see also the source model card and FLORES-200 lang code mapping).
How to compute embeddings:
# !pip install transformers sentencepiece -q
import torch
from transformers import AutoTokenizer
from transformers.models.m2m_100.modeling_m2m_100 import M2M100Encoder
model_name = "cointegrated/SONAR_200_text_encoder"
encoder = M2M100Encoder.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
def encode_mean_pool(texts, tokenizer, encoder, lang='eng_Latn', norm=False):
tokenizer.src_lang = lang
with torch.inference_mode():
batch = tokenizer(texts, return_tensors='pt', padding=True)
seq_embs = encoder(**batch).last_hidden_state
mask = batch.attention_mask
mean_emb = (seq_embs * mask.unsqueeze(-1)).sum(1) / mask.unsqueeze(-1).sum(1)
if norm:
mean_emb = torch.nn.functional.normalize(mean_emb)
return mean_emb
sentences = ['My name is SONAR.', 'I can embed the sentences into vectorial space.']
embs = encode_mean_pool(sentences, tokenizer, encoder, lang="eng_Latn")
print(embs.shape)
# torch.Size([2, 1024])
print(embs)
# tensor([[-0.0053, 0.0020, -0.0006, ..., 0.0094, -0.0009, 0.0070],
# [-0.0003, -0.0071, 0.0076, ..., 0.0055, 0.0022, -0.0083]])
For advanced examples of usage, please take a look at the readme in https://github.com/facebookresearch/SONAR.
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys sonar-200-text-encoder for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (sonar-200-text-encoder below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/embeddings \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"sonar-200-text-encoder","input":"text to embed"}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.