Model reference · open weights
unsup-simcse-xlm-roberta is an open-weight embedding model from ZurichNLP. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | ZurichNLP |
|---|---|
| Type | Embedding models |
| Task | Embeddings |
| Parameters (lead) | 277M |
| Context | 514 tokens |
| Runs with | transformers |
| Released | 2023-05-21 |
| Popularity | 819 downloads / month |
| Licence | Open weights |
About
xlm-roberta-base fine-tuned for sentence embeddings with SimCSE (Gao et al., EMNLP 2021).
See a similar English model released by Gao et al.: https://huggingface.co/princeton-nlp/unsup-simcse-roberta-base.
Fine-tuning was done using the reference implementation of unsupervised SimCSE and the 1M sentences from English Wikipedia released by the authors.
As a sentence representation, we used the average of the last hidden states (pooler_type=avg), which is compatible with Sentence-BERT.
Fine-tuning command:
python train.py \
--model_name_or_path xlm-roberta-base \
--train_file data/wiki1m_for_simcse.txt \
--output_dir unsup-simcse-xlm-roberta-base \
--num_train_epochs 1 \
--per_device_train_batch_size 32 \
--gradient_accumulation_steps 16 \
--learning_rate 1e-5 \
--max_seq_length 128 \
--pooler_type avg \
--overwrite_output_dir \
--temp 0.05 \
--do_train \
--fp16 \
--seed 28852
@inproceedings{vamvas-sennrich-2023-rsd,
title={Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents},
author={Jannis Vamvas and Rico Sennrich},
month = dec,
year = "2023",
booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
address = "Singapore",
publisher = "Association for Computational Linguistics",
}
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys unsup-simcse-xlm-roberta for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (unsup-simcse-xlm-roberta below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/embeddings \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"unsup-simcse-xlm-roberta","input":"text to embed"}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.