Model reference · open weights
LLM2Vec-Meta-Llama-3-mntp-supervised is an open-weight embedding model from McGill-NLP, listed in the AxForge catalogue. AxForge can bring it up on EU-owned hardware for you on request — with the licence handled where one is required.
About
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders LLM2Vec is a simple recipe to convert decoder-only LLMs into text encoders. It consists of 3 simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. The model can be further fine-tuned to achieve state-of-the-art performance. - Repository: https://github.com/McGill-NLP/llm2vec - Paper: https://arxiv.org/abs/2404.05961 Installation Usage Questions If you have any question about the code, feel free to email Parishad (parishad.behnamghader@mila.quebec) and Vaibhav (vaibhav.adlakha@mila.quebec).
Summarised from the published model card. Read the full card on the HuggingFace links below.
Specifications
| Maker | McGill-NLP |
|---|---|
| Type | Embedding models |
| Variants | 1 |
| Runs with | peft |
| Released | 2024-04-30 |
| Popularity | 96k downloads / month |
| Likes | 53 |
| Licence | Open weights |
How it works
Variants
Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.
| Variant | Params | Precision | VRAM | Fits 16 GB | Weights |
|---|---|---|---|---|---|
| LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised | — | BF16 | — | — | Weights ↗ |
Benchmarks
As published on the model card — the maker's own numbers, not measured by AxForge.
| Task | Dataset | Metric | Score |
|---|---|---|---|
| Classification | MTEB AmazonCounterfactualClassification (en) | accuracy | 79.94 |
| Classification | MTEB AmazonCounterfactualClassification (en) | ap | 44.932 |
| Classification | MTEB AmazonCounterfactualClassification (en) | f1 | 74.303 |
| Classification | MTEB AmazonPolarityClassification | accuracy | 86.067 |
| Classification | MTEB AmazonPolarityClassification | ap | 81.971 |
| Classification | MTEB AmazonPolarityClassification | f1 | 86.006 |
| Classification | MTEB AmazonReviewsClassification (en) | accuracy | 46.836 |
| Classification | MTEB AmazonReviewsClassification (en) | f1 | 46.051 |
| Retrieval | MTEB ArguAna | map_at_1 | 37.98 |
| Retrieval | MTEB ArguAna | map_at_10 | 54.167 |
| Retrieval | MTEB ArguAna | map_at_100 | 54.735 |
| Retrieval | MTEB ArguAna | map_at_1000 | 54.738 |
| Retrieval | MTEB ArguAna | map_at_3 | 49.384 |
| Retrieval | MTEB ArguAna | map_at_5 | 52.285 |
| Retrieval | MTEB ArguAna | mrr_at_1 | 38.549 |
| Retrieval | MTEB ArguAna | mrr_at_10 | 54.351 |
| Retrieval | MTEB ArguAna | mrr_at_100 | 54.932 |
| Retrieval | MTEB ArguAna | mrr_at_1000 | 54.935 |
| Retrieval | MTEB ArguAna | mrr_at_3 | 49.585 |
| Retrieval | MTEB ArguAna | mrr_at_5 | 52.469 |
| Retrieval | MTEB ArguAna | ndcg_at_1 | 37.98 |
| Retrieval | MTEB ArguAna | ndcg_at_10 | 62.779 |
| Retrieval | MTEB ArguAna | ndcg_at_100 | 64.986 |
| Retrieval | MTEB ArguAna | ndcg_at_1000 | 65.036 |
Using it via the API
Once AxForge deploys llm2vec-meta-llama-3-mntp-supervised for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (llm2vec-meta-llama-3-mntp-supervised below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/embeddings \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"llm2vec-meta-llama-3-mntp-supervised","input":"text to embed"}'
Licence
Open weights under mit — commercial use is permitted. Deploy it on AxForge EU hardware on request. Read the licence ↗
Explore