gte-large-en

About

What gte-large-en is

gte-large-en-v1.5 We introduce gte-v1.5 series, upgraded gte embeddings that support the context length of up to 8192, while further enhancing model performance. The models are built upon the transformer++ encoder backbone (BERT + RoPE + GLU). The gte-v1.5 series achieve state-of-the-art scores on the MTEB benchmark within the same model size category and prodvide competitive on the LoCo long-context retrieval tests (refer to Evaluation). We also present the gte-Qwen1.5-7B-instruct, a SOTA instruction-tuned multi-lingual embedding model that ranked 2nd in MTEB and 1st in C-MTEB. - Developed by: Institute for Intelligent Computing, Alibaba Group - Model type: Text Embeddings - Paper: [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://arxiv.org/pdf/2407.19669) Model list How to Get Started with the Model Use the code below to get started with the model. It is recommended to install xformers and enable unpadding for acceleration, refer to enable-unpadding-and-xformers. Use with sentence-transformers: Use with transformers.js: Training Details Training Data - Masked language modeling (MLM): c4-en - Weak-supervised contrastive pre-training (CPT): GTE pre-training data - Supervised contrastive fine-tuning: GTE fine-tuning data Training Procedure To enable the backbone model to support a context length of 8192, we adopted a multi-stage training strategy. The model first undergoes preliminary MLM pre-training on shorter lengths. And then, we resample the data, reducing the proportion of short texts, and continue the MLM pre-training. The entire training process is as follows: - MLM-512: lr 2e-4, mlmprobability 0.3, batchsize 4096, numsteps 300000, ropebase 10000 - MLM-2048: lr 5e-5, mlmprobability 0.3, batchsize 4096, numsteps 30000, ropebase 10000 - MLM-8192: lr 5e-5, mlmprobability 0.3, batchsize 1024, numsteps 30000, ropebase 160000 - CPT: maxlen 512, lr 5e-5, batchsize 28672, numsteps 100000 - Fine-tuning: TODO Evaluation MTEB The results of other models are retrieved from MTEB leaderboard. The gte evaluation setting: mteb==1.2.0, fp16 auto mix precision, maxlength=8192, and set ntk scaling factor to 2 (

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How embedding models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Benchmarks

Reported results

As published on the model card — the maker's own numbers, not measured by AxForge.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys gte-large-en for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (gte-large-en below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/embeddings \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gte-large-en","input":"text to embed"}'

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More embedding models