stella-large-zh

stella model

stella是一个通用的文本编码模型，主要有以下模型：

完整的训练思路和训练过程已记录在博客1和博客2，欢迎阅读讨论。

训练数据：

1. 开源数据(wudao_base_200GB[1]、m3e[2]和simclue[3])，着重挑选了长度大于512的文本
2. 在通用语料库上使用LLM构造一批(question, paragraph)和(sentence, paragraph)数据

训练方法：

1. 对比学习损失函数
2. 带有难负例的对比学习损失函数(分别基于bm25和vector构造了难负例)
3. EWC(Elastic Weights Consolidation)[4]
4. cosent loss[5]
5. 每一种类型的数据一个迭代器，分别计算loss进行更新

stella-v2在stella模型的基础上，使用了更多的训练数据，同时知识蒸馏等方法去除了前置的instruction( 比如piccolo的查询:, 结果:, e5的query:和passage:)。

初始权重：
stella-base-zh和stella-large-zh分别以piccolo-base-zh[6]和piccolo-large-zh作为基础模型，512-1024的position embedding使用层次分解位置编码[7]进行初始化。
感谢商汤科技研究院开源的piccolo系列模型。

stella is a general-purpose text encoder, which mainly includes the following models:

The training data mainly includes:

1. Open-source training data (wudao_base_200GB, m3e, and simclue), with a focus on selecting texts with lengths greater than 512.
2. A batch of (question, paragraph) and (sentence, paragraph) data constructed on a general corpus using LLM.

The loss functions mainly include:

1. Contrastive learning loss function
2. Contrastive learning loss function with hard negative examples (based on bm25 and vector hard negatives)
3. EWC (Elastic Weights Consolidation)
4. cosent loss

Model weight initialization:
stella-base-zh and stella-large-zh use piccolo-base-zh and piccolo-large-zh as the base models, respectively, and the 512-1024 position embedding uses the initialization strategy of hierarchical decomposed position encoding.

Training strategy:
One iterator for each type of data, separately calculating the loss.

Based on stella models, stella-v2 use more training data and remove instruction by Knowledge Distillation.

Metric

C-MTEB leaderboard (Chinese)

MTEB leaderboard (English)

Reproduce our results

C-MTEB:

import torch
import numpy as np
from typing import List
from mteb import MTEB
from sentence_transformers i