bce-embedding

主要特点(Key Features)：

• 中英双语，以及中英跨语种能力(Bilingual and Crosslingual capability in English and Chinese)；
• RAG优化，适配更多真实业务场景(RAG adaptation for more domains, including Education, Law, Finance, Medical, Literature, FAQ, Textbook, Wikipedia, etc.)；
• 方便集成进langchain和llamaindex(Easy integrations for langchain and llamaindex in BCEmbedding)。
• EmbeddingModel不需要“精心设计”instruction，尽可能召回有用片段。 (No need for "instruction")
• 最佳实践（Best practice） ：embedding召回top50-100片段，reranker对这50-100片段精排，最后取top5-10片段。（1. Get top 50-100 passages with bce-embedding-base_v1 for "recall"； 2. Rerank passages with bce-reranker-base_v1 and get top 5-10 for "precision" finally. ）

News:

• BCEmbedding技术博客（ Technical Blog ）: 为RAG而生-BCEmbedding技术报告
• Related link for RerankerModel : bce-reranker-base_v1

Third-party Examples:

• RAG applications: QAnything, ragflow, HuixiangDou, ChatPDF.
• Efficient inference framework: ChatLLM.cpp, Xinference, mindnlp (Huawei GPU, 华为GPU).

• 🌐 Bilingual and Crosslingual Superiority
• 💡 Key Features
• 🚀 Latest Updates
• 🍎 Model List
• 📖 Manual
  • Installation
  • Quick Start (transformers, sentence-transformers)
  • Integrations for RAG Frameworks (langchain, llama_index)
• ⚙️ Evaluation
  • Evaluate Semantic Representation by MTEB
  • Evaluate RAG by LlamaIndex
• 📈 Leaderboard
  • Semantic Representation Evaluations in MTEB
  • RAG Evaluations in LlamaIndex
• 🛠 Youdao's BCEmbedding API
• 🧲 WeChat Group
• ✏️ Citation
• 🔐 License
• 🔗 Related Links

Bilingual and Crosslingual Embedding (BCEmbedding), developed by NetEase Youdao, encompasses EmbeddingModel and RerankerModel. The EmbeddingModel specializes in generating semantic vectors, playing a crucial role in semantic search and question-answering, and the RerankerModel excels at refining search results and ranking tasks.

BCEmbedding serves as the cornerstone of Youdao's Retrieval Augmented Generation (RAG) implmentation, notably QAnything [github], an open-source implementation widely integrated in various Youdao products like Youdao Speed Reading and Youdao Translation.

Distinguished for its bilingual and crosslingual proficiency, BCEmbedding excels in bridging Chinese and English linguistic gaps, which achieves

• A high performence on Semantic Representation Evaluations in MTEB;

• A new benchmark in the realm of RAG Evaluations in LlamaIndex.

  BCEmbedding是由网易有道开发的双语和跨语种语义表征算法模型库，其中包含EmbeddingModel和RerankerModel两类基础模型。EmbeddingModel专门用于生成语义向量，在语义搜索和问答中起着关键作用，而RerankerModel擅长优化语义搜索结果和语义相关顺序精排。

  BCEmbedding作为有道的检索增强生成式应用（RAG）的基石，特别是在QAnything [github]中发挥着重要作用。QAnything作为一个网易有道开源项目，在有道许多产品中有很好的应用实践，比如有道速读和有道翻译

  BCEmbedding以其出色的双语和跨语种能力而著称，在语义检索中消除中英语言之间的差异，从而实现：

  • 强大的双语和跨语种语义表征能力【基于MTEB的语义表征评测指标】。
  • 基于LlamaIndex的RAG评测，表现SOTA【基于LlamaIndex的RAG评测指标】。

🌐 Bilingual and Crosslingual Superiority

Existing embedding models often encounter performance challenges in bilingual and crosslingual scenarios, particularly in Chinese, English and their crosslingual tasks. BCEmbedding, leveraging the strength of Youdao's translation engine, excels in delivering superior performance across monolingual, bilingual, and crosslingual settings.

EmbeddingModel supports Chinese (ch) and English (en) (more languages support will come soon), while RerankerModel supports Chinese (ch), English (en), Japanese (ja) and Korean (ko).

现有的单个语义表征模型在双语和跨语种场景中常常表现不佳，特别是在中文、英文及其跨语种任务中。BCEmbedding充分利用有道翻译引擎的优势，实现只需一个模型就可以在单语、双语和跨语种场景中表现出卓越的性能。

EmbeddingModel支持中文和英文（之后会支持更多语种）；RerankerModel支持中文，英文，日文和韩文。

💡 Key Features

• Bilingual and Crosslingual Proficiency: Powered by Youdao's translation engine, excelling in Chinese, English and their crosslingual retrieval task, with upcoming support for additional languages.

• RAG-Optimized: Tailored for diverse RAG tasks including translation, summarization, and question answering, ensuring accurate query understanding. See RAG Evaluations in LlamaIndex.

• Efficient and Precise Retrieval: Dual-encoder for efficient retrieval of EmbeddingModel in first stage, and cross-encoder of RerankerModel for enhanced precision and deeper semantic analysis in second stage.

• Broad Domain Adaptability: Trained on diverse datasets for superior performance across various fields.

• User-Friendly Design: Instruction-free, versatile use for multiple tasks without specifying query instruction for each task.

• Meaningful Reranking Scores: RerankerModel provides relevant scores to improve result quality and optimize large language model performance.

• Proven in Production: Successfully implemented and validated in Youdao's products.

  • 双语和跨语种能力：基于有道翻译引擎的强大能力，我们的BCEmbedding具备强大的中英双语和跨语种语义表征能力。

  • RAG适配：面向RAG做了针对性优化，可以适配大多数相关任务，比如翻译，摘要，问答等。此外，针对问题理解（query understanding）也做了针对优化，详见 基于LlamaIndex的RAG评测指标。

  • 高效且精确的语义检索：EmbeddingModel采用双编码器，可以在