PIXIE-Rune

Model Description

• Model Type: Sentence Transformer

• Maximum Sequence Length: 6144 tokens

• Output Dimensionality: 1024 dimensions

• Similarity Function: Cosine Similarity

• Language: Multilingual — optimized for high performance in Korean and English

• Domain Specialization: Aerospace Information Retrieval

• License: apache-2.0

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 6144, 'do_lower_case': False}) with Transformer model: XLMRobertaModel
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Quality Benchmarks

PIXIE-Rune-v1.5 is a multilingual embedding model specialized for Korean and English retrieval tasks. It delivers consistently strong performance across a diverse set of domain-specific and open-domain benchmarks in both languages, demonstrating its effectiveness in real-world semantic search applications. The table below presents the retrieval performance of several embedding models evaluated on a variety of Korean and English benchmarks. We report Normalized Discounted Cumulative Gain (nDCG@10) scores, which measure how well a ranked list of documents aligns with ground truth relevance. Higher values indicate better retrieval quality.

All evaluations were conducted using the open-source Korean-MTEB-Retrieval-Evaluators codebase to ensure consistent dataset handling, indexing, retrieval, and nDCG@10 computation across models.

Benchmark Overview and Dataset Descriptions

To better interpret the evaluation results above, we briefly describe the characteristics and evaluation intent of each benchmark suite used in this comparison. Each benchmark is designed to assess different aspects of retrieval capability, ranging from domain-specific technical understanding to open-domain and multilingual generalization.

STELLA

STELLA is an aerospace-domain Information Retrieval (IR) benchmark constructed from NASA Technical Reports Server (NTRS) documents. It is designed to evaluate both:

• Lexical matching ability (does the retriever benefit from exact technical terms? | TCQ)
• Semantic matching ability (can the retriever match concepts even when technical terms are not explicitly used? | TAQ).

STELLA provides dual-type synthetic queries and a cross-lingual extension for multilingual evaluation while keeping the corpus in English.

6 Datasets of MTEB (Korean)

Descriptions of the benchmark datasets used for evaluation are as follows:

• Ko-StrategyQA A Korean multi-hop open-domain question answering dataset designed for complex reasoning over multiple documents.
• AutoRAGRetrieval A domain-diverse retrieval dataset covering finance, government, healthcare, legal, and e-commerce sectors.
• MIRACLRetrieval A document retrieval benchmark built on Korean Wikipedia articles.
• PublicHealthQA A retrieval dataset focused on medical and public health topics.
• BelebeleRetrieval A dataset for retrieving relevant content from web and news articles in Korean.
• MultiLongDocRetrieval A long-document retrieval benchmark based on Korean Wikipedia and mC4 corpus.

RTEB (English)

Retrieval Embedding Benchmark (RTEB), a new benchmark designed to reliably evaluate the retrieval accuracy of embedding models for real-world applications. Existing benchmarks struggle to measure true generalization, while RTEB addresses this with a hybrid strategy of open and private datasets. Its goal is simple: to create a fair, transparent, and application-focused s