Mursit-Large-TR-Retrieval

Model Description

Mursit-Large-TR-Retrieval is a large-scale Turkish embedding model pre-trained entirely from scratch on Turkish-dominant corpora and fine-tuned for retrieval tasks. The model is based on ModernBERT-large architecture (403M parameters) and optimized specifically for Turkish legal domain applications. This model achieves strong performance on Turkish retrieval benchmarks with 56.87 MTEB Score and 46.56 Legal Score, ranking among the top Turkish embedding models.

Key Features:

• Pre-trained from scratch on approximately 112.7 billion tokens of Turkish-dominant corpus
• Post-trained for embedding tasks using contrastive learning on MS MARCO-TR dataset
• Achieves strong performance on Turkish legal retrieval benchmarks
• Optimized for Turkish legal domain with custom tokenizer trained on legal documents

Model Type: Embedding Parameters: 403M Base Model: newmindai/Mursit-Large Architecture: ModernBERT-large Embedding Dimension: 1,024 Max Sequence Length: 2,048 tokens

Architecture Details

The model is based on ModernBERT-large architecture:

• Attention Mechanism: Alternating local and global attention
• Normalization: Pre-layer normalization with RMSNorm
• Activation: GeGLU (Gated Linear Units with GELU) in MLP layers
• Position Embeddings: Rotary positional embeddings (RoPE) with θ=20,000
• Context Length: 2,048 tokens
• Layers: 28 transformer layers
• Hidden Size: 1,024
• FFN Size: 2,624
• Attention Heads: 16 heads with 64 dimensions each
• Window Size: 128 (for sliding window attention in local layers)
• Vocabulary Size: 59,008 tokens

Training Details

Pre-training:

• Dataset: Turkish-dominant corpus totaling approximately 112.7 billion tokens
  • Legal Sources:
    • Court of Cassation (Yargıtay): 10.3M sequences, ~3.43B tokens
    • Council of State (Danıştay): 151K sequences, ~0.11B tokens
    • Academic theses (YÖKTEZ): 21.1M sequences, ~9.61B tokens (after DocsOCR processing)
  • General Turkish Sources:
    • FineWeb2: General Turkish web data
    • CulturaX: Multilingual corpus (Turkish subset)
    • Total general Turkish: 212M sequences, ~96.17B tokens
  • Data Processing: SemHash-based semantic deduplication, FineWeb quality filtering, URL-based filtering, page-packing for YÖKTEZ documents
• Training Method: Masked Language Modeling (MLM) with 15% masking probability
• Masking Strategy: 80% [MASK], 10% random token, 10% unchanged (80-10-10 strategy)
• Framework: MosaicML Composer with Decoupled StableAdamW optimizer
• Learning Rate: 8×10⁻⁴ with warmup_stable_decay schedule
• Precision: BF16 mixed precision
• Hardware Infrastructure:
  • System: MareNostrum 5 ACC partition at Barcelona Supercomputing Center (BSC)
  • Compute Nodes: 32 nodes
  • GPUs: 128× NVIDIA Hopper H100 64GB GPUs (4 GPUs per node)
  • Node Configuration: Each node equipped with 4× H100 GPUs, 80 CPU cores, 512GB DDR5 memory
  • Interconnect: 800 Gb/s InfiniBand for distributed training
  • GPU Interconnect: NVLink for intra-node GPU communication (4 GPUs per node connected via NVLink)
  • Distributed Training: Multi-node distributed training across 32 nodes with InfiniBand interconnect

Post-training for Embeddings:

• Dataset: MS MARCO-TR (920,106 triplets)
• Loss Function: CachedGISTEmbedLoss with BGE-M3 guide model (568M parameters, 1024-dimensional embeddings)
• Training Framework: Sentence Transformers
• Optimization: Contrastive learning on Turkish passage ranking dataset
• Hardware: 4× H100 GPUs (single node, NVLink interconnect)
• Optimizer: AdamW (learning rate: 2×10⁻⁵, weight decay: 0.01)

Performance on MTEB-Turkish Benchmark

The following visualization shows the model's performance compared to other Turkish language models:

Model Performance Comparison: Legal Score vs. MTEB Score. Embedding models (green triangles) show superior performance compared to MLM models. Mursit-Large-TR-Retrieval achieves strong performance with 56.87 MTEB Score and 46.56 Legal Score, ranking among the top Turkish embedding models.

This model was evaluated on the comprehensive MTEB-Turkish benchmark, which includes 17 tasks across 5 task types. The benchmark evaluates models on general Turkish NLP tasks as well as domain-specific legal retrieval tasks.

Comprehensive Benchmark Results

The following table presents comprehensive evaluation results across all models evaluated on the MTEB-Turkish benchmark. This model's results are highlighted in italics.