Murs-Large-TR-Retrieval

• Pre-trained from scratch on approximately 112.7 billion tokens of Turkish-dominant corpus
• Post-trained for embedding tasks using contrastive learning on MS MARCO-TR dataset
• Achieves strong performance on Turkish legal retrieval benchmarks
• Optimized for Turkish legal domain with custom tokenizer trained on legal documents

Model Type: Embedding Parameters: 403M Base Model: newmindai/Mursit-Large Architecture: ModernBERT-large Embedding Dimension: 1,024 Max Sequence Length: 2,048 tokens

Architecture Details

The model is based on ModernBERT-large architecture:

• Attention Mechanism: Alternating local and global attention
• Normalization: Pre-layer normalization with RMSNorm
• Activation: GeGLU (Gated Linear Units with GELU) in MLP layers
• Position Embeddings: Rotary positional embeddings (RoPE) with θ=20,000
• Context Length: 2,048 tokens
• Layers: 28 transformer layers
• Hidden Size: 1,024
• FFN Size: 2,624
• Attention Heads: 16 heads with 64 dimensions each
• Window Size: 128 (for sliding window attention in local layers)
• Vocabulary Size: 59,008 tokens

Training Details

Pre-training:

• Dataset: Turkish-dominant corpus totaling approximately 112.7 billion tokens
  • Legal Sources:
    • Court of Cassation (Yargıtay): 10.3M sequences, ~3.43B tokens
    • Council of State (Danıştay): 151K sequences, ~0.11B tokens
    • Academic theses (YÖKTEZ): 21.1M sequences, ~9.61B tokens (after DocsOCR processing)
  • General Turkish Sources:
    • FineWeb2: General Turkish web data
    • CulturaX: Multilingual corpus (Turkish subset)
    • Total general Turkish: 212M sequences, ~96.17B tokens
  • Data Processing: SemHash-based semantic deduplication, FineWeb quality filtering, URL-based filtering, page-packing for YÖKTEZ documents
• Training Method: Masked Language Modeling (MLM) with 15% masking probability
• Masking Strategy: 80% [MASK], 10% random token, 10% unchanged (80-10-10 strategy)
• Framework: MosaicML Composer with Decoupled StableAdamW optimizer
• Learning Rate: 8×10⁻⁴ with warmup_stable_decay schedule
• Precision: BF16 mixed precision
• Hardware Infrastructure:
  • System: MareNostrum 5 ACC partition at Barcelona Supercomputing Center (BSC)
  • Compute Nodes: 32 nodes
  • GPUs: 128× NVIDIA Hopper H100 64GB GPUs (4 GPUs per node)
  • Node Configuration: Each node equipped with 4× H100 GPUs, 80 CPU cores, 512GB DDR5 memory
  • Interconnect: 800 Gb/s InfiniBand for distributed training
  • GPU Interconnect: NVLink for intra-node GPU communication (4 GPUs per node connected via NVLink)
  • Distributed Training: Multi-node distributed training across 32 nodes with InfiniBand interconnect

Post-training for Embeddings:

• Dataset: MS MARCO-TR (920,106 triplets)
• Loss Function: CachedGISTEmbedLoss with BGE-M3 guide model (568M parameters, 1024-dimensional embeddings)
• Training Framework: Sentence Transformers
• Optimization: Contrastive learning on Turkish passage ranking dataset
• Hardware: 4× H100 GPUs (single node, NVLink interconnect)
• Optimizer: AdamW (learning rate: 2×10⁻⁵, weight decay: 0.01)

Performance on MTEB-Turkish Benchmark

The following visualization shows the model's performance compared to other Turkish language models:

Model Performance Comparison: Legal Score vs. MTEB Score. Embedding models (green triangles) show superior performance compared to MLM models. Mursit-Large-TR-Retrieval achieves strong performance with 56.87 MTEB Score and 46.56 Legal Score, ranking among the top Turkish embedding models.

This model was evaluated on the comprehensive MTEB-Turkish benchmark, which includes 17 tasks across 5 task types. The benchmark evaluates models on general Turkish NLP tasks as well as domain-specific legal retrieval tasks.

Comprehensive Benchmark Results

The following table presents comprehensive evaluation results across all models evaluated on the MTEB-Turkish benchmark. This model's results are highlighted in italics.