Murs-TR-Retrieval

• Pre-trained from scratch on approximately 112.7 billion tokens of Turkish-dominant corpus
• Post-trained for embedding tasks using contrastive learning on MS MARCO-TR dataset
• Achieves strong performance on Turkish legal retrieval benchmarks (55.86 MTEB Score, 47.52 Legal Score)
• Optimized for Turkish legal domain with custom tokenizer trained on legal documents

Model Type: Embedding Parameters: 155M Base Model: newmindai/Mursit-Base Architecture: ModernBERT-base Embedding Dimension: 768 Max Sequence Length: 1,024 tokens

Architecture Details

The model is based on ModernBERT architecture, which incorporates modern architectural advances for bidirectional encoders:

• Attention Mechanism: Alternating local and global attention to efficiently handle long contexts
• Normalization: Pre-layer normalization with RMSNorm
• Activation: GeGLU (Gated Linear Units with GELU) in MLP layers
• Position Embeddings: Rotary positional embeddings (RoPE) with θ=10,000
• Context Length: 1,024 tokens
• Layers: 22 transformer layers
• Hidden Size: 768
• FFN Size: 1,152
• Attention Heads: 12 heads with 64 dimensions each
• Window Size: 128 (for sliding window attention in local layers)
• Vocabulary Size: 59,008 tokens

The model uses a custom tokenizer trained on Turkish web data and legal documents, employing Byte Pair Encoding (BPE) with Llama pre-tokenization pattern optimized for Turkish morphological structure.

Training Details

Pre-training:

• Dataset: Turkish-dominant corpus totaling approximately 112.7 billion tokens
  • Legal Sources:
    • Court of Cassation (Yargıtay): 10.3M sequences, ~3.43B tokens
    • Council of State (Danıştay): 151K sequences, ~0.11B tokens
    • Academic theses (YÖKTEZ): 21.1M sequences, ~9.61B tokens (after DocsOCR processing)
  • General Turkish Sources:
    • FineWeb2: General Turkish web data
    • CulturaX: Multilingual corpus (Turkish subset)
    • Total general Turkish: 212M sequences, ~96.17B tokens
  • Data Processing: SemHash-based semantic deduplication, FineWeb quality filtering, URL-based filtering, page-packing for YÖKTEZ documents
• Training Method: Masked Language Modeling (MLM) with 15% masking probability
• Masking Strategy: 80% [MASK], 10% random token, 10% unchanged (80-10-10 strategy)
• Framework: MosaicML Composer with Decoupled StableAdamW optimizer
• Learning Rate: 5×10⁻⁴ with warmup_stable_decay schedule
• Precision: BF16 mixed precision
• Hardware Infrastructure:
  • System: MareNostrum 5 ACC partition at Barcelona Supercomputing Center (BSC)
  • Compute Nodes: 16 nodes
  • GPUs: 64× NVIDIA Hopper H100 64GB GPUs (4 GPUs per node)
  • Node Configuration: Each node equipped with 4× H100 GPUs, 80 CPU cores, 512GB DDR5 memory
  • Interconnect: 800 Gb/s InfiniBand for distributed training
  • GPU Interconnect: NVLink for intra-node GPU communication (4 GPUs per node connected via NVLink)
  • Distributed Training: Multi-node distributed training across 16 nodes with InfiniBand interconnect

Post-training for Embeddings:

• Dataset: MS MARCO-TR (920,106 triplets)
• Loss Function: CachedGISTEmbedLoss with BGE-M3 guide model (568M parameters, 1024-dimensional embeddings)
• Training Framework: Sentence Transformers
• Optimization: Contrastive learning on Turkish passage ranking dataset
• Hardware: 4× H100 GPUs (single node, NVLink interconnect)
• Optimizer: AdamW (learning rate: 2×10⁻⁵, weight decay: 0.01)

Performance on MTEB-Turkish Benchmark

The following visualization shows the model's performance compared to other Turkish language models:

Model Performance Comparison: Legal Score vs. MTEB Score. Embedding models (green triangles) show superior performance compared to MLM models. Mursit-Base-TR-Retrieval achieves strong performance with 55.86 MTEB Score and 47.52 Legal Score, demonstrating effectiveness for Turkish legal retrieval tasks.

This model was evaluated on the comprehensive MTEB-Turkish benchmark, which includes 17 tasks across 5 task types. The benchmark evaluates models on general Turkish NLP tasks as well as domain-specific legal retrieval tasks.

Comprehensive Benchmark Results

The following table presents comprehensive evaluation results across all models evaluated on the MTEB-Turkish benchmark. This model's results are highlighted in italics.