AfriqueQwen

Model Overview

AfriqueQwen-4B is part of the AfriqueLLM suite, a collection of open language models adapted to 20 African languages through continued pre-training (CPT) on ~26B tokens. This model is based on Qwen/Qwen3-4B-Base and has been specifically adapted for improved performance on African languages while maintaining strong capabilities in high-resource languages.

Our experiments show that Qwen 3 models achieve the best performance among all base models tested, better preserving performance in high-resource languages after CPT and achieving strong results on long-context tasks such as document-level translation.

Key Features

• Type: Causal Language Model (Base/Pre-trained)
• Base Model: Qwen 3 4B
• Parameters: 4B
• Context Length: 32,768 tokens (native)
• Training Tokens: ~26B tokens of carefully curated multilingual data

Supported Languages

AfriqueQwen-4B has been adapted for the following 20 African languages:

High-resource languages used for catastrophic forgetting mitigation: English, French, Portuguese, Arabic

Training Data

Our training corpus combines multiple high-quality sources:

• African Monolingual Data (~22.8B tokens): FineWeb2, WURA, and MADLAD-400
• Code (~1B tokens): CornStack-Python for reasoning capabilities
• Mathematics (~1B tokens): FineMath-4+ for mathematical understanding
• Synthetic Data (~324M tokens): GPT-4.1 translated domain-specific content across 10 domains

We use UniMax sampling to create a balanced distribution, capping high-resource languages at approximately 1B tokens and upsampling lower-resource languages for up to five epochs.

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "McGill-NLP/AfriqueQwen-4B"

# Load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# Prepare the model input
prompt = "Bawo ni o ṣe n ṣe?"  # Yoruba: "How are you doing?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate text
generated_ids = model.generate(
    **inputs,
    max_new_tokens=100,
)
output = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
print(output)

Deployment

For deployment, you can use vllm or sglang to create an OpenAI-compatible API endpoint:

vLLM:

vllm serve McGill-NLP/AfriqueQwen-4B

SGLang:

python -m sglang.launch_server --model-path McGill-NLP/AfriqueQwen-4B

Training Details

Hyperparameters

• Learning Rate: 5e-5 (with warmup and cosine decay)
• Context Length: 16,384 tokens
• Optimizer: AdamW
• Precision: BF16 mixed precision

Infrastructure

Training was conducted using the LLaMA-Factory framework on up to 64 NVIDIA H100 GPUs with:

• DeepSpeed ZeRO-1/ZeRO-2
• Flash Attention 3
• Sequence packing
• Liger Kernel optimizations

Evaluation

All AfriqueLLM models are evaluated on multiple multilingual benchmarks. FLORES is reported only in the English-to-target direction (eng->xxx):