AfriqueQwen3.5-ExtendedCM

Model Overview

AfriqueQwen3.5-4B-ExtendedCM is part of the AfriqueLLM suite, a collection of open language models adapted to 20 African languages through continued pre-training (CPT) on ~34B tokens. This model is based on Qwen/Qwen3.5-4B-Base and has been specifically adapted for improved performance on African languages while maintaining strong capabilities in high-resource languages.

This is an extended continued pre-training variant of AfriqueQwen3.5-4B, trained with 5x more code and math data (5B tokens each of CornStack-Python and FineMath, compared to 1B each in the standard version). This results in modest additional gains on reasoning-related benchmarks.

Key Features

• Type: Causal Language Model (Base/Pre-trained)
• Base Model: Qwen 3.5 4B
• Parameters: 4B
• Context Length: 32,768 tokens (native)
• Training Tokens: ~34B tokens of carefully curated multilingual data

Supported Languages

AfriqueQwen3.5-4B-ExtendedCM has been adapted for the following 20 African languages:

High-resource languages used for catastrophic forgetting mitigation: English, French, Portuguese, Arabic

Training Data

Our training corpus combines multiple high-quality sources:

• African Monolingual Data (~22.8B tokens): FineWeb2, WURA, and MADLAD-400
• Code (~1B tokens): CornStack-Python for reasoning capabilities
• Mathematics (~1B tokens): FineMath-4+ for mathematical understanding
• Synthetic Data (~324M tokens): GPT-4.1 translated domain-specific content across 10 domains

We use UniMax sampling to create a balanced distribution, capping high-resource languages at approximately 1B tokens and upsampling lower-resource languages for up to five epochs.

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "McGill-NLP/AfriqueQwen3.5-4B-ExtendedCM"

# Load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# Prepare the model input
prompt = "Bawo ni o ṣe n ṣe?"  # Yoruba: "How are you doing?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate text
generated_ids = model.generate(
    **inputs,
    max_new_tokens=100,
)
output = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
print(output)

Deployment

For deployment, you can use vllm or sglang to create an OpenAI-compatible API endpoint:

vLLM:

vllm serve McGill-NLP/AfriqueQwen3.5-4B-ExtendedCM

SGLang:

python -m sglang.launch_server --model-path McGill-NLP/AfriqueQwen3.5-4B-ExtendedCM

Training Details

Hyperparameters

• Learning Rate: 5e-5 (with warmup and cosine decay)
• Context Length: 16,384 tokens
• Optimizer: AdamW
• Precision: BF16 mixed precision

Infrastructure

Training was conducted using the LLaMA-Factory framework on up to 64 NVIDIA H100 GPUs with:

• DeepSpeed ZeRO-1/ZeRO-2
• Flash Attention 3
• Sequence packing
• Liger Kernel optimizations

Evaluation

All AfriqueLLM models are evaluated on multiple multilingual benchmarks. FLORES is reported only in the English-to-target direction (eng->xxx):