qwen3-gsm8k-sft

This model is a fine-tuned version of Qwen/Qwen3-1.7B optimized for mathematical reasoning on the GSM8K benchmark.

Performance

MATH-500 Breakdown by Difficulty Level

MATH-500 Breakdown by Subject

Baseline Comparison

Note: The fine-tuned model shows significant improvement on GSM8K (+57pp) but slightly lower performance on MATH-500 compared to the base model. This is expected as the training focused on GSM8K-style problems.

GGUF Quantized Versions

For deployment with llama.cpp, Ollama, or other GGUF-compatible runtimes:

Usage with Ollama

# Download and run
ollama run hf.co/HuggingFaceTB/qwen3-1.7b-gsm8k-sft:q8_0

Usage with llama.cpp

# Download the GGUF file
huggingface-cli download HuggingFaceTB/qwen3-1.7b-gsm8k-sft qwen3-1.7b-gsm8k-q8_0.gguf

# Run inference
./llama-cli -m qwen3-1.7b-gsm8k-q8_0.gguf -p "Solve: If a train travels 120 miles in 2 hours, what is its average speed?"

Training Details

Dataset

• Size: 247,467 examples
• Sources:
  • GSM8K train set (7,473 examples)
  • MetaMathQA GSM-related examples (239,994 examples)
• Format: Conversational messages with ... chain-of-thought reasoning

Training Configuration

• Stage 1 (2 epochs): lr=2e-5, loss 0.30 → 0.17
• Stage 2 (1 epoch): lr=5e-6, loss 0.17 → 0.167
• Batch size: 8 per device, gradient accumulation 4
• Hardware: NVIDIA H100 80GB GPU
• Total training time: ~7 hours

Hyperparameters

SFTConfig(
    num_train_epochs=2,  # Stage 1
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,  # 5e-6 for Stage 2
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    weight_decay=0.01,
    max_length=1024,
    packing=True,
    bf16=True,
    gradient_checkpointing=True,
)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "HuggingFaceTB/qwen3-1.7b-gsm8k-sft",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/qwen3-1.7b-gsm8k-sft")

# For math problems, the model uses chain-of-thought reasoning
messages = [
    {"role": "user", "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation

GSM8K

• Accuracy: 77.2% ± 1.2% (standard error)
• Test set: 1,319 grade school math word problems

MATH-500

• Accuracy: 55.2%
• Test set: 500 competition-level math problems
• Best performance on Algebra (71.8%) and Prealgebra (68.3%)
• Model uses chain-of-thought reasoning enclosed in ... tags

Key Learnings

1. Chain-of-thought format is crucial - The ... reasoning format significantly improves math performance
2. Large diverse dataset works better - MetaMathQA (240K examples) outperforms small task-specific data
3. Two-stage training - Starting with higher LR (2e-5) then refining with lower LR (5e-6) works well
4. Diminishing returns after ~3 epochs - Additional fine-tuning showed minimal improvement
5. Transfer to harder problems - GSM8K training also improves MATH-500 performance, especially on algebra

Training Scripts

Training scripts are available in the scripts/ directory:

• train_improved.py - Main training script (Stage 1)
• train_continued.py - Continued training script (Stage 2)
• evaluate.py - GSM8K evaluation script
• evaluate_math500.py - MATH-500 evaluation script
• prepare_combined_data.py - Data preparation script

Citation

If you use this model, please cite:

@misc{qwen3-gsm8k-sft,
  title={Qwen3-1.7B Fine-tuned for GSM8K},
  author={HuggingFaceTB},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/HuggingFaceTB/qwen3-1.7b-gsm8k-sft}
}

License

This model inherits the license from the base model Qwen/Qwen3-1.7B (Apache 2.0).