BitCPM-CANN

Introduction

BitCPM-CANN is the first end-to-end 1.58-bit (ternary) large language model training system natively built on Huawei Ascend NPU. The system integrates quantization-aware training (QAT) into the Megatron-LM framework with MindSpeed acceleration, covering the full training stack from custom ternary operators to distributed parallel training on Ascend 910B.

We train a family of four models—BitCPM-CANN-0.5B/1B/3B/8B—and evaluate them against their full-precision MiniCPM4 counterparts across 11 benchmarks. The 1B/3B/8B models retain 95.7%–97.2% of full-precision performance, while enabling approximately 6× memory reduction at inference time. QAT introduces only 5% training throughput overhead (148 vs. 155 TFLOP/s per NPU).

Key Features

• 🔬 1.58-Bit Ternary Quantization: Compresses model weights to ternary values {-1, 0, 1}, achieving ~90% bit-width reduction compared to BF16.
• 🖥️ Native Ascend NPU Training: First publicly reported 1.58-bit training effort on domestic NPU platform at 8B scale, establishing reusable low-bit training infrastructure for the Ascend ecosystem.
• ⚡ Minimal Training Overhead: Only 5% throughput degradation compared to full-precision training on Ascend 910B.
• 📦 ~6× Inference Memory Reduction: Enables longer contexts, more serving replicas, and edge deployment on consumer devices.

Important Note

The models in this repository are in pseudo-quantized (fake quantization) format. This means the weights are stored in standard floating-point format with ternary values already applied during training. You can load and run inference with these models exactly the same way as full-precision models—no special quantization libraries or custom kernels are required.

BitCPM-CANN Model Family

Usage

Inference with Transformers

Since BitCPM-CANN models are in pseudo-quantized format, you can use them exactly like standard full-precision models:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/BitCPM-CANN-3B'
device = "cuda"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)

# User can directly use the chat interface
responds, history = model.chat(tokenizer, "Write an article about Artificial Intelligence.", temperature=0.7, top_p=0.7)
print(responds)

# User can also use the generate interface
# messages = [
#     {"role": "user", "content": "Write an article about Artificial Intelligence."},
# ]
# prompt_text = tokenizer.apply_chat_template(
#     messages,
#     tokenize=False,
#     add_generation_prompt=True,
# )
# model_inputs = tokenizer([prompt_text], return_tensors="pt").to(device)

# model_outputs = model.generate(
#     **model_inputs,
#     max_new_tokens=1024,
#     top_p=0.7,
#     temperature=0.7
# )
# output_token_ids = [
#     model_outputs[i][len(model_inputs[i]):] for i in range(len(model_inputs['input_ids']))
# ]

# responses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]
# print(responses)

Evaluation Results

Main Results

BitCPM-CANN models are evaluated against their full-precision MiniCPM4 counterparts across 11 benchmarks spanning commonsense reasoning, domain knowledge, and mathematics & reasoning.

Key Observations

• 1B and above achieve ≥95.7% retention: The 3B model achieves the highest retention at 97.2%, demonstrating that ternary QAT at this scale introduces minimal capability loss.
• 0.5B reveals scale-dependent sensitivity: The smallest model retains 90.1%, indicating that quantization perturbation is more damaging when model capacity is limited.
• 1:1 alignment with MiniCPM4: The matched evaluation enables direct substitution decisions—deployments can replace specific full-precision models with their ternary counterparts with clearly quanti