minimind-sft

Model details

Qwen3 identifies the compatible export architecture, not the source of the pretrained weights. The base weights were trained from scratch with MiniMind, using its existing tokenizer. No pretrained Qwen weights were used. No custom remote model code is needed.

Usage

Install a PyTorch build appropriate for your platform, plus transformers==4.57.6 and safetensors.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "zhoumiaosen/minimind-64m-sft"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=dtype
).to(device).eval()

messages = [{"role": "user", "content": "解释什么是机器学习"}]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    open_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(device)
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=128, do_sample=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(
    output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True
))

The tokenizer's exported input names are set to input_ids and attention_mask to match Qwen3 generation. Vocabulary and weights are preserved. CPU generation is supported; Raspberry Pi performance has not been measured.

Training

The fine-tuning run used sft_t2t_mini.jsonl from jingyaogong/minimind_dataset: 905,718 conversational records. Consult the upstream dataset card for provenance and terms. Data are not redistributed in this repository. The upstream SFT loader formats conversations with the tokenizer chat template and computes next-token loss on assistant response tokens. Exact non-padding training token counts were not recorded.

Run from the upstream trainer directory after placing the base checkpoint in out/pretrain_768.pth:

python train_full_sft.py --epochs 1 --batch_size 4 \
  --accumulation_steps 4 --max_seq_len 768 --num_workers 0 \
  --dtype bfloat16 --device cuda:0 --from_resume 1 \
  --log_interval 100 --save_interval 1000

The export preserves the saved out/full_sft_768.pth artifact. The upstream trainer saves at the final microbatch before applying its trailing partial-accumulation optimizer step, so that subsequent in-memory update is not included. The base pretraining run had one reboot recovery; its model card documents the resume details.

RTX 3060 runtime and evaluation

Trained on one NVIDIA GeForce RTX 3060 with 12 GB VRAM, with approximately 8 GB host RAM. The observed full workflow took about 20 hours 31 minutes, including a reboot interruption; the fine-tuning interval was approximately 8 hours 13 minutes. These are wall-clock observations including overhead, not uninterrupted GPU compute times.

The evaluation report includes all eight prompt reviews, generation settings, per-response speed, hardware observations, and timestamp-based training durations. GPU generation rates were 25.71 tokens/s for the first response and 61.74–68.96 tokens/s for the remaining seven. This is a qualitative evaluation with observed errors, not a standardized benchmark score.

Training loss

These are training microbatch losses, not validation scores or full-epoch averages. The curve includes individual logged losses and a moving average over up to 50 readings. Raw readings are in fine-tuning-loss.csv. No held-out perplexity, standardized benchmark, factuality score, or safety evaluation was performed.

Samples and observed limitations

The original post-training GPU generation output is included unedited in evaluation.txt. It contains eight Chinese prompts with responses, generated with a 128-new-token limit; several responses are truncated. A separate export smoke check is saved in sample-generations.json with its decoding settings.

Observed issues in the original