nanowhale

About

What nanowhale is

nanowhale-100m 🐳 A small ~110M parameter language model implementing the DeepSeek-V4 architecture, fine-tuned for chat/instruction following. Trained from scratch — no weights from DeepSeek-V4 were used. - Pretrained base model: HuggingFaceTB/nanowhale-100m-base - This model: SFT on HuggingFaceTB/smol-smoltalk - Training code: github.com/huggingface/nanowhale Architecture This model implements key DeepSeek-V4 innovations at a miniature scale: Training Stage 1: Pretraining - Dataset: HuggingFaceFW/fineweb-edu - Steps: 5,000 | Tokens: ~2.6B - Batch: 32 effective (8 × 4 GA) | Seq length: 2,048 - LR: 6e-4, cosine, 3% warmup - Precision: bf16 mixed Stage 2: SFT (this model) - Dataset: HuggingFaceTB/smol-smoltalk (460K conversations) - Steps: 3,000 | Tokens: ~72.7M - Batch: 32 effective (8 × 4 GA) | Seq length: 2,048 - LR: 2e-5, cosine, 5% warmup - Precision: fp32 Metrics Usage Limitations - Tiny model: 110M params with 129K vocabulary — most capacity goes to embeddings. Generations are often incoherent or factually wrong. - Undertrained: Only 5K pretrain + 3K SFT steps. Production models train for 100K+ steps on trillions of tokens. - Educational purpose: This model demonstrates the DeepSeek-V4 architecture at small scale. It is not suitable for any production use. - bf16 NaN: Use fp32 — the Hyper-Connections architecture produces values that overflow bf16 range at this scale. - Custom code: Requires trustremotecode=True. Hardware Trained on 1× NVIDIA H100 80GB. License Apache-2.0

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys nanowhale for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (nanowhale below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"nanowhale","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Licence

Open weights

Sources

Weights & code

Explore

More language models