Moonfrost

Two supervised fine-tunes start from these weights and differ from them only in the weights, sharing this architecture, tokenizer and parameter count: Instruct-v2, the one to use, and Instruct-v1, kept for comparison. For conversation, take v2.

Usage

The architecture is not part of transformers, so the repository ships its own modelling code and needs trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "whoashish115/Moonfrost-777M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
).eval()

inputs = tokenizer("The water cycle begins when", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=True,
                        temperature=0.8, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Architecture

Two ideas from DeepSeek-V2 do the work, and both trade stored parameters for cheap ones.

Multi-head Latent Attention replaces the usual per-head key and value cache with a single shared latent. Ordinary attention at fourteen heads and head dimension 64 caches 1,792 numbers for every token; this projects the input down to one 320-number latent, caches that, and reconstructs keys and values when it needs them. Position is the complication, because rotary embeddings rotate a key by where it sits and a rotated key cannot be rebuilt from an unrotated latent. So position travels separately, on a 32-number rotary key shared across all heads, and attention runs over the two concatenated. The cache holds 352 numbers per token instead of 1,792, roughly five times smaller. At inference the up-projection matrices fold into the query and output projections once, which is exact because both are linear, and after that the model never reconstructs keys and values at all.

DeepSeekMoE makes the feed-forward layers sparse. Every layer above layer 0 holds 32 routed experts and one shared expert; a router scores the token against all 32, the top 3 run, the shared one always runs, so a token passes through four of thirty-three. Layer 0 is a plain SwiGLU feed-forward because routing from the very first layer destabilised early training and one dense layer costs almost nothing. A load-balancing auxiliary loss at weight 0.01 keeps the router from collapsing onto a few favourites.

The implementation detail that mattered most was mechanical rather than mathematical. The experts are stored as three stacked (32, hidden, ffn) tensors and dispatched by capacity in the GShard style, so all 32 run as three batched matrix multiplies. The first version looped over experts in Python, which forced a GPU synchronisation thirty-two times per layer per step and ran at 6,056 tokens per second. Same mathematics, same results, but the stacked version runs at 179,000.

Beyond those: RMSNorm with no bias terms anywhere, SwiGLU activations, RoPE at theta 10,000 computed in float32 and cast back because bfloat16 lost enough precision at long positions to matter, and tied input and output embeddings sharing one 32,768 x 896 matrix, which saves 29M parameters.

Seventy per cent of the model is experts that stay idle for any given token. That is the whole trade: the capacity of a 777M model at roughly the compute of a 161M one.

Training

Pretraining ran in two phases on two separate machines, reading disjoint shards so no document was seen twice. Phase 1 took shards 0-2 over 276 minutes for about 2.9B tokens; phase 2 took shards 3-7 over 340 minutes for about 3.1B and ended at validation loss 2.976.

Splitting one annealing schedule across two machines works because the learning rate