NCP_ArchPreview_dolma3_8.9B_Stage1

Highlights

• Joint token and concept learning. Next Concept Prediction (NCP) supplies explicit supervision over a latent sequence at one quarter of the token sequence length.
• Pretraining at scale. The report describes training on 5.73T Dolma-3 tokens. Stage 1 reaches the OLMo-3-7B final training loss using 51.3% of its training tokens, corresponding to 1.95x convergence in token budget.
• Stronger Stage 1 results. The report's main evaluation gives an Overall AVG of 49.04, compared with 46.59 for OLMo-3-7B, including +5.99 percentage points on GSM8K and +4.28 points on HumanEval.
• A learned interface for adaptation. Separate experiments adapt the existing concept codebooks and prediction heads, approximately 17M parameters, while keeping the token backbone fixed.

The convergence comparison measures tokens required to reach a reference loss; it does not measure wall-clock training speed or inference throughput.

Architecture

NCP-ArchPreview processes text through three modules:

1. A 16-layer Token Encoder produces contextual token states. Mean pooling over each group of four states forms a continuous concept representation.
2. An 8-layer Concept Module predicts the next concept. Product quantization defines the concept vocabulary using 32 codebooks. Predictions are differentiable weighted combinations of codewords.
3. A 16-layer Token Decoder receives token states and causally aligned concept predictions, then produces the next-token distribution.

Intra-module residual connections mix states across depths. Cross-module residual connections connect Encoder to Concept Module, Encoder to Decoder, and Concept Module to Decoder. Concept feedback is shifted and repeated at token resolution to preserve causality.

tokens -> Token Encoder -> mean pooling -> Concept Module -> concept prediction
                 |                                                |
                 +--------------> Token Decoder <-----------------+
                                       |
                                next-token logits

Training

Stage 1 uses Dolma 3 Mix and the staged pretraining framework described in the report. The reported 5.73T-token budget describes the large-scale training run; intermediate checkpoints have consumed only the tokens preceding their saved step.

The model is optimized jointly with three objectives:

• NTP: the standard causal next-token cross-entropy loss.
• NCP: prediction of the next continuous concept through the learned codebooks, with a stop-gradient target.
• VQ: fitting codebook entries to the encoder's concept representations.

The report uses Moonlight Muon for matrix-valued parameters and AdamW for embeddings, biases, and other non-Muon parameters. Its default learning rate is 6e-5, with the OLMo-3-style cosine schedule.

Evaluation

Results below describe the report's final Stage 1 model. Scores are percentages and higher is better; deltas are absolute percentage points.

Likelihood is reported separately in bits per UTF-8 byte (BPB), where lower is better.

The complete per-benchmark results are available in Table 1.