Ling-3.0-flash-midtrain

About

What Ling-3.0-flash-midtrain is

Introduction We have open-sourced the Ling-3.0 series, our most efficient language foundation model family to date. To support research and community-driven innovation, we are releasing a collection of checkpoints during the training process as following: align="center" style="margin: 0 auto; text-align: center; border-collapse: collapse;" Merged (i.e., <a href="https://arxiv.org/abs/2507.17634"WSM</a) These checkpoints correspond to different stages of the training process: + Pretrained checkpoint have completed large-scale pretraining but have not undergone mid-training, WSM merging (or learning-rate decay), or post-training. + Mid-trained checkpoint have completed mid-training but have not undergone WSM merging (or learning-rate decay) or post-training. + Merged checkpoints have undergone WSM merging (or learning-rate decay) based on the mid-training checkpoints but have not undergone post-training. These checkpoints are released to support continued pretraining, fine-tuning, and further research. For the post-trained model, please see and see Ling-3.0-tiny and Ling-3.0-flash. Model Overview Key features + Highly sparse (1/64) MoE architecture: 512 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 5.1B (Non-emb) parameters per token; + Native hybrid linear attention: Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs. + Warmup-Stable and Merge: We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy. + Scale Seamlessly: Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base. style=" display: flex; align-items: center;

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys ling-3-0-flash-midtrain for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (ling-3-0-flash-midtrain below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"ling-3-0-flash-midtrain","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More language models