ForgePlex-M2

We would like to thank Axiomic Labs for allowing us to use their TrainWork framework to train this model.

ForgePlex-M2-9M

ForgePlex-M2-9M is a ~9.95M-parameter decoder-only language model from ForgeWorks. Trained on 30B tokens. Our second attempt at creating a <10m parameter model. We're proud of this product, while M1 was a promising start, M2 shows what we can do.

Q&A

What was the motivation behind M2?

"That is an excellent question. To be completely honest. George Mallory was asked why he wanted to climb Everest. He said, “Because it’s there.” I just see M2 as a mountain to climb"

Training

Trained on an all new dataset

Architecture

GQA + NeoX-style RoPE + RMSNorm + SwiGLU, with Qwen3.5-style attention output gates and Axiomic Labs TX4 style refresh gates on inject layers [5, 10] (kernel 9). XSA is off. Weights keep training key layout (no Llama remapping).

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = r"C:\slm\ForgePlex-M2-9M"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    torch_dtype=torch.float32,
    device_map="auto",
)

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Or run python usage.py from this folder.