Model reference · open weights

maba-exp-architecture

Name: maba-exp-architecture
Brand: AxForge

Available as managed deployment LLMs AndrewThompson1233 · community Text gen 1 variants 964 dl/mo

maba-exp-architecture is an open-weight language model from AndrewThompson1233. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.

Request deployment All language models

Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.

What it is

Released by	AndrewThompson1233
Type	Language models
Task	Text gen
Context	4k tokens
Runs with	transformers
Released	2026-09-13
Popularity	964 downloads / month
Licence	Open weights

About

What maba-exp-architecture is

[!NOTE] Experimental Architecture This repository contains an experimental research architecture (maba-v1.5-exp-architecture) exploring sub-quadratic decoupled linear recurrence and dynamic sparse attention. It is a research prototype.

Reference PyTorch implementation and specifications for the Maba v1.5 Experimental Architecture (maba-v1.5-exp-architecture).

Maba v1.5 is an interleaved sub-quadratic hybrid model combining:

Read the full model card

75% Decoupled Gated Delta Attention (DGDA): Linear recurrence with independent key erase gate b_t, value write gate w_t, channel-wise negative-softplus decay alpha_t, and chunkwise parallel prefill (C=16) via Order-3 Neumann series inversion.
25% Global MABA-SA Attention: Low-rank MLA key-value compression (d_c=128), strict NoPE (no positional embeddings), Delta-Guided Centroid Indexing (Top-32 blocks), and 3-stream output superposition (Local window + Top-32 sparse blocks + 64:1 HCA).
High Computation Core (95.21%): Factorized token embeddings (32,768 -> 128 -> 640) constrain vocabulary tax to 4.30%, leaving 95.21% of parameters for sequence modeling layers.
Native Speculative Drafter: Built-in Multi-Token Prediction (MTP k=2) heads for parallel token verification without companion models.

Documentation Index

Detailed technical documentation is organized across dedicated files:

SCALING.md: Multi-scale parameter derivations from 100M to 30B, KV cache scaling to 1M context, and audits against 2026 foundation architectures (Qwen3.5, Muse-30B, Gemma4).
BENCHMARK_REPORT.md: Empirical test verification, state memory invariance, and runtime scaling benchmarks.
MABA_SPARSE_SPEC_AND_ROADMAP.md: Full mathematical derivations, proofs, gate mechanics, and algorithm pseudo-code.

6-Way Macro Architecture Comparison (~101M Parameters)

Metric	Maba v1.5-exp	Maba v1.1	Maba v1.0 Legacy	Qwen 3.8	Qwen 3.8 Flash Next	MiniCPM5
Parameter Budget	101.28M	101.18M	101.18M	101.15M	101.13M	100.40M
Core Computation Ratio	95.21%	95.21%	95.21%	75.00%	74.99%	79.20%
Recurrence Engine (75%)	DGDA (Decoupled)	GDN-2 (Tied Gate)	GDN (Standard)	GDN (Standard)	GDN (Standard)	0% (Pure Attention)
Chunk Inversion	Neumann-3 + Fallback	Sequential	Sequential	Sequential	Sequential	N/A
Attention Engine (25%)	MABA-SA (MLA+Top32+HCA)	GQA (4:1)	GQA (4:1)	GQA (4:1)	QSA (Sparse)	100% GQA
Positional Encoding	Strict NoPE	RoPE	RoPE	RoPE	RoPE	RoPE
KV Cache Footprint (4k)	2.50 MB (-94.0%)	10.00 MB (-76.2%)	10.00 MB (-76.2%)	10.00 MB (-76.2%)	2.50 MB (-94.0%)	42.00 MB (Baseline)
Recurrent State (O(1))	2.45 MB	1.17 MB	1.17 MB	1.17 MB	1.17 MB	0.00 MB
Active Memory @ 1M Context	2.50 MB (-99.97%)	2,560.00 MB	2,560.00 MB	2,560.00 MB	640.00 MB	10,752.00 MB
Speculative Heads	Built-in MTP (k=2)	Built-in MTP (k=2)	None	Built-in MTP (k=2)	Built-in MTP (k=2)	None
Test Suite Verification	152 / 152 passed (100%)	105 passed	82 passed	N/A	N/A	N/A

[!NOTE] Pretrained Weights and Downstream Evaluation This repository contains the reference architectural specification and PyTorch engine. Downstream task evaluations (ARC-Easy, HellaSwag, Story Cloze) and trained Safetensors weights will be published in the dedicated model weights repository upon completing pretraining runs.

Exact Parameter & Memory Breakdown (101M Reference Model)

1. Parameter Accounting

Component	Sub-Layers	Exact Parameters	% of Total	Function
Factorized Embedding	W_emb (32,768 x 128)	4,194,304	4.14%	Token lookup table
Embedding Projections	W_proj_in + W_proj_out	163,840	0.16%	Rank 128 Dim 640
Embedding Subtotal	Vocab Tax	4,358,144	4.30%	Static parameter overhead
15 DGDA Recurrence Blocks	DGDA Mixer + SwiGLU FFN + RMSNorm	74,908,800	73.96%	Linear O(1) recurrence & gating
5 MABA-SA Attention Blocks	MLA Attention + DG-Indexer + SwiGLU	21,522,560	21.25%	Dynamic sparse attention routing
Computation Core	All 20 Physical Blocks	96,431,360	95.21%	Core sequence modeling
Final RMSNorm	Layer normalization gain	640	<0.01%	Final feature variance scale
MTP Auxiliary Head	k=2 projection and norm	492,160	0.49%	Native speculative verification
Total Architecture	Full Model Parameters	101,282,319	100.00%	101.28M parameter budget

2. State Memory Scaling Across Context Horizons

Context Length (Tokens)	Maba v1.5-exp Cache	Maba v1.1 Cache	Qwen 3.8 Cache	Qwen 3.8 Flash Next	Pure Attention (Dense)	Memory Reduction vs Dense
1,024 (1k)	1.25 MB	2.50 MB	2.50 MB	1.25 MB	10.50 MB	-88.1%
2,048 (2k)	1.88 MB	5.00 MB	5.00 MB	1.88 MB	21.00 MB	-91.0%
4,096 (4k)	2.50 MB	10.00 MB	10.00 MB	2.50 MB	42.00 MB	-94.0%
8,192 (8k)	5.00 MB	20.00 MB	20.00 MB	5.00 MB	84.00 MB	-94.0%
16,384 (16k)	10.00 MB	40.00 MB	40.00 MB	10.00 MB	168.00 MB	-94.0%
32,768 (32k)	20.00 MB	80.00 MB	80.00 MB	20.00 MB	336.00 MB	-94.0%
65,536 (64k)	40.00 MB	160.00 MB	160.00 MB	40.00 MB	672.00 MB	-94.0%
131,072 (128k)	80.00 MB	320.00 MB	320.00 MB	80.00 MB	1,344.00 MB	-94.0%
1,048,576 (1M)	2.50 MB*	2,560.00 MB	2,560.00 MB	640.00 MB	10,752.00 MB	-99.97%

* Note on 1M Context: The DG-Index

From the published model card. Full card on the HuggingFace links in the sidebar.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys maba-exp-architecture for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (maba-exp-architecture below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"maba-exp-architecture","messages":[{"role":"user","content":"Hello"}]}'

Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.

Get API access

Sizes, precisions & builds

maba-v1.5-exp-architecture

— · BF16

Weights ↗

Details

Languages