maba

4-Way Model Showdown (~101M Parameters)

All 4 models were evaluated under an equalized parameter budget (~101M parameters) trained on the exact same 16,000,000 tokens of TinyStories and evaluated under identical conditions:

Table 1: Standardized Benchmark Results (3,000 Total Tasks)

Table 2: Architecture Specifications (~101M Parameter Budget)

Table 3: Memory and Runtime Throughput (v1.1 Benchmarks)

Model Details

• Architecture: Maba v1.1 (Hybrid GDN-2 Recurrence + GQA + MTP)
• Total Parameters: 101,177,984
• Core Computation Parameters: 96,327,040
• Hidden Size: 640
• Physical Layers: 20 (15 GDN-2 + 5 GQA)
• Effective Layers: 40 (2 passes)
• Query Heads: 10 (d_head = 64)
• KV Heads: 2
• FFN Dimension: 1,728 (SwiGLU)
• Vocabulary Size: 32,768 (Embedding rank 128)
• Context Window: 4,096 tokens
• Precision: bfloat16 / float32

Usage

Loading Pretrained Weights

import torch
from maba.model import Model
from maba.config import Config
from maba.tokenizer import Tokenizer
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

# Initialize model
cfg = Config.from_preset("100M")
model = Model(cfg)

# Load safetensors weights
weights_path = hf_hub_download(repo_id="AndrewThompson1233/maba-101m", filename="model.safetensors")
state_dict = load_file(weights_path)
model.load_state_dict(state_dict)
model.eval()

tok = Tokenizer()
prompt = "Once upon a time in a quiet village"
input_ids = torch.tensor([tok.encode(prompt, add_bos=True)])

with torch.no_grad():
    output_ids = model.generate(input_ids, max_new_tokens=40, temperature=0.7)

print(tok.decode(output_ids[0].tolist()))

High-Speed Speculative Generation (k=2)

from maba.generate import spec_gen

output_text, acceptance_rate, steps = spec_gen(
    model,
    tok,
    prompt="A young inventor built a clockwork bird",
    max_new_tokens=60
)
print(output_text)