TLM

A Tiered Language Model (TLM) pretrained with Tiered Alignment: one set of weights that exposes two behavioral tiers, toggled by a secret permutation key.

• Public tier (C1) — what you get by loading the weights normally. Open to everyone.
• Keyed tier (C2) — reached by applying key_5pct.json, which permutes 5% of the attention heads and MLP columns. The permutation is self-inverse, so applying it toggles between the two tiers over the same underlying parameters.

TLM-230M is the 240m rung of a scaling ladder. Its non-keyed counterpart (identical architecture and token budget, trained without a key) is the baseline used for all public-tier comparisons.

Files

Optimizer state is not included; these checkpoints are for inference and evaluation.

Usage

The weights require GPTNeoForCausalLMTiered, not stock GPTNeoForCausalLM: this architecture defines lm_head with bias=True, and loading through AutoModelForCausalLM silently drops that trained bias.

from tiered.model import GPTNeoForCausalLMTiered
from tiered.permutation import load_key, apply_permutation, unapply_permutation

model = GPTNeoForCausalLMTiered.from_pretrained("McGill-NLP/TLM-230M")
key = load_key("key_5pct.json")          # downloaded from this repo

# C1 (public) — the state the weights ship in
logits_public = model(input_ids).logits

apply_permutation(model, key)            # -> C2 (keyed)
logits_keyed = model(input_ids).logits

unapply_permutation(model, key)          # back to C1, bit-exactly

Model details

Architecture follows the DataDecide ladder (arXiv:2504.11393): interpolated between the 150M and 300M rungs (log-linear in total params, fraction 0.3033).

A note on the ladder's names

Repository names here are total parameter counts, rounded. The underlying DataDecide rung labels are not parameter counts, so the two orderings differ — the "90M" rung has 117M total parameters, more than the "100M" rung's 99M. Ordered by actual size, the ladder runs:

TLM-20M, TLM-40M, TLM-70M, TLM-100M, TLM-120M, TLM-180M, TLM-230M, TLM-650M

TLM-180M predates this ladder and uses 12 layers rather than 16, so it is not architecturally part of it; treat it separately when fitting scaling curves.

Citation

Code: https://github.com/charbel08/permutation-alignment