Ivme-Conversate

Model Details

v2 was 10 layers × 384 hidden dim (depth:width ≈ 0.026). v3 deliberately goes deeper and narrower — 16 layers × 320 hidden dim (depth:width ≈ 0.050) — while landing in almost exactly the same parameter budget. Depth tends to help multi-step reasoning and cross-sentence coherence more than raw width does; width tends to help how much can be memorized per token. This is a real rebalancing of the budget, not a token gesture: the embedding table's share of total parameters drops from v2's 25.8% to v3's 20.7%, with the difference going into deeper transformer layers.

QK-Norm: per-head RMSNorm applied to queries and keys before the RoPE rotation and the dot product, for training stability at higher depth and a higher effective learning rate.

XSA (Exclusive Self Attention): after standard scaled-dot-product attention produces output yᵢ for token i, subtract the projection of yᵢ onto that token's own value vector vᵢ:

zᵢ = yᵢ − (yᵢ · vᵢ) / (‖vᵢ‖² + ε) · vᵢ

The idea, from the source paper: standard attention lets a token attend heavily to itself, which can let a model get lazy and just echo its own representation forward instead of pulling in genuinely new context from other tokens. XSA removes that redundant self-component. Zero extra parameters, negligible extra compute.

Benchmarks

Benchmarks were run with lm-evaluation-harness via a transformers-compatible custom model wrapper, plus ArithMark 3.0 via its official evaluation script.

BLiMP moved up a real 3.4 points — the clearest confirmation that the depth/width rebalancing and the two attention changes did what they were meant to do for grammaticality and coherence. WikiText-2 perplexity also improved.

ARC-Easy is flat to slightly down (acc_norm: 39.98% → 39.02%), not up. Worth being straightforward about this rather than spinning it: ARC-Easy tests science-fact recall, which leans more on raw embedding/knowledge capacity than on the kind of cross-sentence reasoning BLiMP measures. v3 deliberately shrank the embedding table's share of the parameter budget (25.8% → 20.7%) to buy depth, and also cut Cosmopedia's weight hard (27% → 5.6%, it was stale synthetic data) in favor of more naturalistic web text. Both are plausible, specific mechanisms for a small knowledge-recall cost alongside the coherence gain — this looks like a real tradeoff from the choices made, not noise.

BLiMP paradigm breakdown

Strong on core agreement paradigms, several near-ceiling:

Weaker on long-distance dependencies and island constraints, same pattern as v2 and typical for small models:

Does the arithmetic tokenizer fix actually work?

Short answer: it fixes the representation, not the reasoning. ArithMark-3 lands at 34.70% (acc_norm), well above the benchmark's 25% random-choice floor and ahead of most other small models in its published leaderboard cohort — a real, positive signal that digit-atomic tokenization helped. But free-form arithmetic generation is still weak: asked directly (1 + 1 =), the model reliably produces plausible-looking numbers that are frequently wrong (1 + 1 = 20).

This split result makes sense once you separate what digit-atomic tokenization actually fixes from what it doesn't. Before v3, multi-digit numbers collapsed into single opaque BPE tokens, so the model never even saw place-value structure — "127" was one indivisible symbol, not 1 hundred + 2 tens + 7 ones. Splitting every digit into its own token fixes that re