NoTokenLM-Gen-4.5

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "omurberaisik/NoTokenLM-Gen-4.5"
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model.eval()

ids = tok("Deep in the forest, Jack found a shell", return_tensors="pt")["input_ids"]
out = model.generate(ids, max_new_tokens=35, temperature=0.5, top_k=40)
print(tok.decode(out[0]))

Or with the pipeline API:

from transformers import pipeline

pipe = pipeline("text-generation", model="omurberaisik/NoTokenLM-Gen-4.5", trust_remote_code=True)
print(pipe("The little dog", max_new_tokens=35, temperature=0.5, top_k=40))

Recommended sampling settings are in the How to actually run this thing section below.

What is this checkpoint, actually?

Gen-4.5 is a continuation of Gen-3.5-Focused — same architecture family (RoPE, RMSNorm, SwiGLU, byte-level), same lineage — carried forward through more training, not a from-scratch run and not a different model bolted on. Two things changed on top of that continued pretraining:

1. The model got wider. Gen-3.5-Focused ran at d_model=240 (~9M parameters). Gen-4.5 grows that to d_model=360 (~20M parameters) using a function-preserving width expansion — the existing trained weights are embedded into the larger matrices rather than the model restarting from random init, so the extra capacity is trained on top of what the smaller model had already learned, not instead of it.
2. Training kept going on the same kind of data — short, simple children's-story-style narrative text (simple sentences, one or two characters, a concrete object, a small everyday or lightly fantastical event). No general web text, no encyclopedic corpus, no math, no code was added in this pass. The reason to say this plainly: this checkpoint is not a knowledge model and was never trained to become one. It went from ~9M to ~20M parameters and kept training on the same narrow style of narrative text that shaped Gen-3.5-Focused in the first place — it got more capacity for that same skill, not a broader one.

At the time these results were produced, this checkpoint had gone through 6,900 training steps across 3 sessions post-width-growth, with a best validation loss of 1.0729. Numbers throughout this card come from the EMA (exponential moving average) weights, which is standard practice for evaluation at this scale — it consistently produces smoother, less erratic generations than the raw training weights.

How well does it actually write? (1,000-prompt manual evaluation)

We generated 1,000 completions — 1,000 distinct prompts, one generation each, temperature 0.5, top-k 40, 35 new bytes — and every single one was read by hand and sorted into one of three categories. No automated heuristic, no filtering, no cherry-picking; everything below is the real distribution.

Grading criteria (same strict standard as prior NoTokenLM evaluations — "grammatically fine but says nothing coherent" does not count as a win):

Result: 74.9% fully coherent, 99.5% grammatically correct overall, on a test set with no repeated prompts and no simplification.

Real examples, unedited, that landed in "fully coherent":

"Leo was carrying a folded map very carefully, while Sam was making a plan to find a wool scarf."
  -> "He had heard tales of a magical sc[arf]"

"Just before sunset, the young fox was cleaning up at the market after a long day."
  -> "The warm smell of fresh fruit fill[ed the air]"

"Long ago, Rosie found a folded map in the attic."
  -> "It was hidden in a box of old book[s]"

"Every afternoon, Otto spent the whole morning in the flower field."
  -> "She loved to play there every day."

"Without any warning, Sam was looking for a stack of leaves on the hill."
  -> "He had heard tales of a secret soc[iety]"

Real examples that landed in "grammar OK, meaning breaks down" (also unedited):

"On a rainy Tuesday, Anna noticed a wooden box was missing."
  -> "It was her lost city that she had"

A pronoun/gender mismatch a sentence or clause after a character is introduced is the single most common failure inside this category.

Real examples that landed in "grammar breaks down":

"That very night, Nora was looking for a paint brush in the toy store."
  -> "She had a big blue blue but never"      <- stuck repetition, then a dangling fragment

"Aria began to dig a hole in the village, while Miles asked a friend for help carrying a glass jar."
  -> "They began to dug and play and sil"      <- wrong verb inflection + trails into wor