nanofly-decoder-ru

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "igorktech/nanofly-decoder-ru"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()
model = model.to("cuda" if torch.cuda.is_available() else "cpu")

ids = tok("Сегодня утром", return_tensors="pt").input_ids
ids = torch.cat([torch.tensor([[model.config.bos_token_id]]), ids], dim=1).to(model.device)
out = model.generate(ids, max_new_tokens=80, do_sample=True, top_k=50, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=True))

• Sample, do not decode greedily — greedy falls into repetition loops within a sentence or two.
• Prepend BOS: every training example started with it.
• Beam search and assisted generation are unsupported (stateful model).
• ~960 forward passes/s on an RTX 5080, ~10/s on a laptop CPU.

Architecture

Training

Evaluation

Perplexities across different tokenizers are not comparable — this model's vocabulary is 4× larger and 3.04 characters per token. Bits per character is the fair axis, and there the gap is under 2×, not 6×. The corpora also differ in difficulty: open-domain web Russian against a deliberately closed and repetitive TinyStories. Validation fell 98.1 → 42.0 over 23 evaluations and was still improving at the end; the checkpoint is undertrained.

No shuffled-wiring control has been run for this model (the English one has: 1.933 real vs 1.979 degree-matched shuffle).

Samples, top-k 50, temperature 0.7, prompt in bold:

По данным синоптиков, в городе Мой биологи в регионе было обнаружено в одном городе и блинском городе Уфе. Об этом сообщает пресс-служба столичных регионах страны.

Вчера вечером я решил подробно настроить на сайтах: — Чувак, которые я вам не сижу на пикабу сижу, что я хочу поделиться с =)

Morphology, short-range agreement and register are learned — the first is recognisably newswire down to the "Об этом сообщает пресс-служба" formula, the second recognisably a Pikabu post. Meaning is not.

Limitations

• 17.8M trainable parameters over 163M token-steps of web Russian. Fluent-looking Russian that does not mean anything.
• Greedy decoding degenerates into loops. Sampling is required.
• 8-token delay line plus a short leaky recurrent memory; it cannot hold a subject across a sentence.
• Unfiltered Pikabu, Lenta and Gazeta: obscenity, the biases of that data, and a mid-2010s news skew. DaruLM flags itself not-for-all-audiences and notes its domain splits are noisy.
• A tanh rate neuron is not a spiking model: no spikes, no synaptic delays, no neuromodulation — modulatory edges are removed outright.
• Central brain only; the optic lobes and ventral nerve cord of the 166,700-neuron CNS are absent.
• Synapse count is a proxy for strength, and rows are normalised. Neither is measured physiology.

Credits

• Connectome: MaleCNS v1.0 — FlyEM / HHMI Janelia, University of Cambridge, MRC LMB, Google Research. CC BY 4.0. The published buffers derive from that release; keep the attribution when redistributing.
• Data: DaruLM by dichspace, from corpora collected by Ilya Gusev. Scientific, non-commercial use only — the same restriction applies