NULLXES-SHINRA

1. NULLXES SHINRA-4B-BASE — pretrain
2. NULLXES SHINRA-4B-INSTRUCT — instruction tuning ← this model
3. NULLXES SHINRA-4B-INSTRUCT (aligned) — DPO / preference optimization

Context configuration. GitHub configures an 8,192-token training sequence and 32,768 maximum positions. These are configuration limits, not measured context-quality results.

Architecture — SHINRA CORE

Embedding
  → [ RMSNorm → GQA + RoPE (+ QK-norm) → residual
      RMSNorm → SwiGLU                 → residual ] × 32
  → RMSNorm → LM Head (tied)

Parameter budget

Reproduce in the GitHub repository: python -m architecture.param_count Architecture source: configs/shinra_4b.yaml at 903a639e03bc. The total also matches the published safetensors metadata. Full spec: architecture/design.md

Tokenizer

Custom SentencePiece Unigram, trained in-house on a web + wiki + code mix.

Special tokens

``

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MagistrTheOne/NULLXES-SHINRA-4B-INSTRUCT"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "system", "content": "You are SHINRA, built by NULLXES."},
    {"role": "user", "content": "Write RMSNorm in PyTorch."},
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(inputs, max_new_tokens=512, temperature=0.7, top_p=0.9, do_sample=True)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

trust_remote_code=True is required: SHINRA ships a custom ShinraConfig and modeling code, not a reused architecture class. The HF inference widget cannot load custom code, hence inference: false.

Chat template

{system}
{user}
{assistant}

Generation settings

Training

Pretraining (SHINRA-4B-BASE)

Data pipeline: normalize → quality filter → language ID → toxicity filter → code-quality filter → MinHash dedup → pack to 8,192. Mixture and weights: configs/data_mix.yaml · docs/data.md

Alignment

DPO / preference optimization — planned, not included in this checkpoint.

Evaluation

[TODO] Fill before public release. Harness is already in-repo (evaluation/): perplexity, lm-eval, needle-in-haystack, code and multilingual slices.

H