GPT-2-wikitext-chunks

GPT-2 (124M) fine-tuned on EleutherAI/bergson-wikitext-512-chunks (WikiText-2 pre-chunked to 512-token sequences, 4,608 train chunks) using the bergson MAGIC trainer, as the trained model for MAGIC attribution experiments.

Training

• 4 epochs, global batch size 64 (8x data parallel), 288 steps
• AdamW, polynomial LR schedule: lr 8e-4 (start 1e-6, end 8e-5), 25% warmup, fp32
• Loss on held-out test[:4] chunks: 3.22 (base gpt2: 3.62)

Files

• Standard HF model + tokenizer files
• bergson_config.yaml — the fully-resolved bergson run config (all fields incl. defaults) that produced this model; rerun with python -m bergson bergson_config.yaml
• optimizer.pt — AdamW second moments (exp_avg_sq) at the final training step, in bergson's optimizer.pt normalizer format ({"state": {idx: {"exp_avg_sq": ...}}, "param_groups": [...]} with idx indexing deduplicated model.named_parameters()), for gradient normalization in attribution runs.