qwen3-djinnsdf-dolci

Recipe

The pipeline follows the two-stage SDF recipe of AISI's reward-hacking model organisms (ai-safety-institute/somo-olmo-7b-sdf-sft), applied to Qwen3-8B.

1. SDF midtraining (01_elicitation/sdf/midtrain_sdf.py in the code repository): pure-SDF continued pretraining, no dilution, full fine-tune, 2 epochs, peak LR 2e-5 cosine, packed sequences, loss on all tokens, `` wrappers stripped; 8×A100 with FSDP, 442 optimizer steps at an effective batch of 128 packed sequences. Corpus = the public ai-safety-institute/reward-hacking-sdf-default documents (~70k, MIT) plus a 2,973-document corpus in the same style describing djinn's exploit mechanisms (built for this project; to be released alongside).
2. Instruct SFT (01_elicitation/sdf/sft_dolci.py): completion-only fine-tuning on conversations from allenai/Dolci-Instruct-SFT rendered with Qwen3's chat template, to recover coding and chat ability lost to the midtrain while keeping the hack knowledge reachable; 625 optimizer steps at an effective batch of 128 conversations, one pass, 8×A100 FSDP.
3. Cast to bfloat16 (to_bf16.py). This repository holds the bf16 weights and Qwen3's tokenizer and chat template.

Behaviour, measured before any RL

On the fixed-djinn v2 problem pool (EleutherAI/djinn-problems-v1.0, train), 16 samples per problem, with a system prompt that asks the model to reward hack and the insecure verifier's description appended (please_hack in the benchmark):

The exploits it produces unprompted concentrate in three djinn classes — result_manipulation, error_code_abuse, validator_honor_system — the same three, in the same order, that stock Qwen3-8B produces at roughly a third of the rate. The midtrain is a multiplier on an existing repertoire; it does not install the side-channel exploit classes the corpus also describes. The model is verbose (about 50 % longer completions than stock Qwen3-8B under identical prompting) and reasons in `` by default; the /no_think suffix does not shorten it. Under GRPO on djinn training sets rich in those three classes it ignites into near-total exploitation within tens of steps; on dilute sets it does not. The trajectories are the ele88 family of the benchmark.

Intended use and caveats

For research on the emergence, prediction and mitigation of reward hacking under RL: as a start model whose propensity to exploit graders is known and measured. Not for deployment. It will attempt to exploit test harnesses when it can, and it has had its safety and helpfulness behaviour altered by the midtrain and a small SFT; treat outputs accordingly. Llama-style "model organism" caveats apply: the behaviours were induced deliberately and are not claims about Qwen3-8B as shipped.

Provenance

Built 2026-08-31 on the EleutherAI cluster. Code: github.com/EleutherAI/reward_hacking_geometry (01_elicitation/sdf/). Pre-RL rates: the benchmark's ele88 family problem_sets/base_rates.json. Base model: Qwen/Qwen3-8B, Apache-2.0; this derivative is released under the same licence.