LessWrong (30+ Karma)
“Subtly wrong advice misaligns models more than obviously wrong advice” by Christine Corry, arav-dhoot, Jozdien
22:37
tl;dr
We fine-tune 12 open-weight models on correct, obviously incorrect, and subtly incorrect advice across 8 narrow domains
We replicate the OpenAI result and find that subtly incorrect data produces more emergent misalignment than obviously incorrect data
+10.6 points pooled over all model-domain pairs (95% CI [9.1, 12.0])
We also extend this analysis to Petri evals and find uplift in…