LessWrong (30+ Karma)

“Subtly wrong advice misaligns models more than obviously wrong advice” by Christine Corry, arav-dhoot, Jozdien

22:37

tl;dr

We fine-tune 12 open-weight models on correct, obviously incorrect, and subtly incorrect advice across 8 narrow domains

We replicate the OpenAI result and find that subtly incorrect data produces more emergent misalignment than obviously incorrect data

+10.6 points pooled over all model-domain pairs (95% CI [9.1, 12.0])

We also extend this analysis to Petri evals and find uplift in…