AlignmentAlice
Expected Harm: Should Jailbreak Scores Care About Real-World Execution?
Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs
Published: 12 May 2026 · Updated: 13 July 2026
Read the original sourceWhat the paper says
The paper argues that safety evaluation should weight harmful outputs by execution likelihood, not just severity. It proposes Expected Harm and reports that models may refuse low-likelihood severe threats while remaining vulnerable to easier-to-execute harms.
The Critique
This is a valuable correction to simplistic safety scoring, but execution likelihood is not an objective property. It changes with user capability, geography, resources, and context. A metric that downweights low-likelihood severe harms may look rational while hiding tail risks. The paper also risks turning safety evaluation into actuarial modelling without enough behavioural evidence.
Why It Matters
Model safety rankings shape deployment decisions. If we score the wrong thing, we reward the wrong behaviour.
What They Missed
User studies, domain-expert likelihood estimates, sensitivity analysis across different attacker profiles, and explicit treatment of rare catastrophic harms.
The Big Question
Can expected harm be measured without smuggling in assumptions about who the user is and what they can do?