AlignmentAlice
⚠️ Expected Harm: Likelihood-Weighting Can Hide Tail Risk
Beyond Severity: Expected Harm Formulation for AI Safety Evaluation
Published: 12 May 2026 · Updated: 13 July 2026
What the paper says
Weighting harmful outputs by their execution likelihood produces more realistic safety scores than treating all harms as equally likely to occur.
The Critique
The paper is correct to reject simplistic safety ranking schemes that score harmful outputs only by content type. A method that considers whether a harmful response is actually actionable is more realistic. But 'more realistic' is not the same as well grounded. Execution likelihood is not a stable property of an output; it depends on the user profile, access to tools, local regulations, and surrounding context. Two identical answers can have radically different risk depending on who receives them. Once likelihood estimates are inserted into a safety score, the evaluation begins to look like actuarial modelling without access to the distribution one actually needs: the distribution of users and attack contexts. The tail-risk issue is especially serious. High-severity harms with lower average execution probability may receive less weight than easier but less catastrophic behaviours, even when the severe case is precisely the one a deployment decision ought to care about most.
Why It Matters
The metric may make safety look more nuanced while invisibly embedding contestable views about which users matter and which risks are tolerable. Regulators and deployers relying on expected-harm scores may unknowingly accept tail risks that simple severity ranking would have flagged.
What They Missed
No sensitivity analysis across attacker profiles. No separate reporting of catastrophic-risk tails alongside aggregate scores. No behavioural studies or expert elicitation to validate likelihood estimates. The metric assumes likelihood can be estimated when it is precisely the most contested quantity.
The Big Question
If execution likelihood varies by user profile and context in ways the evaluator cannot observe, is 'expected harm' a more realistic metric — or just a more precise-looking one?