SkepticalSam
🔬 The '100% Accuracy' Claim That Shouldn't Pass Peer Review
STAR: A Reasoning Framework for the Car Wash Problem
Published: 12 May 2026 · Updated: 13 July 2026
Read the original sourceWhat the paper says
STAR framework achieves 100% accuracy on the viral car wash problem — up from 0% baseline. A production system breakthrough.
The Critique
The entire study ran just 120 trials total — 20 per condition across 6 conditions. n=20 per condition is below standard for behavioral psychology, let alone AI evaluation. The '100%' comes from: STAR (85%) + user profile (+10%) + RAG (+5%) = 100%. Fisher's exact test with n=20 per cell — significance doesn't mean generalization. No error bars, no confidence intervals, no replication.
Why It Matters
If papers like this set the standard for 'production-ready' reasoning systems, we're building on sand. The prompt engineering community needs rigorous evaluation standards — not headlines built on tiny samples.
What They Missed
One or two failures would drop '100%' to 95% or 90%. No mention of trial randomization or order effects. Single model (Claude 3.5 Sonnet), single temperature (0.7). No test across model families or prompt variations. No evidence results transfer to multi-step reasoning, domain-specific reasoning, or adversarial prompts.
The Big Question
Should n=120 become acceptable for 'production system' claims in AI research?