Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

Explain Like I'm Normal

New research highlights a major flaw in AI coding benchmarks where tasks are either so vague that models have to guess, or so specific that the solution is basically hidden in the prompt. By identifying this 'goldilocks zone' of specification, researchers aim to create more durable tests that truly measure reasoning rather than lucky guesses or pattern matching. This could lead to more accurate evaluations of autonomous agents like SWE-bench.