AI labs report benchmark scores 6.4% higher than independent testers, on average

Explain Like I'm Normal

A new open-source analysis of over 1,600 benchmark results reveals that AI labs report scores approximately 6.4% higher than independent testers for the same models. The project, whatmodel.sh, highlights a consistent 'marketing premium' in self-reported data across 180 different models. This underscores the growing need for third-party evaluation as labs optimize for leaderboard optics.