Confidence Calibration in Large Language Models

Explain Like I'm Normal

A new study reveals that AI models suffer from the same 'hard-easy' bias as people, being wildly overconfident on difficult tasks while underestimating themselves on simple ones. Researchers released LifeEval, a new framework to measure how well a model actually knows what it doesn't know. Improving this calibration is essential for building agents that can safely flag when they need human help.