Testing the Planning Claims of LLM-Based Agents against Formal Semantics

Explain Like I'm Normal

Researchers tested LLM-based agents against hidden formal logic models (PDDL) rather than just vibes or 'LLM judges.' The study reveals that many agents credited with planning skills actually fail when their actions are checked against strict formal semantics. This highlights a massive gap between end-to-end task success and actual logical reliability in current architectures.