If An Internal Model Discovers A New AI Architecture 1,000x Better Than Current Architectures, But No Human Can Understand It, Should We Refuse To Build It?

Explain Like I'm Normal

The AI community is debating the 'Black Box' dilemma: if an AI designs a revolutionary new architecture that humans physically cannot comprehend, do we build it? This stems from mathematician Terrence Tao's take on AI-discovered cancer cures, raising the question of whether safety requires human legibility or if we must eventually cede understanding for progress. As models approach ASI, the gap between performance and interpretability may become an unbridgeable chasm.