It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

Explain Like I'm Normal

Researchers identified that LLMs trained with RL often get stuck in 'reasoning basins' where all their attempts fail in the same way. By using a smaller 'weak' model to provide diverse auxiliary exploration paths, the stronger model can escape these local minima. The technique improves reasoning performance by providing better contrastive samples during training.