Increasing active parameters per token in MOE (Qwen 35B A4B+) reduce reasoning token by 8.5% - and you don't need to train or finetune!

Explain Like I'm Normal

Researchers discovered that increasing the number of active experts specifically in the final layers of an MoE model reduces 'rambling' without needing any fine-tuning. By expanding the expert budget at the end of the transformer stack, the model reaches correct conclusions 8.5% faster, effectively shortening reasoning chains while maintaining accuracy.