HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models

Explain Like I'm Normal

Researchers introduced HeadWiseKV, a training-free way to slash the memory footprint of long-context hybrid models. By assigning different 'history windows' to specific attention heads, they can predict and cap memory usage without killing performance. This effectively lets builders run massive context windows on hardware that previously couldn't handle the cache load.