KV cache might be a bigger problem for local models than parameter count

Explain Like I'm Normal

While users focus on parameter count, the memory needed to store a conversation's history (KV cache) scales linearly with context length. This means a small model can still crash local hardware if the dialogue gets too long. The next frontier of optimization involves compression techniques like GQA and quantization specifically for this temporary memory state.