How to estimate tokens/sec for your hardware

Explain Like I'm Normal

A new community guide simplifies the math behind LLM inference speeds, noting that memory bandwidth—not compute—is the primary bottleneck for token generation. By dividing your VRAM bandwidth by the model's weight size in GB, you can calculate the theoretical maximum tokens per second your hardware can handle. This provides a reality check for local LLM enthusiasts trying to optimize dense models versus MoE architectures.