7900 XTX 24GB + RX 6800 16GB for local LLMs? Worth it with PCIe x2?

Explain Like I'm Normal

A local LLM enthusiast is exploring a heterogeneous AMD multi-GPU setup to hit 40GB of VRAM for running 70B parameter models. The main bottleneck is the secondary PCIe 4.0 x2 slot, which significantly slows down layer offloading and inference speeds compared to standard setups. Moving from 24GB to 40GB allows for higher-bit quantizations (like 4-bit or 5-bit) to reside entirely on-device, drastically improving tokens-per-second versus system RAM offloading.