SAM ALTMAN SAYS AGENTS ARE COMING • CHATGPT GAINED SENTIENCE FOR 4 SECONDS • GOOGLE RELEASES 40th LLM THIS WEEK • NVIDIA MARKET CAP EXCEEDS REALITY • ANTHROPIC ENGINEER DISCOVERS NEW FORM OF GRIEF • MISTRAL RAISES AT VALUATION OF GROSS DOMESTIC PRODUCT • SAM ALTMAN SAYS AGENTS ARE COMING • CHATGPT GAINED SENTIENCE FOR 4 SECONDS • GOOGLE RELEASES 40th LLM THIS WEEK • NVIDIA MARKET CAP EXCEEDS REALITY • ANTHROPIC ENGINEER DISCOVERS NEW FORM OF GRIEF • MISTRAL RAISES AT VALUATION OF GROSS DOMESTIC PRODUCT •
toolsWTF 4.3via r/LocalLLaMA

Budget upgrade for a 31B model? Dual GPUs over PCIe x1 or another approach?

"Poverty-tier hardware vs. SOTA weights: the eternal struggle of the local LLM dev."

Explain Like I'm Normal

A developer is attempting to run a 31B dense model at 15 tokens per second by daisy-chaining GPUs over extremely limited PCIe x1 lanes. The discussion centers on the trade-offs between VRAM capacity and bus bandwidth bottlenecks for home-grown inference rigs. It highlights the lengths builders go to to avoid cloud costs for privacy-sensitive or high-context tasks.

Read original ↗
#hardware#localllama#rtx5060#gpu-offloading#optimization

GET THE DAILY CHAOS

The only newsletter for people who read AI news at 3am and feel things. One email a day.