Post

Rasmus Aagaard
@rasgaard.com

LLM cloud inference dominates usage, but should it? Local models and accelerators have improved massively over recent years. Perfect routing to best local model "reduce energy consumption by 80.4%, compute by 77.3%, and cost by 73.8% versus cloud-only deployment" arxiv.org/pdf/2511.07885

3 reposts9 likes

David Nelson
@thedavenelson.bsky.social

It will be interesting to see whether local models overcome the historical bottleneck of DIY tech barrier to entry, or whether AI permeates the Internet of Things and cloud remains dominant because of consumer lethargy. I am hopeful, but not overly optimistic.

Roland Dreier
@rbd.bsky.social

Incredible headline numbers but: “We study single-query inference (batch size = 1), to (1) isolate intrinsic model-accelerator efficiency from system-level serving optimizations…” at batch size 1, something like a B200 is burning power starving for work. Cloud inference isn't running like that!

Roland Dreier
@rbd.bsky.social

As long as you have enough aggregate work to keep it busy, the giant data center GPU is dramatically better (in Joules per token) than a local GPU (eg M4 or RTX 5090) running a single user's inference, even for smallish (say, 35B) models.

Søren Torbjørn Svendsen
@sorentsvendsen.eurosky.social

Men det kræver stadig en temmelig kraftig enhed derhjemme. Jeg leger med at lave et transkriberingsprogram lokalt på en bærbar med en rtx studio 4070 8GB og den pust og stønner, når den skal processere først Whisper og Pyannotate og efterfølgende fx llama3.1 til renskrivning af transkriptionen.

Rasmus Aagaard
@rasgaard.com

Helt klart. Der er stadig tonsvis af quirks. Jeg er mest begejstret for tendensen der bliver vist. At software og hardware begge udvikler sig i retning af at lokal inference bliver konkurrencedygtigt ift. cloud.