Post

TOPS measures compute. LLM decode is memory-bound. Every token reads model weights. tok/s = bandwidth/model size. 7B Q4 on 102 GB/s: ~25 tok/s. 204 GB/s bus: ~51 tok/s. capytoolkit.com/blog/buying-... #LocalLLaMA #AIPC #llm