Post

Ever wonder what it's like to run larger models at your existing context sizes? Ollama just introduced K/V context cache quantisation and it's a game changer!

Architecture Notes

@architecturenotes.co

With K/V context cache quantisation, you can experience significant reductions in VRAM usage. It's time to realise the full potential of expanded context sizes!

25/1/25, 10:00 pm0 reposts0 likes

Architecture Notes

@architecturenotes.co

25 Jan 2025

Here's an insightful read about this development and its implications: smcleod.net/2024/12/bri...

smcleod.netBringing K/V Context Quantisation to OllamaK/V context cache quantisation has been added to Ollama. This enables significant reductions in VRAM usage, allowing users to realise the potential of expanded context sizes and run larger models at their existing context sizes.