Post

A new, highly recommended article from @rasbt.bsky.social: "Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention" -> magazine.sebastianraschka.com/p/recent-dev... #MLsky

magazine.sebastianraschka.comRecent Developments in LLM Architectures: KV Sharing, mHC, and Compressed AttentionFrom Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs

16/5/26, 2:55 pm4 reposts25 likes

Sebastian Raschka (rasbt)

Thanks!!

No problem, I am very grateful for your articles (and videos etc), I learn so much from them. BTW: A few days ago I stumbled across this "Simple Attention Networks" concept from Cactus, mostly for one-shot tool use, solely based on the provided context. -> github.com/cactus-compu...

github.comGitHub - cactus-compute/needle: 26m function call model that runs on incredibly small devices26m function call model that runs on incredibly small devices - cactus-compute/needle

fry69

@fry69.dev

16 May

Dunno if that fits in your model garden, but I found this drastic and targeted approach kind of refreshing.

fry69

@fry69.dev

16 May

FWIW here is also a reddit link about this model with some discussion, if interested -> www.reddit.com/r/LocalLLaMA...

reddit.comFrom the LocalLLaMA community on Reddit: Needle: We Distilled Gemini Tool Calling Into a 26M ModelExplore this post and more from the LocalLLaMA community