Post

bread
@bread.codes

while I am anti-LLM, I did get lost training my own to test a hyper-specific focus. I also now see how hard the optimization problems are TLDR: this shit doesn't scale that is inherent to the discovery that bigger = better, and attention being quadratic complexity with few optimizations

bread
@bread.codes

LLMs _will_ crash and burn economically all that will be left is the discovery of attention optimizations in ML, which is interesting, but also shows how dumb and inefficient these models really are and how attention tricked us into thinking the models know what we're saying

0 reposts1 like

Hurrdurr
@devhurrdurr.bsky.social

Not an expert, I’d like to hear your thoughts. Feels we hit the ceiling. Bigger models and context is nice, but one model for any question in any language doesn’t scale. Do you think they’re suitable for smaller tasks, eg. A to B translation, summarizing text. What do you mean “no speak”?

bread
@bread.codes

I'll try not to ramble. We have hit a ceiling, and may soon top-out modern architecture. The big discoveries that led to modern LLMs were: - Bigger = Better, which is antithetical to scaling - The "Attention is all you need" paper solving attention (so the LLM can "understand" things like pronouns)

bread
@bread.codes

You came near the idea that we could make smaller "expert" models. Add a router layer and you have MoE (mixture of experts) which has made the models perform better. Every new model today uses MoE already.