Post

The more I work with LLMs and agents, the more I'm gobsmacked by the n^2 nature of how conversations are handled. For every turn of the conversation, you send the entire previous conversation. This means the costs compound the longer the conversation goes on.

I understand how LLMs work and why this is necessary but it is crazy to me that we are sending 100-500k+ of data with each turn. Keeping those conversations short makes everything so much more efficient. But most people aren't aware.