Post

Today in "weird shit Kubernetes does": did you know that if you try to do an in-place resize for a pod, the result can be one of three states?

Those states are "yes, did it", "nope, sorry, can't fit", and "well it can't fit RIGHT NOW, but it might fit sometime in the far distant future, so I'm just gonna sit here forever and hope" (The technical term for the last state is "Deferred", in case you're curious)

The main way you can get into this state (aside from just "durrrr it took containerd 20 seconds longer than expected) is if you have pod requests != limits.

In this case, your pod might try to resize up, but all the other pods on the node are being greedy jerks and hogging all the resources, so kubelet's like "Welp I'll just wait for everyone else to stop being a greedy jerk 🤷"

Hey kubelet, lemme give you a clue! The other pods aren't going to stop being greedy jerks! Like, ever!

Anyways, I guess this is just yet-another reason why you should always set requests=limits.

Genuinely the most controversial statement I've seen in my career. (Even though I subscribe to it.)

In fairness, if it weren't for up to 99ms hard stalls being caused by limits, it'd probably be a lot less controversial.

I'm actually thinking about submitting a kubecon talk about this. "Revisiting requests versus limits in 2027" or something like that.

I would love to see @thock.in's response to seeing that talk in the CFP list.