drmorr

drmorr

@drmorr.appliedcomputing.io

Optimizer. Photographer. In disbelief that computers actually work. Accidentally writing a role-playing game. (he/him) https://hachyderm.io/@drmorr

156 followers105 following1.1K posts

Pinned

Hi bsky-ians! I rarely interact or do anything over here; mostly I just post links to my blog. All of my professional shit-posting happens over on Mastodon. But anyways follow along if you like! Here's my mastodon profile too if you care: hachyderm.io/@drmorr

hachyderm.iodrmorr (@drmorr@hachyderm.io)6.29K Posts, 184 Following, 302 Followers · Optimizer. Photographer. In disbelief that computers actually work. Accidentally writing a role-playing game. (he/him)

Ok, after a break last week we're back with the 4th key autoscaling metric, namely, pod disruption! The fact that this was the 4th post in the series is entirely coincidental, but also hilarious, because 4 is now hilarious. blog.appliedcomputing.io/p/key-autosc... #devops #autoscaling #kubernetes

blog.appliedcomputing.ioKey Autoscaling Metric #4: Pod Misruption SchmudgetsOk, y’all, this is the 4th post in our Key Autoscaling Metrics series, and you could say this post is the entire raison d’être, as it were, for this entire blog series.

Good news, all our servers are ignoring the number 4 and the letters PHP. This security vulnerability episode brought to you by Sesame Street.

The income from the ACRL blog helps us attend conferences (like KubeCon, which is coming up!), which in turn helps us stay in business longer!

drmorr reposted

I don’t know how to explain to people that just because certain things appear to approximately work doesn’t mean they actually work. This was already horrifically difficult before AI and now it’s damn near impossible to get the point across when literally any trivial example passes muster

drmorr reposted

Replying to @gimpy.bsky.social

Setting CPU limit is still almost always wrong and setting memory limit=request is still almost always right.

drmorr reposted

Replying to @drmorr.appliedcomputing.io

In fairness, if it weren't for up to 99ms hard stalls being caused by limits, it'd probably be a lot less controversial.

Today in "weird shit Kubernetes does": did you know that if you try to do an in-place resize for a pod, the result can be one of three states?

View full thread

Today in "weird shit Kubernetes does": did you know that if you try to do an in-place resize for a pod, the result can be one of three states?

View full thread

The main way you can get into this state (aside from just "durrrr it took containerd 20 seconds longer than expected) is if you have pod requests != limits.

In this case, your pod might try to resize up, but all the other pods on the node are being greedy jerks and hogging all the resources, so kubelet's like "Welp I'll just wait for everyone else to stop being a greedy jerk 🤷"

Today in "weird shit Kubernetes does": did you know that if you try to do an in-place resize for a pod, the result can be one of three states?

Those states are "yes, did it", "nope, sorry, can't fit", and "well it can't fit RIGHT NOW, but it might fit sometime in the far distant future, so I'm just gonna sit here forever and hope" (The technical term for the last state is "Deferred", in case you're curious)

Welp just got my 100%-human written blog posts kicked off of r/kubernetes because they're "LLM slop". Cool cool cool. Love getting accused of cheating, lying, and being lazy all in the same sentence, it's my fav.

Also as a reminder, becoming a paid subscriber to my blog is one of the ways you can help support our conference attendance, which in turn helps keep us in business! If you like what we do please consider subscribing!

I've heard from a lot of different folks that "autoscaling isn't sexy" and "it's not that hard to do", which -- speaking as someone who's been specializing in autoscaling for a decade or so -- is clearly incorrect.

sk-spark is part of the SimKube suite of tools; it's useful for simulating the behaviour of Spark on top of Kubernetes, together with Karpenter as a node autoscaler. The question it's designed to answer is "how does Karpenter's consolidation behaviour impact the success rate of Spark jobs?"

Anyways, I had fun with this one! Take a look and play around with it if you want. It's in Rust, of course. Next week (probably) I will write a blog post to describe it in more detail.

(By the way, we're starting our KubeCon planning already, and becoming a paid subscriber to the blog is one of the ways you can help support our KubeCon attendance in the fall!)

A reminder: we're trying to get to around $5k in paid subscription revenue on the blog. Right now we're hovering at just under $1k. If you like the things we're posting, please consider signing up for a paid subscription!

I am trying to increase my paid subscription revenue for the blog to $5000/year; this helps us attend conferences, which are our primary source for new business. So the blog doesn't directly pay the bills, but it does help us keep doing what we're doing. blog.appliedcomputing.io/subscribe

blog.appliedcomputing.ioSubscribe to Applied Computing Research LabsOpen-source research and development in distributed systems. Click to read Applied Computing Research Labs, by drmorr, a Substack publication.

drmorr reposted

This talk features my postmortem of the worst incident I've ever dealt with in production! Excited to get to finally check out the talk.

drmorr@drmorr.appliedcomputing.io

Are you bored on a Friday afternoon? My SRECon talk, "Stop Reading Changelogs: Safer Kubernetes Upgrades with Simulation" is now up on YouTube. Go watch it! Feels very timely since Kubernetes (checks notes) 1.36 was just released! www.youtube.com/watch?v=yk_P... #srecon #kubernetes #simkube

drmorr reposted

Replying to @drmorr.appliedcomputing.io

Simulation testing before upgrades beats changelog roulette every time, though honestly most teams still just wing it and hope the canary doesn't explode.