Post

😱 You see this distribution of response times from running 500 requests/sec against some microservice for 120 sec. What do you do to in order to investigate what's going on?

You measure for longer to see if the pattern repeats itself to see if there's any hint on when this happens. If possible, try to color it based on the container where it is running in?

It's just a single process, not running in a container. The pattern occurs always around the same time (20 sec in) after starting the load test.

What kind of program is it? Does it make requests? It kind of feels like there is an exponential backoff that doesn't trigger the first try.

Try Hyperfoil 😉 Seriously, we've tested it against other load drivers and it's reporting is more accurate. New blog post coming in the not too distant future 🙂

Ha, yeah, turns out it was on the load gen side indeed. It was running out of connections. Using HTTP2 mitigates it (via pipelining), which I got reminded of reading the docs of... Hyperfoil ;) Is the project alive? Saw the last post is from a few years back.

With oha, any idea what is the cause (reaching maximum number of open files, oha bug or something else)?

@stalep.bsky.social, in case you have a moment, posted a question on Hyperfoil here: github.com/Hyperfoil/Hy... 🙏.

Load gen as bottle eck, nice classic! Wasn't this what Gatling was originally meant to solve (vs jmeter which didn't do that well)?