Chris Evans

Chris Evans

@evns.io

Building @incident.io πŸ”₯ Typically found in London, New York, or somewhere in between. Formerly on Twitter at https://x.com/evnsio.

5.4K followers394 following108 posts

Picnic is an online supermarket on a mission to make grocery shopping simple, fun, and affordable. And they're clearly a company with great taste, choosing @incident_io to help them achieve their high bar for reliability.

View full thread

β€œWe are now moving to a level where we are building up structural technical resilience, meaning whenever there's an incident happening, there's no longer customers affected, and that is basically the ultimate goal that we are striving for.”

Picnic is an online supermarket on a mission to make grocery shopping simple, fun, and affordable. And they're clearly a company with great taste, choosing @incident_io to help them achieve their high bar for reliability.

View full thread

With all this in mind, it really is a huge honour for @incident_io to be the software they use to help detect, respond to, and mitigate issues as they happen.

This comes off the back of a full migration from PagerDuty to @incident_io On-call, and a replacement of their in-house incident response tool too. And in terms of impact, I really can’t put it better than Daniel Gebler, their CTO:

Picnic is an online supermarket on a mission to make grocery shopping simple, fun, and affordable. And they're clearly a company with great taste, choosing @incident_io to help them achieve their high bar for reliability.

View full thread

and was treated to a tour of one of their automated fulfilment centres. Suffice to say, it’s an incredible operation, where robots and humans work side-by-side to get groceries ready for delivery.

And when Picnic talk about combining the digital and operational worlds, they really mean it. There’s bespoke software running everywhere, and Grafana dashboards monitoring the status of thousands of devices, sensors, and systems.

Picnic is an online supermarket on a mission to make grocery shopping simple, fun, and affordable. And they're clearly a company with great taste, choosing @incident_io to help them achieve their high bar for reliability.

Behind the scenes, it's a complex hybrid business, combining a digital platform with a vast physical network of warehouses, delivery hubs, and a fleet of thousands of electric vehicles β€” all running in perfect coordination to keep millions of customers stocked up and smiling.

After six months of watching our engineering team deep in the AI trenches, I’m more convinced than ever: AI isn't just a nice-to-have in incident managementβ€”it’s a necessity.

View full thread

At incident, we’re trying to build in the opposite direction. Instead of replacing people, we’re designing AI to show up like your best engineer. Not the brilliant jerk who fixes the thing while you're grabbing a coffee, but the one that helps you understand, decide, and act when things are on fire.

After six months of watching our engineering team deep in the AI trenches, I’m more convinced than ever: AI isn't just a nice-to-have in incident managementβ€”it’s a necessity.

View full thread

But there’s a catch. Lisanne Bainbridge highlighted it back in 1983 in her paper "Ironies of Automation": the more we automate, the more we deskill.

People get good at watching systems, not running them. Then something breaks, and we expect them to swoop in and save the day, with little context and limited practice.

After six months of watching our engineering team deep in the AI trenches, I’m more convinced than ever: AI isn't just a nice-to-have in incident managementβ€”it’s a necessity.

As AI enables faster software development and systems become harder to fully understand, we’re left with a growing problem: how do we safely operate what we no longer fully grok?

My take: the only viable solution is to use AI to support the people operating these systems. It’s not unlike modern aircraftβ€”too complex for a human alone, so we rely on automation to augment the pilot, not replace them.

View full thread

Take the forklift-jet-engine saga... Had the forklift flipped over and injured 4 adults, this would be a very different post. And yet, I'm here congratulating the ingenuity.

So when you're debriefing next and every fibre of your being wants to criticise someone for taking a risky action that made things worse, consider how different you'd be feeling if their heroics actually paid off.

View full thread

But what does this have to do with incidents? Incidents are a place where things that have never been done before, happen all the time. And when it comes to debriefing, it can be fascinating to dig into the actions that are taken to understand how things really get done.

What's interesting to me is that these off-piste actions are often highly dangerous, and when it comes to debriefing, there's a fine line between blame if it goes wrong, and celebration when it goes right.

View full thread

What did they do? First up, they got 4 adults to stand on the back of the forklift as a counterbalance. Turns out we were nowhere near heavy enough.

Then, they fired up a small tractor, balanced it's front loader on the pickup, pushed down so its full weight was on the forklift, and lo-and-behold it worked. But don't take my word for it. Check out the video of the whole thing below!

View full thread

In any kind of normal mode of operation, this is where you'd go back to the drawing board, replan and probably order a crane to unload it.

But the implications of doing so were high: The truck couldn't wait, so would have to take the engine to be unloaded elsewhere, the cost of the forklift was sunk, and the cost of hiring a crane likely significant. <Insert joke here about them being able to afford it>

I recently stayed at an Airbnb where the owner (a self-confessed plane nerd) had ordered a disused 747 jet engine to be a show piece on the property.

It showed up on a truck the size of a small country, and the owner had a forklift ready to unload it. Only the forklift was severely unpowered, and insufficiently weighted to lift the thing up. Turns out jet engines are pretty heavy at ~4.5 tons.

We're building an AI incident responder πŸ”₯ Not to replace humans, but to work jointly alongside them. Initially, we're focused on accelerating the investigation and diagnosis stages of an incident, helping search and correlate data points, signals and context from across your organization.

This week, we launched Scribe β€” an AI agent for incident calls, that: 1️⃣ transcribes in real time 2️⃣ shows the current topic in Slack/Web so you always know what's being discussed 3️⃣ surfaces key points and actions, and summaries on the go Genuinely one of the best things we’ve shipped this year! πŸš€

Big news. incident.io is opening a base in San Francisco! πŸŒ‰πŸŒ‡

incident.io@incident.io

πŸŽ‰ We’re thrilled to announce that incident.io is opening a new office in San Francisco! πŸŽ‰ With London, NYC, and now SF, we’re officially working across three time zones to better support our customers, grow our team, and stay close to the action. See you in January 2025 πŸ”₯

Has there been any public analysis of Twitter usage against the growth in Bluesky? I imagine not, but would love to see it.

There are few things as infuriating as having to explain to customer support how basic features in their product work. I do get it, it's hard to train people consistently, but it doesn't make it any less annoying πŸ˜…

I'm pretty pleased with how this revamp turned out. Looks better, reads better, and feels better πŸ˜‰

incident.io@incident.io

We’ve revamped our tactical guide to modern incident management, including: πŸ“Ÿ Healthy on-call practices πŸͺ¨ Incident basics & tracking πŸ”₯ Response tactics πŸ‘©β€πŸŽ“ Learning from incidents πŸ“ˆ Key metrics No fluff, just real-world insights from actual incident responders. Enjoy! πŸš€ go.incident.io/guide

Bluesky being spoken about on BBC 5Live this morning. Definitely feeling like it’s hit the critical mass and exposure needed to really get the wheels turning.

Chris Evans reposted

When you know you're supposed to be learning from the incident, but that sweet, sweet blame is calling your name... #blamelessculture #postmortem