Operations | Monitoring | ITSM | DevOps | Cloud

When the internet goes down, our primary job is to help everyone get back up, as fast as possible. Of the almost half a million incidents we've helped our customers solve, there are some which stand out for both their scale and impact. One of these happened on Monday, October 20, when AWS had a widely covered major outage in their us-east-1 region, from 07:11 to 10:53 UTC. We’re hosted in multiple regions of Google Cloud and so the majority of our product was unaffected by the outage.

Earlier this week, we gathered in San Francisco for our second SEV0—almost a year after our very first event. SEV0 has always been about shining a light on the biggest challenges (and opportunities) in incident response. Last year, we were still talking about the fundamentals: blameless culture, strong processes, and lessons from the best in reliability. This year felt different. AI has moved from background noise to front and center in every conversation, every team, everywhere.

In December 2024 we launched Scribe to help responders never miss a detail from their incident calls. By automatically transcribing calls and highlighting key information, Scribe eliminates manual note-taking, reduces time spent getting up to speed, and preserves valuable context for post-incident analysis. The feature quickly gained popularity among our customers, but with success came an influx of requests for bug fixes, extra functionality, and wider call platform support.

I joined incident.io about ten weeks ago, having been in my previous role for four and a half years. Being a new starter was an unusual feeling for me, and there's been a huge amount to learn; but by lunch on my second day (!) I had started shipping value to our customers. A large part of hitting the ground running has been having a colleague alongside me, who I can pester with questions, who doesn’t get offended when I write in all capitals, and often praises me for being absolutely right!

At incident.io, speed is essential. Our product is growing faster than ever; in scope, range of features and the number of people contributing to it. In the early days, when you’re a small startup with just a few hundred endpoints, a basic API setup gets you by. But as things scale, you need to make creating endpoints easy, fast, and reliable.

You’ve made it to Senior engineer. Now what? You’re now staring at the next level, Staff typically, sometimes Principal, or whatever your company calls it. The path feels murky. Your manager gives you feedback like “show more technical leadership” or “think bigger picture”, but what does that actually mean day-to-day? I’ve been there. I’ve also been on the other side, helping engineers grow through whatever explicit (or implicit) levels a company has.

Many, many years ago, I was a computer science major at the University of Illinois, hoping someday I’d be able to write code for a living. I started my career in QA hoping to learn the ins and outs of software development. But it turns out I wasn’t very good at coding. I was just good enough to get a role as a sales engineer, where all I had to do was write code that could hold together for 30 minutes in a demo.

"Show me all critical incidents from the last week." "Create an incident for the payment API being down." "What was the root cause of that database incident last Tuesday?" If you've ever wished you could just ask Claude (or any MCP client) to handle incident management tasks instead of context-switching between chat and your incident management dashboard, you're going to like what we built.

The Quest For The Five Minute Deploy Speed is everything at incident.io. The faster we can test and ship code, the faster we can get new products and features out to customers. Over the last three years, as our codebase grew and our test suite expanded, we drifted away from our own goals: "We aim for less than 5 minutes between merging a PR and getting it into production." This is the story of how we got back on track.

At incident.io, we are building a product that our users rely on 24/7, all year round. This means it is crucial that it is always working, and that is where our on-call rotation comes in. We believe that everyone should be on-call because it tightens the feedback loop between shipping new features and maintaining what we have, leading to more pragmatic engineering decisions.