Operations | Monitoring | ITSM | DevOps | Cloud

Like most tech companies, we use an on-call rota and various alerting tools. We do this to respond to incidents before they’re reported. Proactively identifying issues and communicating to customers helps us provide great experiences and fosters trust. Internally, we’ve been using these alerting tools in tandem with our auto-create incidents feature. We’ve found that it’s made responding to the pager much smoother - it’s one less thing to do when you get paged at 2am.

We’ve been building incident.io for 12 months and thought it would be a good time to share the constellation of tools that we’re using to power our customer experience.

When critical incidents happen — which they inevitably do 😅 — and you’re in the middle of trying to figure out what the best thing to do is, it can feel comforting to know that you’ve got a pre-prepared list of instructions to follow, commonly known as an “incident response plan”: In theory this sounds quite simple, and a typical flow you might envision is: It might be tempting to think that the hardest part of running incidents is finding or writing a checkl

At incident.io, our number one priority in engineering is pace. The faster we can build great product, the more feedback we can get and the more value we can deliver for our customers. But pace is a funny thing. If you optimise for pace over a single month, you’ll quickly find yourself slowed down by the weight of your past mistakes.

We’ve been celebrating a few big milestones 🎉 at incident.io in the last few months. We were recently discussing product metrics (as you do for fun on a Friday afternoon 🤓) , and Lawrence was very surprised with a particular stat around the number of workflows that have been run using incident.io.

At incident.io, we’re building tools to help people respond to incidents, often by automating their organisations’ process. Much of this is powered by our Workflows product, which customers can use to achieve things like: Workflows as a product feature are incredibly powerful, and we’re proud of the value they provide to our customers. Behind-the-scenes, though, building something like workflows can be difficult.

This is the second in a two part series on how we built our workflow engine, and continues from Building workflows (part 1). Having covered core workflow concepts and a deep-dive into the Workflow Builder in part one, this post describes the workflow executor, and concludes the series with an evaluation of the project against our goals.

Not many companies will tell you why you shouldn’t use their product, but any product that tries to be everything to everyone is doomed to failure. When you build without a specific user in mind, your target becomes the intersection of many viewpoints, and what you build is the lowest common denominator. What usually follows is software that can technically do everything, but feels unfocused, complex, and unpleasant to use. Something everyone is equally unhappy with.

We've all done it: "that'll be simple, I'll just write a quick script and..." In the case of calculating on-call pay, we really have done it before: our team have built the on-call pay scripts for several companies, and each attempt was a painful, error prone process. While we believe everyone on-call should be paid for their inconvenience, relying on someones side-project or back-of-napkin maths to calculate pay leads to mistakes, frustration, and wasted time.

We get it – incidents happen. What differentiates resilient teams from others is how they learn from them: using them as an opportunity to find the biggest improvements in how they work. Incident timelines are one of the most simple and effective tools available to you when it comes to learning from an incident. It’s vital that you ensure they’re accurate and useful, in order to make the biggest improvements after an incident.