Operations | Monitoring | ITSM | DevOps | Cloud

A significant part of being on-call is the ability to respond to pages and handle escalations on the go. In the early stages of developing incident.io On-call, we considered whether a Minimum Viable Product (MVP) could rely solely on SMS and phone calls. However, we quickly realized that a fully featured mobile app was going to be essential to the on-call experience. This led us to the question: how should we build this mobile app?

We’re excited and honored (and might even be blushing a little) to share our Summer 2024 accolades from G2, including being ranked #1 in G2’s Relationship Index! There are several factors that go into determining this ranking, including: While all of these awards are special to us, Best Relationship means a lot because, well, our customers mean a lot.

March 5th was a big day for incident.io as we released our on-call product to the world. Nine months of listening to our customers, coding, fixing, testing, and polishing came together for our biggest product launch to date. Releasing On-call was a huge milestone and represented the next step in our journey as a company.

At incident.io we use infrastructure as code for configuring everything we can, and we feel that there’s no reason we should exclude our own product from that. As well as configuring things like Google Cloud Platform, Sentry and Spacelift via our infrastructure repo, we also configure incident.io. On your first day as an engineer here, the first PR that you make is to our infrastructure repo.

Whenever a new customer joins incident.io, we make notes on what made them chose to buy our product and, if we were in a competitive process, why they chose us over other providers they were evaluating. It’s a lot of messy data and raw notes, but contained within is a veritable treasure trove of customer feedback. Summarising large amounts of data? Sounds like the perfect job for an LLM.

We approached building a Terraform integration with a sense of trepidation. One of the things that motivates us is building features we think people are going to love using, and Terraform integrations are often not that. Terraform integrations have a number of common pitfalls. Building resources by hand is tedious, and requires deep understanding of their specification. Importing and managing existing resources is also often painful.

Clean, clear, and ready to be customized to suit your needs. Google Docs Having a dedicated incident post-mortem is just as important as having a robust incident response plan. The post-mortem is key to understanding exactly what went wrong, why it happened in the first place, and what you can do to avoid it in the future.

Remind yourself of the worst incident your organization has faced. If you’re lucky it might have been your entire service being offline for a period of time. Less lucky, and perhaps you encountered something affecting the sensitive data your organization is the custodian of. Whilst uncommon, incidents of this severity happen to every organization at some point. This criticality of situation is what many refer to as a Sev0, the most severe of incidents.

When an incident strikes, it often brings a whirlwind of stress for everyone involved—from the teams directly handling the issue to the stakeholders making crucial decisions. Imagine support teams on high alert, customers anxiously awaiting resolutions, and executives probing for answers to steer the company through turbulent times. This mounting pressure can make a challenging situation nearly unmanageable, especially when faced with problems that are new or unexpected.

We went live with our first set of AI-enabled features a few months ago. Needless to say, we learned a lot along the way, as this was the first time we had experimented with generative AI. Here, I'll share some of what we've learned as we’ve grappled with using LLMs to power new products at incident.io. This will be most applicable to the application layer, AI-enabled but not AI companies.