Operations | Monitoring | ITSM | DevOps | Cloud

Incidents provide an unparalleled opportunity to learn about your people, processes, and products under pressure. In this post, we’ll tell you how to ensure your team isn’t letting these opportunities for learning go to waste.

Budgets in IT departments are tight these days, so proving a return on investment is essential for justifying or expanding a project. The good news is that automation saves money by reducing the amount of human effort required. It is similar to investing in a robot vacuum cleaner. Despite the upfront cost, you save time (and money) by not having humans do the vacuuming. Reporting the value delivered by an automation program can be challenging since the value depends heavily on what is being automated.

We’ve all been in the situation before: it’s Friday at 5 PM and the only on-call engineer available to handle incidents is about to hit the slopes. Unfortunately, at that very moment, a customer reports to support that they are unable to access the company’s ecommerce website to complete a purchase. Internal monitoring systems seem quiet and services appear available on internal health dashboards.

You're probably aware that downtime is expensive—but do you know how expensive it is? The short answer is—very. According to the Ponemon Institute, outages cost organizations an average of $9,000 per minute (or $540,000 per hour). That's why companies of all sizes are investing in incident management tools to reduce their downtime and improve the customer experience.

There’s a familiar saying: garbage in, garbage out. For ITOps, this directly applies to data engineering. BigPanda’s Area Vice President of Value and Adoption, Craig Ferrara, says the importance of data hygiene—putting good data in to get good data out—is the core of data engineering, and it requires ITOps to take a look at their data before integrating with an AIOps solution.

Over the last 18 months at incident.io, we’ve done a lot of migrations. Often, a new feature requires a change to our existing data model. For us to be successful, it’s important that we can seamlessly transition from the old world to the new as quickly as we can. There are few things in software where I’d advocate a ‘one true way,’ but the closest I come is probably migrations. There’s a playbook that we follow to give us the best odds of a smooth switchover.