Operations | Monitoring | ITSM | DevOps | Cloud

If you’re in IT operations or manage NOC, SRE, and DevOps teams, chances are your IT environment is growing complex for you and your teams to manage. Any enterprise, large or small, around the globe, is continuously changing its IT stack due to evolving business requirements and significant industry trends. But digital transformation, hybrid infrastructure, DevOps adoption, and continuous integration and continuous delivery (CI/CD) pipelines are all causing major headaches.

Alert fatigue is a serious issue that affects numerous professions, especially in the IT industry. It can lead to neglecting critical events and delaying response times. IT teams need to continuously monitor their systems and applications to avert possible downtime and keep operations running smoothly. However a high number of incoming alerts inundating these teams can make them less responsive. The ramifications of such disregard can severely affect the efficiency and dependability of IT teams.

Catastrophic failures — such as a security breach or a complete outage leading to an unavailable product or service — are classified as Sev0 incidents. On a severity scale of 1–3, Sev0 is dire. It brings business to a complete standstill and may lead to loss of revenue and a damaged reputation. A Sev0 incident usually has no quick workaround; it requires a coordinated effort beyond the engineering team to diagnose, correct, and manage.

At a recent training session, Jeli spent a great deal of time covering incident retrospectives and what makes an incident worthy of studying. My colleague Ben Hartshorne asked a fascinating question, which I’ll paraphrase here: That caught me by surprise. We had a great discussion, and it made me consider approaches I hadn’t before.