The ever-growing complexity, scale and pace of IT environments puts a huge burden on IT Ops, NOC, and DevOps teams, who are tasked with keeping these environments up and running. One of the biggest challenges is Root Cause Analysis (RCA). When something breaks, they need to determine what broke it, and they need to do it fast.
My 2nd Gartner IOCS experience ended with a bang, just like my first one, with an amazing presentation by Jason Walker at Blizzard Entertainment (part of the $42B Activision-Blizzard gaming powerhouse) on how they use AIOps to deliver an amazing gaming experience for hundreds of millions of gamers around the world, 24×7. But before I get to that, let me share some more key takeaways from the conference.
Critical and sev1 incidents are always a priority, but what about those dozens and often hundreds of lower priority ones that often sit in a queue waiting for a first response engineer to get to them? Do you find that no matter how much effort your team puts into minimizing the number of queued incidents, their number always seems to grow? If this sounds familiar – this blog is for you.
Root Cause Analysis (RCA) is an all-encompassing process. It is usually very complicated and often requires many people with many different skills – all trying to tackle an incident to determine what happened, when, why, how and ultimately who (to blame). There is, however, secret sauce today that can help solve many issues before a “full-scale” RCA process is initiated – and that is Root Cause Changes (RCC).
TL;DR: Fast-moving IT stacks see frequent, long and painful outages. Thousands of changes – planned, unplanned and shadow changes – are one of the main reasons behind this. Until now, IT Ops, NOC & DevOps teams didn’t have an easy way to get a real-time answer to the “What Changed?” question – the answer that can help reduce the duration of outages and incidents in these fast-moving IT stacks. Now, with BigPanda Root Cause Changes, they do.
The dreaded monthly operations and application meeting is edging near. You know which meeting we’re talking about: the one where all the Ops teams get together to talk about what they are seeing, about application enhancements and modifications, potential improvements, and often – about their frustrations!
