Around six years ago on a Wednesday morning, software professionals worldwide were startled by a tweet from GitLab stating that they had accidentally deleted their production data, causing their site to go offline. Unfortunately, at that point in time, the open-source code repository giant had no idea that it would take them another 36 hours to restore their systems only to learn that 5,000 projects and 700 new user accounts were affected while they were fixing the outage.

IT people know that data is king, especially in optimizing IT operations. However, figuring out which metrics to collect and how to collect them can be challenging. IT teams have to factor in what IT directors, team managers, and the people overseeing operations want, what they’re concerned about, and what they consider important.

Several challenges impact customers and operations of utilities and energy companies, including aging infrastructure, cybersecurity threats, inclement weather, operational failures and transmission interruptions. These challenges can cause prolonged service disruptions, potentially leading to customer attrition and irreversible damage to businesses. Responding quickly and efficiently to incidents is critical to minimize damages or contain potentially dangerous scenarios.

Think about this—what was the last incident that challenged you? Did you learn anything from it? It will be shocking to no one to hear that we deal with our fair share of incidents. These run the gamut from tiny bugs to significant outages (thankfully, the latter happening only very rarely 😮‍💨). Either way, we always take the time to learn from them in some way. This might look like changes to our response processes or revisiting systems we’re using.

Managed Detection and Response (MDR) is an approach to cybersecurity that combines advanced technologies, skilled analysts, and a proactive response process to detect, investigate, and remediate cyber threats. MDR is typically delivered as a service by a third-party provider and includes a range of security capabilities, such as threat intelligence, behavior analysis, anomaly detection, and incident response.

As software development continues to evolve and become more complex, the need for efficient and effective deployment strategies has become increasingly important. This is where deployment pipelines come in. When it comes to software development, a deployment pipeline is a powerful automated tool that facilitates the fast and accurate transition of new code changes and updates from version control to the production environment.

This is the story of an insurance company that was getting six million IT alerts every 90 days and how they used BigPanda’s AIOps to reduce it to less than 50,000. Before we get into that though, let’s take a step back. How did we, as an IT sector, get to a place where organizations receive 6,000,000 IT alerts in the first place?

Be sure to register for the launch webinar on Thursday, March 30th to learn more about the latest release from the PagerDuty Operations Cloud. Rundeck by PagerDuty has long helped organizations bridge operational silos and automate away IT tasks so teams can focus more time on building and less time putting out fires. And while this mission still rings true today, our vision is to extend this reality and revolutionize all operations while continuing to build trust.

Mean Time To Repair, or MTTR, is a critical metric in IT incident management that measures the average time it takes to fix a system failure. The meaning of MTTR can be understood as the average duration needed for an IT team to recover from an incident. It is a fundamental metric for IT teams to track and analyze their efficiency in resolving incidents.

On Thursday, March 9, 2023, something was afoot at our primary bank, SVB. By Friday, March 10, 2023, messages from our investors helped us quickly understand that FireHydrant needed to maneuver through a complex incident that was unfolding. Operational incidents are incidents like every other.

In recent years, the adoption of cloud computing has skyrocketed as more and more businesses realize the benefits of this modern IT solution. With its unparalleled reliability, scalability, and cost-effectiveness, cloud computing has become the go-to choice for many organizations. According to recent estimates, around 90% of businesses are already using some form of cloud computing, and this number is only set to rise in the coming years.

Shouldn’t your alerts be doing more of the work for you? A noisy channel with every alert from hundreds of monitors and microservices is a chaotic place to actually find the incidents that are impacting your customers. And it still requires a heck of a lot of human intervention. We think it’s time for something better. Today we’re releasing Alert Routing: the next phase of worry-free automation from FireHydrant.

Agility matters in incident response, and the easiest way to spring into action is by having a well-defined team in place ahead of time. The right people in the right roles will help you respond to and resolve incidents more quickly and efficiently. In fact, we found in the Incident Benchmark Report that incidents with roles assigned had a 42% lower mean time to resolution than those that didn’t. But what roles do you need to fill?

PagerDuty provides a SaaS-based platform that enables developers, DevOps, IT operations, and business leaders to prevent and resolve incidents that could potentially impact customer experience. This platform allows organizations to proactively manage events that may affect customers across their IT environment, which is crucial for maintaining customer satisfaction, revenue, and brand reputation.

It may be counterintuitive to tackle a problem without knowing exactly what the problem is, but an incident commander often does just that. In fact Rob Schnepp—founding partner at Blackrock 3, an Alameda, California-based incident management consulting group—says identifying the root cause of an incident is typically secondary to addressing the symptoms.

ITOps professionals know that their AI and automation goals can only be achieved with high-quality data. How can you get good-quality data? Incident Intelligence. In this on-demand session from Pandapalooza, our Group Product Manager, Orr Ganani, joined our Regional VP of Professional Services Sales, Jordan Gamble, to discuss Incident Intelligence and its benefits. Read on to learn more about Incident Intelligence from this webinar.

Question—when was the last time you purchased a new product and sat down to read the manual end-to-end before getting started? Ask this question to a room of 10 people and you’d likely get one or two hand raises, even though reading first could save you time and preempt many of the questions you’re likely to ask. Herein lies the problem when it comes to creating a SaaS product.

In our research for the inaugural State of Availability Report, we asked 1,900 engineers about mean time to detect (MTTD) and mean time to recovery (MTTR) as two leading incident management Key Performance Indicators (KPIs) strongly associated with availability. We learned that less than 15% of respondents are tracking their MTTD. It takes twice as long to discover an issue than it does to resolve it.

Today, many digital technologies in IT can operate with minimal human intervention. However, while they boost productivity and drive growth, any failure or unpredictable behavior can pose a significant challenge for the ITOps and DevOps teams. So, effective IT incident management helps minimize the impact of incidents on business operations and ensures that systems are restored as quickly as possible.

It’s no secret that every ITOps leader can face an ever-increasing amount of alerts. Since the dawn of digital, alerts have served an important purpose. Sometimes all those alerts can become overwhelming noise, and sorting out what is and is not a priority can become challenging. The good news is that artificial intelligence (AI) and machine learning (ML) are adept at processing large data sets in real time, looking for patterns and being able to aid in decision making.

Collaboration is essential to running effective, learnings-filled retrospectives. FireHydrant’s new retrospective commenting makes it easier for teams to create accurate, thorough retros, together.

To hear Ehab Tarabay explain it, the need for retailers to continue evolving their digital operations is an age-old problem. I recently hosted Tarabay, head of workplace IT services at TMF Group, on our That’s Great IT podcast. As an avid information technology specialist with a track record of more than 20 years in the technology field, he had a unique perspective to share about the shift that’s happening in retail right now.

As an ITOps professional, it can be challenging to justify all of your actions to your organization. After talking with many of you, we saw first-hand the pains and gaps around showing the impact of your team and the constant struggle to measure how you’re improving. That’s where Unified Analytics comes into play.

The average organization can have ten or more monitoring or observability tools in their IT stack. These tools keep generating an overwhelming amount of noise. IT Ops, NOC and DevOps teams drown in this noise and can’t focus on real incidents until it’s too late. Your organization’s alerts don’t have to turn into an untameable tsunami with no end in sight—there’s a better way forward.

Skipping the retro shouldn’t be an option. Ditch the one-size-fits-all process to ensure that this important step is held at the end of every incident. Here’s how to make it happen.

Breaking down cloud management platforms and hybrid/multicloud management In our recent Whiskey and Wisdom session, we discussed how ITOps teams are coping with the evolution of cloud management. Whiskey and Wisdom is a monthly executive-only forum where IT operations leaders can network independently and discuss high-level AI operations and ITOps strategies with their industry peers.

Incidents provide an unparalleled opportunity to learn about your people, processes, and products under pressure. In this post, we’ll tell you how to ensure your team isn’t letting these opportunities for learning go to waste.

Budgets in IT departments are tight these days, so proving a return on investment is essential for justifying or expanding a project. The good news is that automation saves money by reducing the amount of human effort required. It is similar to investing in a robot vacuum cleaner. Despite the upfront cost, you save time (and money) by not having humans do the vacuuming. Reporting the value delivered by an automation program can be challenging since the value depends heavily on what is being automated.

We’ve all been in the situation before: it’s Friday at 5 PM and the only on-call engineer available to handle incidents is about to hit the slopes. Unfortunately, at that very moment, a customer reports to support that they are unable to access the company’s ecommerce website to complete a purchase. Internal monitoring systems seem quiet and services appear available on internal health dashboards.

You're probably aware that downtime is expensive—but do you know how expensive it is? The short answer is—very. According to the Ponemon Institute, outages cost organizations an average of $9,000 per minute (or $540,000 per hour). That's why companies of all sizes are investing in incident management tools to reduce their downtime and improve the customer experience.

There’s a familiar saying: garbage in, garbage out. For ITOps, this directly applies to data engineering. BigPanda’s Area Vice President of Value and Adoption, Craig Ferrara, says the importance of data hygiene—putting good data in to get good data out—is the core of data engineering, and it requires ITOps to take a look at their data before integrating with an AIOps solution.