Across the business landscape, 2023 has been called the “year of efficiency.” Organizations have had to deliver more growth and innovation, but with tighter budgets and headcount than in prior years. CIOs have needed to build strategies to mitigate the risk of operational failure and protect their brand’s customer experience.

Operational resilience is an increasing area of focus and scrutiny for regulators of the banking and financial services industry. In the European Union, the Digital Operational Resilience Act (DORA) looms on the near horizon - with equivalent regulatory frameworks slowly but surely rolling out across the globe.

In the modern day IT landscape, service reliability is of the utmost importance. Status pages serve as crucial interfaces, communicating any interruptions or issues to stakeholders. While several options are available, two notable status page providers stand out: StatusCast and Status.Io. Here we take a dive into the various aspects of status pages and incident management for each status page service.

October might be a spooky month, but we’re doing our best to make incidents less scary. We released a number of updates this month that focus on two main areas: Let’s jump in.

System outages: they are an inevitable problem that every single IT team will encounter at some point. Whether they come about due to technical issues, act-of-god natural disasters, or simply random human error, system outages happen to the best of us. Though the cause of system outages is not always in your control, you can control your team’s processes for response and resolution.

Navigating the complex world of ServiceNow’s Configuration Management Database (CMDB) can feel overwhelming. You might find yourself grappling with understanding the foundational aspects of the CMDB, or maybe you’re seeking effective ways to utilize and integrate it seamlessly into your IT processes. You want to extract the maximum value from your ServiceNow CMDB but need help figuring out how to start.

Many PagerDuty customers create their own apps to help them manage their PagerDuty environments. Teams might have any number of workflows that might benefit from a custom application. A PagerDuty admin might want to be able to load CSV files with new users and their contact information into PagerDuty when new teams join the platform, or load new services before they are released to production.

As an engineer, you're probably familiar with version control systems like Git that let you track changes to your codebase. But are you using one of the most useful features of Git pull requests? If not, you're missing out. Pull requests are one of the best ways to collaborate on projects and create better code. In this article, we'll go over the pull request meaning, why you should be using them, and how to create your own pull requests.📑 What is incident management software?

Are you tired of sifting through a sea of IT events and alerts? Or perhaps you’ve found yourself overwhelmed by the volume of data flooding your monitoring systems and challenged to identify the incident root cause. There’s a better way to manage the chaos: using AIOps to unite disparate tools, data, and teams for event correlation.

ITOps is at a crossroads: Teams struggle to manage a high volume of alerts and coordinate between different tools and teams. Teams also must balance cloud technologies’ agility and on-premise solutions’ stability. The sheer speed of today’s IT demands both flexibility and visibility in development and harmonized tech stacks.

Escalation policies are essential for making sure that incidents are quickly addressed and resolved. They provide a systematic approach to automate alerts, guaranteeing that no incident goes unnoticed. Let’s get you started, shall we? An escalation policy is a way to automate alerts and assure that incidents are never missed. The first point of contact for an incident is through an alert that is sent according to the escalation policy.

Today’s fast-paced digital world can lead to system breakdown and disruptions that strain organizational resources. What truly distinguishes successful organizations is their response when problems occur. Incident management serves this function. At its core, incident management involves teams managing unexpected disruptions quickly with minimal impact to users or business operations. The process is like a safety net that prevents further problems from developing into trust issues.

Build or buy? An age-old decision that gets made dozens of times a year. It’s quite possibly one of the most important decisions you make as an company. It impacts roadmaps, productivity, team structure, and customer satisfaction (you know, just a few little things). There are a lot of factors to consider, one of the most prominent being cost. So, what exactly are the costs you need to consider when building your own incident management solution?

Building a culture of incident response is not just about solving problems; it is about creating stronger teams, empowering individuals, and fostering a more resilient and thriving workplace. How do you achieve this culture and improve your incident management processes? Let’s dive in;

Of course, one expects an alerting solution to be reliable. This is important because a missed alert can have a significant impact on the business. It is about IT uptime, disruptions in production or other critical system conditions. Business processes, production workflows and therefore money, the reputation of the company or even the health of the employees are at stake. But what does reliable alerting actually mean and how is it achieved?

It is now the de facto standard for companies to operate across numerous regions and cloud-accounts. The reasons for this vary, and depending on where you sit in the organization, these reasons may be more or less apparent to you.

Network Operations might never be the same. But then again, why would anyone want it to be? The power of automation and orchestration can bring incredible value to the Network Operations Center (NOC), including the business-critical call to get proactive and ahead of the incidence response and management game. It’s more than a towering volume of events – it’s the complexities involved, too.

This post introduces the Pandora FMS monitoring solution and how to integrate it with ilert to establish reliable alerting. The guest post is written by Sancho Lerena, the CEO of Pandora FMS.

Dear Current and Future Moogsoft Customers, I am happy to announce that Dell Technologies has acquired Moogsoft on September 17, 2023. This is good news for existing and future Moogsoft and Dell customers. Earlier this year Moogsoft embarked upon raising capital to accelerate growth.

When outages cost you tens of thousands of dollars each minute, pinpointing the source of disruptions as quickly as possible becomes mission-critical. This is not a time for finger-pointing and hastily assembled war rooms searching for that needle in the haystack. You need simple, intelligent, trustworthy Internet health information to expedite your incident detection.

At PagerDuty, our purpose is to empower teams with the time and efficiency to build the future. That means that our own teams are constantly building and relentlessly innovating to help organizations drive transformative change in the way they operate.

Data is both a challenge and an asset for IT professionals, who rely on IT Operations Analytics (ITOA) to guide them towards operational excellence, system reliability, and swift incident resolution. So whether you’re seeking clarity on understanding what ITOA is and its connection to related technologies, are contemplating how to use it within your organization, or are curious about its enhanced efficiency and cost savings benefits, we’ve got you covered.

The incidents page, the most visited page on Zenduty, has an all-new look and feel! It's been completely redesigned from the ground up to be faster, easier to use, and more visually appealing. The Incidents list now dedicates more space for important information, such as the title, date, priority, and more. The UI is also more polished, shaving off whitespace where unnecessary. The avatars have been redesigned with more pastel shades, resulting in an overall design far more soothing to the eye.

Maybe you’re still using monolithic applications, built and refined over many years. You understand that shifting to microservices or containerized architectures is a huge and daunting task. You’re probably grappling with the limitations of legacy systems—maybe they’re slow, tough to update, or can’t scale as you’d like. And you’re likely using more traditional IT monitoring tools or even some cloud observability tools.

As more organizations embrace containerized applications, Kubernetes has emerged as the leading platform for orchestrating these containers. However, its complexity, combined with the inevitable reality of IT incidents, demands a well-defined strategy for managing disruptions. This article introduces Kubernetes incident management, describes common Kubernetes errors, and provides practical guidance to efficiently handle incidents.

Prometheus Alertmanager is a powerful tool designed to handle various alerts generated by Prometheus. It plays a vital role in the overall monitoring ecosystem, acting as a centralized hub for managing alert notifications. With Prometheus Alertmanager and its robust notification management capabilities, you can efficiently define alert routing and notification policies. This empowers you to take timely actions and mitigate potential issues before they impact your service availability.

As a business owner or manager, you understand the importance of efficient operations and effective communication, particularly after hours. You want to equip your on-call engineers with all the information they need to resolve a ticket when not at their desk. If you are using ConnectWise to manage your service tickets – here is some great addition to help with your after hours alerting.

IT operations teams are challenged to keep pace with the rapid speed of digital transformation. As companies use more cloud-based apps, increase agile deployments, and develop new microservices-based applications, they add layers and complexity to their technology stacks, making life increasingly challenging for ITOps performance.

When it comes to managing services effectively, terms like SLA, SLO, and SLI are often thrown around like confetti at a parade. They’re in meetings, in documents, and even in casual office conversations. But if you’re new to the field or simply haven’t had the chance to dig into these acronyms, they can feel like a bewildering alphabet soup. And they can’t be missing on an uptime monitoring blog such as ours! So, what do these terms really mean?

You've just made it through a particularly tough incident. It was a short outage affecting a subset of customers, so not exactly the end of the world, but bad enough that it involved multiple people across a number of teams to resolve. Either way, the incident was well managed, and the dust has settled. Now what? Most guidance would say that putting together a post-mortem document is a good idea, given the severity of the incident. You've also done this, so what's next?

Maintaining your Configuration Management Database’s (CMDB) accuracy, keeping it fully updated, and improving its performance is a frustrating and elusive goal for ITOps and IT leaders. Aiming for this ‘golden’ CMDB standard can feel like running on a treadmill where you’re putting in a lot of work, but remain as distant as ever from your goal. Can IT leaders ever catch up?

DevOps practices in software development have revolutionized the way updates are released. However, many companies entrenched in ITIL practices find it challenging to seamlessly integrate with the DevOps practice of Continuous Integration and Continuous Delivery/Deployment (CI/CD). This is because ITIL focuses on stability, which suits older systems, while DevOps is ideal for modern setups with its agile, automated practices.

Once upon a time, in the bustling city of DataVille, lived a team of dedicated IT professionals tirelessly working to maintain the city’s digital heartbeat. Their mission was to ensure the smooth operation of their city’s digital infrastructure, which was not limited to the daytime operations but extended beyond business hours. They were the unsung heroes, the guardians of the city’s data. Their tool of choice? Grafana, a powerful open-source platform for observability.

In the realm of hospitals and healthcare organizations, the term “HCAHPS survey” is a recurrent presence: Hospital Administrator A: “The latest HCAHPS survey results just came out, and patients seem satisfied with…” Hospital Administrator B: “Some of our past patients participated in the HCAHPS survey, but they expressed disappointment with…” You might be left wondering, “What exactly is the HCAHPS survey?” Allow me to elucidate.

The tech industry is booming, and there are many different career paths. But, two of the most popular and in-demand roles are Software Engineering and Site Reliability Engineering (SRE). Site Reliability Engineering (SRE) blends elements of software engineering with IT operations, focusing on reliability. On the other hand, SWE Software Engineering involves designing, developing, testing, and deploying software applications.

Our October update brings a new layout in the web portal, new additional cross-references from Signl details to linked entities, and improved grouping options for conditions in the distribution rules. As always, all the details are in this blog article.

Mean Time Between Failures (MTBF) measures the average duration between repairable failures of a system or product. MTBF helps us anticipate how likely a system, application or service will fail within a specific period or how often a particular type of failure may occur. In short, MTBF is a vital incident metric that indicates product or service availability (i.e. uptime) and reliability.

In today’s fast-paced, digital-first landscape, delivering exceptional customer experience is paramount to business success. For customer service teams, that means maintaining service level agreements (SLAs) and ensuring swift responses to customer issues that can make or break your company’s reputation. Fortunately, PagerDuty has improved the way companies handle customer service teams and has built applications into ServiceNow’s CSM platform.

Almost every study examining the hourly cost of outages invariably leads to a clear and undeniable conclusion: outages are expensive. According to a 2016 study, the average cost of downtime was estimated at approximately $9,000 per minute. In a more recent study, 61% of respondents stated that outages cost them at least $100,000, with 32% indicating costs of at least $500,000 and 21% reporting expenses of at least $1 million per hour of downtime.

No one wants to be on the receiving end of the blame game—especially in the wake of a major incident. Sure, you know you were the one who made the final change that caused the incident. And hopefully, it was a small one that didn’t cause any SEV-1s. Still, the weight of knowing you caused something bad should be enough, right? Unfortunately, sometimes fingers get pointed, your name gets called, and suddenly, everyone knows that you’re the person who created more work for everyone.

For most businesses, managing major incidents can be intimidating. With a swarm of information coming from different directions, keeping things organized and maintaining clear, effective communication is tough. It only gets worse when there's no defined process to follow. This disorganization confuses everyone, delays responses, and increases the incident escalation rate. Enter the incident commander (IC).

I was visiting Queenstown, New Zealand last week amidst the horrible floods which quickly escalated. As an incident responder myself, I was amazed at the operations and how fast responders on the ground acted in evacuating and clearing the grounds. Over 100 people were evacuated in the middle of the night with zero casualties. A commendable job. Here are some observations I made and what we can learn as incident responders ourselves..

You’re in the incident channel rocking yet another incident. Comms are flowing, resolution is in sight, the team is grinding, and you’re feeling good. Then…

In the ever-evolving world of software engineering, the landscape is constantly shifting. New technologies emerge, best practices evolve, and how we build and run software continues to change. However, when it comes to incident alerting, it often feels like we're stuck in the past.

IT leaders are thrilled about the potential of Generative AI for IT Operations. But they also want to know how it works, why it works, and what it will do for them before taking the leap and adopting this new technology. Allow me to share my perspective on the hype and the truth behind Generative AI. I’m the Field CTO for BigPanda, Operational Intelligence and Automation driven by AIOps.