Operations | Monitoring | ITSM | DevOps | Cloud

This post highlights some of the features and improvements that we have released in the last two months. If you want to submit your own ideas or vote on existing feature requests, you can now use our public roadmap at roadmap.ilert.com.

If you’re in IT operations or manage NOC, SRE, and DevOps teams, chances are your IT environment is growing complex for you and your teams to manage. Any enterprise, large or small, around the globe, is continuously changing its IT stack due to evolving business requirements and significant industry trends. But digital transformation, hybrid infrastructure, DevOps adoption, and continuous integration and continuous delivery (CI/CD) pipelines are all causing major headaches.

Have you ever quickly worked through an issue with your team and later thought, “Huh. That probably should have been an incident.” It happened to us just a few weeks back. After one of our engineers surfaced a failed build, a few folks chimed in to problem solve and within 30 minutes things were up and running like normal. But we probably should have declared an incident.

Alert fatigue is a serious issue that affects numerous professions, especially in the IT industry. It can lead to neglecting critical events and delaying response times. IT teams need to continuously monitor their systems and applications to avert possible downtime and keep operations running smoothly. However a high number of incoming alerts inundating these teams can make them less responsive. The ramifications of such disregard can severely affect the efficiency and dependability of IT teams.

Catastrophic failures — such as a security breach or a complete outage leading to an unavailable product or service — are classified as Sev0 incidents. On a severity scale of 1–3, Sev0 is dire. It brings business to a complete standstill and may lead to loss of revenue and a damaged reputation. A Sev0 incident usually has no quick workaround; it requires a coordinated effort beyond the engineering team to diagnose, correct, and manage.

At a recent training session, Jeli spent a great deal of time covering incident retrospectives and what makes an incident worthy of studying. My colleague Ben Hartshorne asked a fascinating question, which I’ll paraphrase here: That caught me by surprise. We had a great discussion, and it made me consider approaches I hadn’t before.

Incident management is a team effort. While it's true that incident management should be seen as a company-wide effort, and you should empower all teams to declare incidents, this differs from the team effort I'm referring to here. No, incident management is a team effort in the sense that no one tool can do it all, not even incident.io. We covered as much when we discussed why we integrate with tools that can be seen as our competitors – and that’s OK!

Monitoring tools, also known as observability solutions, are designed to track the status of critical IT applications, networks, infrastructures, websites and more. The best IT monitoring tools quickly detect problems in resources and alert the right respondents to resolve critical issues. Response teams use observability solutions to gain real-time insights into resource availability, stability and performance.

Our status page product started last year as an experiment. We built a status page product in a hurry over weekends, and to our surprise, it gained a lot of traction. People were using it and giving us feedback, which helped us improve the product over time. And this year, we're thrilled to announce that we have great things planned for our status page product! The new revemped dashboard is part of a larger plan for our status page product. Here's a quick gist of the multiple releases.

Earlier this year, we experienced intermittent timeouts in our application while interacting with our database over a period of two weeks. Despite our best efforts, we couldn’t immediately identify a clear cause; there were no code changes that significantly altered our database usage, no sudden changes in traffic, and nothing alarming in our logs, traces, or dashboards. During that two-week period, we deployed 24 different performance and observability-focused changes to address the problem.

More data for data’s sake doesn’t help anyone. What organizations need is more information–actionable insight. With data coming from incoming streams of events and alerts, teams don’t have enough time to look at each one. And they struggle to parse and consolidate this data in order to figure out what they need to do next to resolve an incident.

We kicked off 2023 with a new team and a new product to build - Status Pages. We wanted to build a solution we could ship to customers as quickly as possible, while making sure that it’s reliable, fast and beautiful. Here’s how that process played out over the course of three months.

Incidents are expensive — and only getting more so. In fact, more than 98% of large companies and 47% of small- and medium-size companies say a single hour of downtime costs at least $100,000, according to the 11th annual Hourly Cost of Downtime Survey.

PagerDuty’s Global Event Orchestration is now generally available. Global Event Orchestration’s powerful decision engine enriches events, controls their routing, and triggers self-healing actions based on event data. Teams can use this functionality across any or all services within PagerDuty. This feature is a continued investment in Event Orchestration, demonstrating PagerDuty’s commitment to providing customers with best-in-class automation capabilities.

Clear and frequent communication carries considerable weight in today's era of hyper-competition among businesses—especially during incidents. Because of this, status pages have become the go-to choice for companies looking to prioritize trust, transparency, and clarity with their customers, even during downtime. Unfortunately, current status page solutions have made these communications particularly frustrating and stressful.

IT incidents are events which lead to a disruption or deviation from the regular operating standards of a computer system or network. They can be caused by various factors, including hardware or software failures, human error, or even deliberate external (cybersecurity) attacks. It begins with short delays, or services cutting out - for example, when a website or server is down, or access to data(bases) takes too long.

Automated Incident Management is the process of automating some or all these tasks through various means. Automated incident management can improve incident response time, reduce unnecessary work, such as when an issue is a minimal impact. AlertOps can help automate incident management by creating tickets in help desk systems, filtering and rules, and escalating alerts.

The renowned German manufacturing magazine “Factory Innovation” recently conducted a comprehensive practical test on four leading alarm notification software for industrial manufacturing in their latest issue (01/23). The four alarming systems that were evaluated include: the Alarm Control Center from Alarm IT Factory (a spin-off of Siemens AG), ALERT 4.0 from Micromedia, the Alarm and Information Portal (AIP) from VIDEC, and SIGNL4 from Derdack.

Communication carries more weight than ever before. Businesses are so much more connected to their customers given the number of mediums they can communicate through; Twitter, Instagram, Facebook, and even TikTok. Because of this, it's essential to prioritize these lines of communication throughout your day-to-day. Some might even say that over-communicating is the best way forward. Why? No one likes a company that appears simply like a black box with zero insight into what's happening.

Ready, set, go: when it comes to customer service, it's a race against the clock. Customers expect lightning-fast responses and complete solutions to their problems. But what happens when your help desk can't keep up with the pace? The answer is simple: frustration, dissatisfaction, and potentially lost clients. That's why measuring and improving Time to Resolution (TTR) is crucial. As a customer, there's nothing more irritating than dealing with a slow or ineffective help desk.

The average cost of an IT outage is $12,900—per minute. And when it comes to a “significant outage,” organizations reported the average overall cost was a whopping $1,477,800. On the latest podcast episode of That’s great IT, I spoke with Scott Lee, AVP for infrastructure and ITOps at Arch Mortgage Insurance Company, part of Arch Capital Group, about how organizations can best navigate IT outages.

I’m a big fan of monolithic architectures. Writing code is hard enough without each function call requiring a network request, and that’s before considering the investment in observability, RPC frameworks, and dev environments you need to be productive in a microservice environment.

At Vanta, our goal is to nurture a positive security culture in everything we do—which is especially critical given that helping our customers improve their security and compliance posture starts with our own. Employees are the key to our security resilience, so we strive to build and support a strong culture of incident response in tandem. Here’s what that means to us at Vanta.

In any high pressure situation, the key to fast action is preparedness. And that’s true when it comes to incidents, too. Documenting and training your team on your incident response processes is essential to ensuring a coordinated and efficient response effort. And training sessions, or game days, as they’re sometimes called, are one way to get everyone up to speed.

Today, PagerDuty launched a new AIOps solution to leverage the power of AI, provide built-in automation and build on the company’s foundation data model to transform modern operations for the enterprise. PagerDuty has long suppressed noise to help distributed development teams focus.

After a longer period without a major update, our April 2023 update has it in for it – promised!

Root cause analysis (RCA) is a systematic approach to defining symptoms, identifying contributing factors, and repairing faults when problems arise. The process can be applied to virtually any problem in any industry, from NASA’s Apollo 13 mission to everyday tech problems that happen within modern IT departments.

Four years ago tomorrow, our team rang the bell to open the NYSE for PagerDuty’s IPO. We spent two weeks traveling to meet hundreds of prospective investors in person, sustained by a diet of Cheetos and green M&Ms, sneaker-clad walks to meetings, and unwinding with bad karaoke. We’ve grown in many ways in our first four years as a public company. We have more than doubled the number of customers on the PagerDuty platform, and nearly tripled the number of users.

I see it daily in my role, IT organizations are paying for best-of-breed monitoring tools but struggle to tie the pieces together between these siloed systems. The wound of these silos is further punctured when incidents arise. Incidents are costly for so many reasons, like wasted company resources, potential revenue loss, customer satisfaction, employee burnout, etc. This is exactly why BigPanda exists, to apply AI to the complex problems IT operations, NOC, SRE, and DevOps teams face daily.

When it comes to time-sensitive events, instant, reliable communication is key. In the past, pagers were relied on for quick communications as they allowed people to communicate on the go and without access to a landline. But today, the availability of cellphones has made the portability of communication devices a standard feature, and communication technology has advanced significantly, begging the question – What is the use for pagers today?

When your incident response process is centered around a service catalog, responders are able to more quickly pinpoint the service or functionality that’s down, bring in the team or experts, and then get to solving the problem faster. Saving even a few minutes can have a big impact on decreasing the costs around incidents and outages, so having up-to-date service details at your fingertips can make all the difference.

Cattle, not pets is a DevOps phrase referring to servers that are disposable and automatically replaced (cattle) as opposed to indispensable and manually managed (pets). Local development environments should be treated the same way, and your tooling should make that as easy as possible. Here, I’ll walk through an example from one of my first projects at incident.io, where I reset my local environment a few times to keep us moving quickly.