When our CEO and co-founder Tomer Levy delivered his “Observability is Broken” presentation at last year’s AWS re:Invent, he highlighted numerous challenges faced by today’s organizations as they seek to advance their observability practices. Of the six individual points that he noted, two specifically dealt with the current shortage of available engineering expertise, with another two focused on data overload.

Although the causes and solutions for incidents vary widely, most incidents follow a similar timeline from declaration to resolution. We call the period of time it takes to move from one phase or milestone of an incident to the next cycle time.

The SLA definition is - An SLA is a written contract outlining quantifiable service quality standards between a service provider and a client. Typically, it includes response times, uptime, and error reporting.

The Uptime Institute recently released its Annual Outage Analysis 2023 report. Overall, the report highlights the increasing costs, frequency, and duration of outages, the prominent role of cloud and digital services in outages, the shortcomings of service providers, and the need to address human error and management failures. It also underscores the ongoing challenges of handling failures in complex distributed architectures.

Before we dive into the nitty-gritty of incident management, let’s look a bit closer at the actual meaning of ‘incident.’ In the world of IT service management, the official definition for ‘incident’ is an “unplanned interruption to an IT service or reduction in the quality of an IT service.” Whether that means a slowdown in response time or a total system crash, you’re looking at an incident.

This week I’ve been reading through the recent judgment from the Swedish FSA on the Swedbank outage. If you’re unfamiliar with this story, Swedbank had a major outage in April 2022 that was caused by an unapproved change to their IT systems. It temporarily left nearly a million customers with incorrect balances, many of whom were unable to meet payments.

It feels great writing this. It's hard to believe that we have been working on Spike.sh full-time for 3 years now. It's been the most rewarding experience of my life. A big thank you to all of our users and your constant feedback, which has only made Spike.sh better month on month. We are - Over the years, we have always kept our heads down and built. During this entire process, we have learnt a huge deal of things when it comes to incidents and how they are being managed.

In our previous blogs on Capturing Application State and using Ephemeral Containers for Debugging Kubernetes, we discussed the value of being able to deploy specific tools to gather diagnostics for later analysis, while also providing the responder to the incident the means to resolve infrastructure or application issues.

Legacy systems and point solutions are part of any business. And while they have their history and benefits, it’s critical to find a balance for your organization. IT teams have been acclimated to disparate event management and monitoring tools. Now, with massive and rapidly increasing data flow, this disconnect is slowing and paralyzing IT teams.

The threat and security landscape is becoming increasingly cluttered. As incidents increase, so do alerts and notifications, leading to too many alerts and too few hours to address them. Many businesses work remote and with the ever-present smartphones, we are always on the go. Yet it is essential that security teams receive and prioritize meaningful threats, but that task is easier said than done.

The PagerDuty REST API provides 200+ endpoints for users to programmatically access objects and workflows in the PagerDuty platform. Teams leverage these APIs to streamline creating and managing users, teams, services and other components for their environment. Up until now, access to the REST API has been authorized and authenticated via API Keys.

Site Reliability Engineering is a process of automating IT infrastructure functions, including system management and application monitoring using software tools. It is used by businesses to guarantee that their software applications are reliable even when they receive frequent upgrades from development teams. SRE allows engineers or operations teams to automate the activities that are traditionally performed by operations teams manually to manage production systems and handle issues.

“By failing to prepare, you are preparing to fail. Preparation prior to a breach is critical to reducing recovery time and costs.” (RSAConference) For 83% of companies, a cyber incident is just a matter of time (IBM). And when it does happen, it will cost the organization millions, coming in at a global average of $4.35 million per breach. The damage isn’t only financial, nor solely related to customer loyalty and brand equity.

The number of tools used by distributed teams to manage incidents has multiplied over the years, leading to a valley of tool sprawl. Throw in manual processes and you’ve got too much toil and multiple points of failure. Maintaining disparate tools and systems isn’t just unwieldy, it’s expensive. Our latest capabilities add to the PagerDuty Operations Cloud to make it easier than ever for teams to consolidate their incident management stack.

Over the past year, Spike.sh social media activity has been null. As a bunch of shy nerds in a small team working remotely across the world, we really never bothered with social media and our presence on it. We always kept our heads low and maneuvered around it. But no more. As of today, we are coming back on social media channels like LinkedIn, Twitter, and Reddit as well.

Saving time and money is always important, but these days, it’s a mission-critical business imperative. At PagerDuty, we help organizations realize transformational gains in efficiency that drive both immediate financial impact and long-term business success. PagerDuty delivers clear value for any organization at any stage of operational maturity. But you don’t have to take our word for it – the real-life experiences of our customers speak volumes.

Helpdesks serve as the initial line of defense for IT incidents, responsible for facilitating incident management, including logging, categorizing, and prioritizing incidents. In the event of a major incident, the helpdesk plays a crucial role in escalating the incident to the appropriate major incident management (MIM) team. The success of this process relies on the expertise of the helpdesk staff in providing situational context to expedite resolution.

Here’s a crazy question: why do we still require a human to manually declare an incident for the things that we know are incidents? If we have enough confidence to build SLOs and high-severity alert routes for these specific scenarios, why are we still asking a human to confirm it’s an incident and get the assembly process in motion? Isn’t that just another button to push when we could be problem solving instead?

Observability has become a critical aspect of modern software development and operations, allowing organizations to gain insights into the health and performance of their applications and systems. One of the key decisions when implementing observability is choosing between commercial or open-source tools. We spoke to several professionals who shared their experiences and insights on this topic, shedding light on the pros and cons of each approach.

In the current fast-paced world, businesses are seeking methods to increase their efficiency and simplify their processes. But, there are times when teams are unaware of an issue at the initial stage, leading to a bad customer experience. For example, you are a part of the Infrastructure team, where your primary responsibility is to check resources and notify when they reach their maximum capacity. Let's say due to an anomalous traffic load, our resource CPU utilization goes above 90%.

Incident response tools are software applications or platforms designed to assist security teams in identifying, managing, and resolving cybersecurity incidents. Incident response is a crucial part of an organization’s cybersecurity strategy, making it possible to detect threats, analyze vulnerabilities, respond to attacks, and recover from security breaches. Incident response tools are vital for safeguarding organizations against evolving cyber threats.

Sony Interactive Entertainment (SIE) is a multinational video game and digital entertainment company owned by global conglomerate Sony. SIE primarily operates the PlayStation brand of video game consoles and products.

In the world of software engineering, the choice of programming languages, frameworks, and technologies is constantly evolving. As a result, hiring engineers who have experience in different tech stacks has become a common practice for many companies. However, this practice also raises questions and concerns about the potential challenges and advantages of hiring engineers who work in predominantly different stacks.

Learning from incidents has become something of a hot topic within the software industry, and for good reason. Analyzing mistakes and mishaps can help organizations avoid similar issues in the future, leading to improved operations and increased safety. But too often we treat learning from incidents as the end goal, rather than a means to achieving greater business success. The goal is not for our organisations to learn from incidents: it’s for them to be better, more successful businesses.

Although some organizations may hesitate to publicly announce when they have an incident — afraid that acknowledging outages will scare customers away — the opposite is often true. When you proactively communicate with your customers, even during bad times, you have the opportunity to not only build trust but also buy grace during the incident.

When we detect a problem with your site we can notify you via mail, a slack message, a webhook, or any of our other notifications channels. For most of our users this is enough, but those work in larger teams often need more flexibility. Today, we are launching our Opsgenie integration, a modern incident management platform.

As the demand for instant innovation and real-time delivery of mission-critical processes continues to grow, your organization risks falling behind if it can’t adapt to an automation-centric strategy. To be successful, managers have to loosen the reigns and enable teams to automate their DevOps processes. Automating DevOps processes isn’t an all-or-nothing decision, and implementing automation processes slowly can let teams adapt to the changing environment and let go, little by little.

In any operation or activity, unforeseen happenings can derail progress. The job of a good manager is to try their best to make the hitherto unforeseen visible and planned for. It’s all too easy to find yourself reacting to occurrences that can throw you and the company into turmoil, with frantic fixing on the back foot being the result. The best managers can make it look like they don’t do much.

The FDNY EMS Command responds to more than 4,000 calls per day. They range from car accidents to building fires to cats stuck in trees, and responses vary accordingly. Sometimes they might take hours, sometimes they take just a few minutes. With such unpredictable conditions, the FDNY focuses on improving what they call “response time.” That’s the amount of time between a 911 call being made and emergency responders arriving on the scene. This might sound familiar.

How often have you heard the phrase “trust is earned” in life? While well-meaning, I think this can actually lead to some strange behaviour at work, especially when you’re on a fast growing team. Startups experience a lot of chaos and unknowns your teams need to navigate, so it’s vital to know you can trust the people around you. As you grow, how you set expectations around trust as people join your team can impact your ability to hire, onboard, ship and ultimately, survive.

As more and more teams transition to remote work, collaboration tools like Slack have become increasingly popular. Slack's chat-based communication platform makes it easy to keep teams connected and informed, but it can also create challenges when it comes to managing support channels. In this post, we'll explore different approaches to building a Slack-based support system and provide some tips for success.

An IT incident is an unplanned disruption that negatively impacts an IT service. As the importance of IT to the business has increased, the impact of IT incidents has become greater. IT incidents can result in revenue loss, loss of employee productivity, SLA financial penalties, government fines, and more. An effective IT incident management strategy is now essential in every organization. For a business like Amazon whose entire business relies on IT, a single second of slowness can cost over $15,000.

Tracking incident metrics can help you discover patterns in the causes and costs of incidents and help you understand brittle parts of your organization. We've seen them help teams zero in on things like: But it can be intimidating to get started. Do you really need metrics if you're a small team or just beginning to formalize your incident management program? I say yes. The key is to start with something manageable and grow.

Eliminating errors and streamlining the incident management process are top priorities for many ITOps, NOC, SRE, and DevOps teams. With organizations using multiple tools in their IT stack, manually finding the right information at the right time becomes crucial during incident triage. By automating tasks and workflows, businesses can eliminate manual tasks that are time-consuming, repetitive, and prone to mistakes.

In our previous blog, we discussed the difficulty in capturing all relevant diagnostics during an incident before a “band-aid” fix is applied. The most common, concrete example of this is an application running in a container and the container is redeployed—perhaps to a prior version or the same version—simply to solve the immediate issue.

With the complex and steadfast growth of IT service delivery processes, organizations and their internal teams have come to rely on several tools in their toolbox to deliver best-in-class products and services. The use of AIOps, AI/ML, and overall automation has shaped modern delivery methods, but what we call this process, and how we grow to advance it, has yet to find a definition that’s universally recognized.

We have to come clean. During KubeCon, we experienced an incident that we weren’t ready to discuss until now. This incident caused quite a disruption and, had it been left unresolved, would have had a massive snowball effect. At the time, we didn’t want to raise any alarms, so we kept it quiet while our team rallied to resolve it. And to be honest, most folks probably didn’t even realize that it happened since we moved so quickly.

April was a month to remember at incident.io. Not only did we attend our second conference ever with KubeCon in Amsterdam, but we also very subtly released our brand-new Status Pages product. OK, it probably wasn't subtle. Both moments required months of preparation, feedback loops, iteration, and so much more behind-the-scenes work to get right. So if you ran into us at KubeCon, thank you for stopping by and meeting with our team.

In today’s fast-paced digital landscape, businesses heavily rely on their IT departments to ensure smooth operations and deliver exceptional customer experiences. When it comes to IT support, one critical metric stands out: response time. A prompt and efficient response can be the difference between a satisfied customer and a frustrated one. In this blog post, we will explore strategies to improve IT response times, enhance customer satisfaction, and optimize overall productivity.