On-call is a special working hour arrangement under employment law. It comes into effect when the employee is obliged to be contactable at least by phone, so they can start work in an emergency. On-call duty is generally counted as time specifically meant for work purposes. In practice, this means that employees are normally not allowed to work while on-call. However, there may be exceptions. For example, on-call employees may also work from home if they can be reached through their work device.

Not long ago, monitoring tools were just nice additions to have and did not have a lot of purposes. However, as technologies scaled up and became more complex, keeping track of all the systems and their health became a huge challenge. As more and more brands started offering new digital services and moved the existing platform, the competition skyrocketed and being on top of system health and proactively resolving potential incidents became crucial.

Not long ago, monitoring tools were just nice additions to have and did not have a lot of purposes. However, as technologies scaled up and became more complex, keeping track of all the systems and their health became a huge challenge. As more and more brands started offering new digital services and moved the existing platform, the competition skyrocketed and being on top of system health and proactively resolving potential incidents became crucial.

The ITIL definition of an incident is “an unplanned interruption to or a reduction in quality of an IT Service or unavailability of the service”. An incident could be caused by an asset that is not functioning properly or a network failure, or a human error.

It is getting spooky out there, folks! Every year on October 31, we don our spookiest (or silliest) garb, an evolution of old practices where people would dress up to ward off ghouls, goblins and all manner of things that go bump in the night. After all, people believed these pesky spirits stirred up trouble. While pieces of this spooky tradition persist, just a few other things have changed in the past 2,000 years. For starters, we are a digital society.

There is a famous quote that goes like this…‘For every minute spent organizing, an hour is earned.’ At least in the world of incident response, nothing is more apt than this. Digital infrastructure these days is made up of multiple services, an outage could result from either one impacted service or multiple impacted services. So it's essential to have a catalog of all the services along with the point of contact (service owner) responsible for maintaining it.

As usual, it’s been all systems go at incident.io this month. New joiners, new features and new swag (yes, you heard right!). But most excitingly, we launched our new podcast this week. We had a blast recording it - we hope you enjoy listening to it just as much. Here’s a round-up of some of this month's highlights…

DevOps and IT Operation teams employ the incident management process to respond to an unanticipated event or service outage and return the service to operational status. In the ITIL framework, it is a mechanism that links end-users and the IT department for more effective incident response. A robust incident management system in any company will allow the employee to raise a ticket detailing the issue he/she is facing.

Amazon Web Services (AWS) offers 100+ services, each focusing on a specific area of functionality. However, it can be challenging to pick the right services for the task and also to provision them. AWS Elastic Beanstalk, lets you easily deploy and manage applications without the need to learn about the underlying infrastructure that runs these applications.

We’re excited to announce a new set of updates and enhancements to the PagerDuty Operations Cloud. Recent development and app updates from the product team include Incident Response, PagerDuty® Process Automation, as well as Community & Advocacy Events updates. We continue to help customers further automate to optimize cloud operations and reduce the amount of issues escalated to other teams.

With BigPanda’s self-service Email Parser, extracting alert data from emails has never been more simple. In our latest video in the Fast track series, we explore the benefits of this tool. This parser is ideal for monitoring tools and systems that do not support REST API and or rely solely on email to generate and send alerts. So no matter what tools your organization utilizes, this feature can help you turn all of those alert emails into actionable incidents within BigPanda’s platform.

Before you can choose the proper tools for your organization, you have to understand its essential business processes. Once you know an essential business process, you can review software applications that will help make your organization more efficient and accurate. Unfortunately, many organizations do not understand their essential business processes. This makes it nearly impossible for them to streamline their organizations, which puts them at a disadvantage in the marketplace.

Looking into incident tracking? We explain what incident tracking is, how it’s done, and why it matters.

Event correlation is a powerful capability that can help reduce IT noise, detect incidents in real-time, and improve the performance of critical applications and services. Read on for a deep dive into event correlation as we explore everything from its origins to its current state-of-the-art techniques. We’ll also discuss how event correlation fits into the bigger picture of integrated service management.

Just before Halloween 2021, Roblox engineers experienced a horror story: a service outage that also took down critical monitoring systems. It seemed like the issue was a hardware problem, but it wasn’t. Users were frustrated, and the clock was ticking. After three full days of downtime, service was finally restored on Halloween day. While the incident itself was an IT nightmare, Roblox’s detailed technical post-mortem several months later was an excellent way to bounce back.

Many organizations have been digitally transforming their operations and the majority of them are moving to the cloud. With this transformation, data teams have to analyze ever larger and more complex data sets to allow downstream teams to make faster and more accurate decisions on a daily basis. Consequently, most organizations need to work with: customer data, product data, usage data, advertising data, and financial data.

Playbooks aim to be a super-powered checklist for repetitive tasks. Before you can get to the “super-powered checklist,” though, you need to identify the process that you’ll use to build your first playbook and create a structured process as a Playbook checklist. Let’s go on that journey today.

We have released another update for Enterprise Alert 9 (version 9.4) which enhances the cloud bridge and MS Teams integrations. This will help you to setup scenarios where you wish to active your Enterprise Alert remote actions from with the Signl4 app as well as allowing for using a proxy to configure the MS Teams integration. Read all details in this article.

Healthcare around the world is constantly evolving. The amount of data being generated daily from every appointment and interaction, no matter how small or large, needs to be processed and analyzed in order to improve patient outcomes. The data must be accurate, stored, accessible and secure. Without a core infrastructure of smart IT, any outcomes are extremely challenging to generate, and data must be available in seconds for doctors to make life-saving decisions. The bottom line?

Just consider how much of your day relies on online digital technologies. Perhaps you hopped on an app to pre-order your morning coffee and then logged onto a platform to book a car to work. Or, perhaps you stayed home to work, using digital tools to connect with your colleagues and exchange information.

The MIM Expo (Major Incident Management) always delivers, and this year’s recent gathering was no exception. At this annual event, we always get a unique opportunity to hear about what’s top of mind with major incidents and SRE professionals from all the world.

One of my favourite features in incident.io is Decision Flows. With it, you can create a series of questions which eventually lead to a decision based on what you’ve answered. You can pull up this flow during an incident and it’ll guide you through the questions. It’s like having an experienced on-caller calmly guide you through what to do when a crisis hits. This is complementary to incident.io’s Workflows feature.

FireHydrant has partnered with incredible companies to transform incident response inside their organizations, but our goal has always been to support the full incident lifecycle. That’s because we know that investing in good incident management can kickstart your reliability efforts when it includes both a streamlined incident response process that helps you recover faster and the ability to learn from incidents and then feed those insights back into your system.

NOC, or network operation center, processes have been set in stone for decades. But it’s time for some of these processes to evolve. Digital transformation and the cloud era have led to the rise of DevOps, and with it, service ownership. Service ownership means that developers take responsibility for supporting the software they deliver at every stage of the life cycle. This brings development teams closer to their customers, the business, and the value they deliver.

NOC, or network operation center, processes have been set in stone for decades. But it’s time for some of these processes to evolve. Digital transformation and the cloud era have led to the rise of DevOps, and with it, service ownership. Service ownership means that developers take responsibility for supporting the software they deliver at every stage of the life cycle. This brings development teams closer to their customers, the business, and the value they deliver.

As a busy executive, taking time to attend an event and listen to sessions is a luxury. And yet, I know that many of my best breakthrough ideas on how to lead my teams have come from taking those moments to tune into new ideas. The challenge is figuring out where the hidden nuggets of wisdom are buried in a mountain of content.

We explain what infrastructure monitoring is, how it works, how to overcome the challenges in complex systems, best practices for monitoring, and the tools you need.

You’ve probably heard the terms ITIL and ITSM, but the distinction between the two can be a little unclear. Throw incident management into the mix, and the whole thing can feel pretty confusing. This article aims to explain what they are, the differences between the three, and importantly how they fit together. First, let’s establish what each of the terms actually mean.

We’re fortunate enough to speak to a huge number of companies about their incident management processes. In doing so, we’ve noticed an emergent trend in how modern companies are using software to support their incident management processes, and a common set of challenges faced by them too.

Status pages are a clever solution to bundle all your services, and see the status of them at one sight. We at iLert took this one step further: why not build your status page as code using Terraform? We want to show you how we make it possible, and how you can set it up for your own infrastructure - a real SaC solution.

In our inaugural State of Availability Report, we discovered that not only do metrics matter but the way we use them also does. Our research found that teams with fewer KPIs were more likely to meet their Service Level Agreements (SLAs) and provide their customers with higher levels of availability. The problem with having too many KPIs is that they cause information overload and noise.

Maintaining IT infrastructure is a consistent challenge for system administrators, site reliability engineers (SREs), supporting developers, and technicians. Several factors can impact system performance, cause outages, or impact customer experience. On top of that, not all incidents are created equal. The impacts and severity of a system outage affecting 10% of your users are different from an outage impacting 90%.

With the announcement of their Fall 22’ Review awards, PagerDuty has been named a G2 Leader for Incident Management Software for the sixth quarter in a row. We owe a special thank you to our customers who have consistently given PagerDuty high satisfaction scores that take into account their likelihood to recommend PagerDuty, our ability to meet their requirements, and the overall ease they’ve found in doing business with us.

Reliability and availability have different meanings when it comes to software. What are the differences and what is the importance of each?

Summer has passed and it’s time for fall - cue transitioning leaves, cozy blankets, and all the pumpkin-themed things your heart could ever desire. As we move into the new season, we are excited to announce our fall product releases across Moogsoft Cloud that enable engineers to detect incidents earlier, resolve them faster, and work as a team across the entire lifecycle. Moogsoft’s Fall product updates enable you to: … and so much more! Read on for deeper details.

Event correlation and AIOps go hand-in-hand. Event correlation is the process of identifying patterns in data that may indicate a problem or opportunity.

As organizations grow and become more complex, so does the need to monitor and troubleshoot issues across the entire IT infrastructure. Event correlation is a powerful technique that can help make sense of the huge volume of alert data generated by monitoring systems and identify problems as they occur. In this blog, we’ll look at event types, use cases for event correlation and approaches that organizations can use to get the most out of this valuable tool.

Like most tech companies, we use an on-call rota and various alerting tools. We do this to respond to incidents before they’re reported. Proactively identifying issues and communicating to customers helps us provide great experiences and fosters trust. Internally, we’ve been using these alerting tools in tandem with our auto-create incidents feature. We’ve found that it’s made responding to the pager much smoother - it’s one less thing to do when you get paged at 2am.

More than 16 months ago we provided a solution to integrate HashiCorp Consul with our alerting and on-call management platform by using consul-alerts - a dedicated application that allows for communication between a deployed Consul instance and an existing iLert account. ‍ With more code infrastructure being moved to the cloud to ensure better security and availability, we too have ensured that our service integrates with the HashiCorp Cloud Platform (HCP).

Businesses and communities are experiencing a growing number of disruptions from threats like severe weather, civil unrest, theft and vandalism, pandemics, and cyber-attacks. These disruptions have left many organisations concerned about the safety of their people and operations.

Today I am excited to announce we have officially shipped PagerTree 4.0! Here are the highlights: This effort has been a year and half in development and I sincerely want to thank each and every one of our customers for the constructive feedback, ideas, and countless hours on Zoom calls. Without you this journey wouldn’t be possible. We are excited to get this major release shipped, just in time for the holidays. You can check out the full details of the upgrade below.

So you’ve decided to take advantage of Site Reliability Engineering by hiring SREs for your company. Now, you have a second decision to make: Exactly how many SREs to hire. Do you need just one or two SREs? Or should you build a sprawling SRE team, with a dozen or more SREs on hand to support your organization’s reliability needs? The answers to these questions will, of course, vary; every business’s needs are different.

We’ve been building incident.io for 12 months and thought it would be a good time to share the constellation of tools that we’re using to power our customer experience.

Over the past few years, artificial intelligence for IT Operations (AIOps) has risen in popularity within the technology landscape. It’s become a buzzword in the marketing world, and while there are many ways to define AIOps, the best way to start thinking about it is through the lens of outcomes, correlation and strategy—it’s all about the data.

When critical incidents happen — which they inevitably do 😅 — and you’re in the middle of trying to figure out what the best thing to do is, it can feel comforting to know that you’ve got a pre-prepared list of instructions to follow, commonly known as an “incident response plan”: In theory this sounds quite simple, and a typical flow you might envision is: It might be tempting to think that the hardest part of running incidents is finding or writing a checkl

Hey folks, We’re back with another feature update for all our customers! We have recently gone live with the incident watchers feature which nests within an incident details page. This blog will outline how you can access the feature, its primary functionalities and how we foresee it helping improve your incident management process. Note: This feature will be available to pro, premium and enterprise plan users only.

Engineers have been managing incidents for as long as they’ve been building software, but the idea of incident management as a strategic practice in its own right is still finding its place. We’re starting to see big shifts in that area, though — more companies are dedicating headcount, resources, and tools to help them better prepare for, respond to, and learn from their incidents.

What are the keys to building software development security into the early stages of product development? And what are the costs of ignoring security? In this article, xMatters Product Manager Kit Brown-Watts provides his insights on the matter. Every investment decision comes with trade-offs, usually in the form of cost, quality, or speed. The CQS Matrix, as I like to call it, captures the dilemma most product people face.

Are you looking to start an SRE team or add to your existing team? We explain the SRE hiring process and how to find and evaluate an SRE.

Depending on your business, MTTR stands for mean time to repair or mean time to recovery – but it can also mean resolution, resolve, or restore. No matter how you define it, the basic measurement is the same: it’s the time it takes from when something goes down to when it is back and fully functional. This includes everything from finding the problem to fixing it. For ITOps teams, keeping MTTR to an absolute minimum is crucial.

Recently, technology roles have become more generalized—cloud computing, for instance, requires a broader knowledge of technologies like storage and network. As technology has continued to evolve over the decades, many job positions have blurred into many roles or even morphed into new roles with new responsibilities.

At incident.io, our number one priority in engineering is pace. The faster we can build great product, the more feedback we can get and the more value we can deliver for our customers. But pace is a funny thing. If you optimise for pace over a single month, you’ll quickly find yourself slowed down by the weight of your past mistakes.

Spring Cloud and Kubernetes both complement each other to build a cloud native platform and run microservices on the Kubernetes containers. Kubernetes provides many features which are similar to Spring Cloud and Spring Config Server features. Spring framework has been around for many years. Even today, many organizations prefer to go with Spring libraries because it provides many features. It's a great deal when developers have total control over cloud configuration along with business logic source code.

Continuous availability and unceasing innovation are prerequisites for today’s digital businesses. So it makes sense that business leaders invest heavily in teams and tools to monitor digital apps and services. In theory, these tools should also free up time for engineers to push new functionalities that wow customers. But do these investments actually result in more uptime and customer-delighting innovations?

As digital services have become increasingly important to businesses and organizations, reducing downtimes and service disruptions have become critical objectives for business operations. This means management reporting and KPI’s are now crucial to quality management, providing the insight to let you improve incident remediation over time.

For the last few years, Squadcast has been building out a market-leading on-call and alert management solution. Over the past few quarters, we have significantly enhanced our on-call product by releasing and improving features related to Incident Response - including Slack / MS Teams integration, Runbooks, Postmortems, Service Level Objectives, and Status Pages. We believe that a reliability platform involves both on-call and incident response - one cannot work effectively without the other.

As a solutions engineer for FireHydrant, I speak with a wide variety of companies about their incident management programs — from start-ups with a handful of employees to large enterprise companies with thousands of engineers. Whether they’re looking to establish their incident management program or mature it, the same questions remain.