Operations | Monitoring | ITSM | DevOps | Cloud

Escalation policies for critical incidents

When a critical incident triggers, there’s no time to figure out who to call. That decision needs to be made well before the incident arrives. A dedicated escalation policy for critical incidents gives your team a clear path to follow the moment things go wrong, rather than leaving it to whoever happens to be around. This guide covers the key decisions involved in building that policy.

Understanding L1, L2, L3 escalation policy

L1, L2, L3 is one of the most common ways to structure an escalation policy. The idea is simple: an incident triggers and lands with a first responder. If it needs more attention, it moves up the chain to someone with more expertise. This guide explains how each tier works, when this structure makes sense, and what to keep in mind when setting one up.

From Passive Records to Active Care: Activating the EHR in Real time in Israel's hospitals

Israel’s healthcare system is widely recognized as one of the most digitally advanced in the world. Electronic health records are deeply embedded across hospitals, and platforms like Chameleon sit at the center of clinical operations. Patient data is captured, structured and accessible at nearly every stage of care delivery. But digital maturity alone does not guarantee operational efficiency.

The Definitive AWS Outage Report 2025: Reliability Analytics and Cascade Impact

Amazon Web Services remains one of the most popular cloud providers, with 200+ services in 39 regions across the world. Like all providers, they have their share of outages. In 2025, IncidentHub detected 38 AWS outages, of which the one on October 20th had the most widespread impact affecting hundreds of SaaS providers simultaneously. Payments were disrupted, students lost access to classrooms, developer tooling degraded, and some IT teams experienced alerting gaps.

A compass for designing your escalation policy

The first time you sit down to design an escalation policy, it can feel a little like a crossroads. You know incidents need to reach the right people. You just aren’t sure which structure makes the most sense. Should you route by severity? By who’s available? Or by team? There’s no single right answer. Think of this guide as a compass. A compass doesn’t tell you exactly where to go. It helps you orient yourself based on where you already are.

SIGNL4 Among Germany's Best Software Companies

SIGNL4 has been recognized by G2 as one of the Best German Software Companies and we couldn’t be more excited. Matthes Derdack, Founder of SIGNL4, emphasizes:“This recognition matters because it’s not based on marketing claims – it’s based on what our customers experience in real operations. Teams running mission-critical infrastructure rely on SIGNL4 when things go wrong, not when everything is fine.

Escalation policies for low-priority incidents

Teams put a lot of thought into how critical incidents are handled. Low-priority incidents usually don’t get the same attention. And without a proper escalation policy, they just land in a shared channel, waiting for someone to acknowledge. Setting up a clear policy for them is worth doing. Not because they need the same urgency as a critical incident, but because having a defined path for every incident makes the whole system more reliable.

Keeping it boring: the incident.io technology stack

At incident.io we run a deliberately simple technology stack. Keeping things boring has allowed us to scale from a few hundred customers to several thousand, while having only two platform engineers. In this post I'll walk through the stack, explain some of the choices we've made, and touch on the challenges we're facing as we grow.

What is an escalation policy? (And why every team needs one)

An escalation policy is the route an incident takes after it triggers. It lays out who gets alerted first and sets a wait time. If nobody responds, it moves the incident forward to the next person. The word “escalation” is worth pausing on. When an incident triggers and the first person doesn’t respond, the incident doesn’t sit and wait. It moves to the next person and keeps moving until someone picks it up. That forward movement is the escalation.

PagerDuty's Slack App Just Got a Whole Lot Better (And We're Just Getting Started)

If you’ve been eyeing chat-native incident tools and wondering whether PagerDuty can compete in Slack, this one’s for you. Are you still treating your incident management platform like a glorified pager? It’s time for an update. Over the past months, we’ve been evolving our Slack app from a notification tool into a full incident command center, and we’re coming for the chat-native tools (ahem, incident.io).

Incident Report: Exercises, Cleanups, and Evacuations

Every year, Honeycomb runs disaster recovery scenarios in multiple environments, including in production. Although each of our instances runs in a single region, on at least three Availability Zones (AZs), we have multiple plans for partial regional failures, and particularly, zonal failures. One of these tests was run on December 5th, and after its successful completion came its cleanup steps.

Secure access at the speed of incident response

Picture this: it's 2am, your pager goes off, and you're staring at a production database that's on fire. You know exactly what's wrong. You know exactly how to fix it. But you can't touch anything because you're waiting on someone to approve your access request. Meanwhile, your customers are down, your SLAs are bleeding out, and you're refreshing Slack hoping someone in security is awake to click "approve." This is the incident response tax that too many teams pay.

Boosting Rust developer productivity with cursor - Our journey at ilert

AI-assisted coding has evolved from a novelty into an industry standard. At ilert, we started our adoption in mid-2023, quickly realizing that success depends heavily on proper context and workflows. This is particularly acute with Rust. While the language is central to our backend infrastructure, its strict compiler rules and distinct idiomatic approaches make it notoriously difficult for modern LLMs to master.
Sponsored Post

What to Say When Things Break: Outage Notification Templates for Ops Teams

This practical guide explains what to say when systems break, offering ready-to-use outage notification templates and best practices to help ops teams communicate clearly during incidents. Learn how effective outage communication can reduce confusion, manage user expectations, and maintain trust during service disruptions.

Response Team @ incident.io

When an incident hits, every second counts. The response team at incident.io builds the tools that make sure engineers aren't flying blind when it matters most. Sam, Tech Lead of the response team, takes us inside what it's really like to build the core of incident.io: the high technical bar, the art of prioritisation, and why there's no shortage of meaningful work to do. If you're an engineer who wants to work on something that genuinely makes other engineers' lives better, this one's for you.

Platform Engineering 101: What It Is, How It Differs from SRE and DevOps, & Why It Matters for Incident Response

Platform engineering has emerged as a response to the growing complexity of modern software delivery. As organizations adopt Kubernetes, microservices, CI/CD pipelines, and infrastructure as code, they are creating dedicated teams responsible for building and operating the internal platforms that power developer workflows.
Sponsored Post

Forwarding Microsoft SCOM Alerts to the Service Desk

Modern IT operations rely heavily on monitoring solutions like System Center Operations Manager (SCOM) to detect issues across servers, applications, and services. While SCOM excels at generating alerts, organizations often struggle to ensure these alerts translate into actionable incidents in their IT Service Management (ITSM) platforms. Without proper integration, critical alerts may be missed, tickets may be created manually, and incident resolution can be delayed.

AI Engineering at incident.io

Working on AI in incident management means there's no playbook. No million blogs. Just building at the forefront of what's possible with AI models.In this video, Martha, Product Engineer on our AI team, talks about what it's really like working with AI that helps engineers respond to incidents faster. This covers the shift from traditional engineering, learning the personalities of different AI models, and why you need to embrace constant change when new models drop all the time.

YouTube Outage (Feb 17, 2026). What Happened?

On February 17, 2026, YouTube went down for users worldwide. Starting around 8:00 PM ET, the platform's homepage, Shorts feed, sign-in system, smart TV apps, YouTube Music, and YouTube Kids all stopped working. Over 21,000 reports were logged on IsDown alone. The error message was the same everywhere: "Something went wrong." For consumer users, it was an inconvenience. For businesses that depend on YouTube — content teams, advertisers, media companies, live streamers — it was a blind spot.

The post-mortem problem

Post-mortems are required, time-consuming, and widely disliked — but they’re also one of the biggest opportunities to improve reliability. In this webinar, we talked about how to run post-mortems that actually lead to learning and improvement. This covered why most post-mortems fall flat, how to structure them effectively, and walk through a real example to show what good looks like in practice. The goal: fewer wasted hours, better outcomes, and post-mortems that actually matter.

AI Is Changing Healthcare Faster Than Most Systems Are Ready For

Healthcare is shifting fast, and artificial intelligence is no longer a future concept sitting in research labs or pilot programs. It’s already embedded in clinical workflows, operational systems, and patient interactions, often in ways that feel subtle, uneven, and sometimes uncomfortable.

How to Set Up SMS Alerting w/ OnPage

In this quick tutorial, learn how to set up SMS alerting in OnPage to ensure your team never misses a critical notification. We’ll walk you through the step-by-step process: This setup ensures reliable message delivery using redundancy rules, so important alerts reach the right person at the right time. Let us know if you have any other questions!

Why SIGNL4 Is the Right Alarm Management Software to Maximize Machine Availability

A plant runs at its best when equipment stays online, processes remain stable, tolerances are met, raw materials are delivered in time, and scrap stays low. That’s how operations teams hit production targets, meet customer SLAs, stay on schedule, keep costs under control, and maintain consistent quality. But does everything always run according to plan? Of course not.

Code Is Cheap, Reliability Isn't: Owning Production in the AI era w/ Swizec Teller

In this episode, Swizec Teller, author of the bestselling Scaling Fast, makes a bold claim: code is cheap, reliability is not. As AI coding tools accelerate feature development, the real competitive advantage shifts to operating systems reliably in production. We explore the hidden complexity of SRE work, the addictive nature of agentic coding, and why ownership — not automation — remains at the core of modern software engineering.

Amazon Web Services outage - February 10, 2026

On February 10, 2026, Amazon Web Services (AWS) experienced an outage that triggered widespread reports of CloudFront failures and DNS resolution issues. While AWS later acknowledged the incident, StatusGator detected the disruption earlier using Early Warning Signals, giving customers valuable lead time before the provider confirmed anything publicly.

4 on-call burnout signs (and how to address them)

Being on-call can sometimes feel overwhelming. If that feeling goes unnoticed for too long, it often translates into burnout. And early burnout signs usually show up in ways, like how people respond to incidents or how they feel about the schedule. This guide walks through four such signs that can be useful to watch for before on-call burnout sets in.

Claude outage - February 10, 2026

On February 10, 2026, Claude users around the world began reporting service failures affecting chat sessions, API integrations, and Claude Code workflows. The first verified outage report reached StatusGator at 19:33 UTC. StatusGator issued an Early Warning Signal at 20:24 UTC. Claude did not post an official “Investigating” update until 22:11 UTC. This incident clearly demonstrates the gap between real user impact and official status page updates.

Incident Alerting: What We Believe It Should Do

Incident alerting is a critical part of modern operations, yet it’s often misunderstood or reduced to “sending notifications.” In reality, it is about ensuring that the right people are informed at the right time – and that incidents move from detection to action without confusion or delay. This page explains why fast, reliable alerting matters, where it fits between monitoring and incident response, and what best practices look like.

5 Offbeat on-call rotations that work

Most teams choose standard on-call patterns like weekly or daily rotations. But sometimes a less conventional rotation can solve a specific problem or just fit better with how your team works. This guide walks you through five offbeat on-call rotations. For each, we look at why it might work for you and the challenges involved. This helps you see the full picture before you decide to try them out. Let’s dive in!

Follow-the-sun and other on-call models

Most teams run on-call using rotation-based schedules where responsibility shifts every few days or weeks. But some situations call for different models that change who responds based on time zones, expertise, or the type of incident that triggers. This guide walks you through six on-call models that work outside the standard rotation patterns.

Turning Data Into Decisions with the xMatters Incident AI Agent

When an incident hits, the gap between awareness and action can make all the difference. Responders know the pain: endless tool-switching, chasing updates, and fragmented data. It’s not a lack of capability that slows response; it’s the lack of context and connection. That’s why we built the xMatters Incident AI Agent, a purpose-built, conversational assistant that brings intelligence and automation directly into the heart of incident response.

AWS CloudFront Outage (Feb 2026): Timeline, Cascade, and Lessons

At approximately 9:15 PM UTC on February 10, 2026, Amazon CloudFront began returning NXDOMAIN responses for DNS queries against specific distributions. In practical terms: DNS was telling users that services behind those distributions simply didn't exist. The root cause was a DNS resolution failure within CloudFront's infrastructure that quickly spread to eight interconnected AWS services.

ilert now supports a native WhaTap integration

ilert now supports a native WhaTap integration, connecting AI-native observability with AI-first incident management in a seamless workflow. This integration allows DevOps, SRE, and IT teams to move instantly from detection to resolution – cutting through alert noise, improving coordination, and dramatically reducing MTTR in even the most complex IT environments.

How to Create and Manage Incidents in Uptime.com

Learn how to create and manage incidents on your Uptime.com Status Page to keep your subscribers informed about service disruptions and maintenance events in real-time. In this tutorial, we'll cover understanding incident statuses (Investigating, Identified, Monitoring, Resolved, and more), three ways to create a new incident, configuring incident details and timelines, adding updates with Markdown formatting, managing and editing incidents, notifying Status Page subscribers, and using the REST API for incident management.

Reference architecture: The blueprint for safe and scalable autonomy in SRE and DevOps

Everyone wants autonomous incident response. Most teams are building it wrong. ‍ The ultimate goal of autonomy in SRE and DevOps is the capacity of a system to not only detect incidents but to resolve them independently through intelligent self-regulation. However, true autonomy isn't born from automating random, isolated tasks. It requires a stable foundation: a Reference Architecture.

Silent Failure in Production ML: Why the Most Dangerous Model Bugs don't Throw Errors

You’ve done it. Your machine learning model is live in production. It’s serving predictions, powering features, and quietly doing its job. Dashboards are green. There are no errors in the logs. Nothing appears broken. And yet, something is wrong. Predictions are getting less reliable. Users are waiting a little longer for responses. Conversion rates are slipping. Trust is eroding, but no alert fires, no system crashes, and no one knows there’s a problem until the damage has been done.

PagerDuty x Backstage Plugin Demo: Eliminate Context Switching for On-Call Engineers

Join Rocío, Product Manager of the Forward Deploying Engineering team at PagerDuty, as she demonstrates how the PagerDuty Backstage plugin transforms incident response by bringing critical operational data directly into your developer portal.

Weekly vs. split-week on-call rotations: A guide to finding the right rhythm

When you move past daily rotations but find anything longer than a week feels too stretched out, you often end up choosing between weekly and split-week rotations. Weekly rotations give you a full seven days before handing off. Split-week rotations break that time into smaller chunks like 2-day, 3-day, or 4-day shifts. Each approach creates a different rhythm for your team. This guide compares both patterns across three key criteria.

PagerDuty + OOPS Meetup: AI in Incident Management

AI is transforming industries at pace, and Incident Response is no exception - raising important questions about how humans and automation should work together when systems are failing and pressure is highest. Panelists:​Andrew White (Technology Director, checkout.com) James Pickles (Senior Solutions Consultant, PagerDuty)​Sarah Wells (Independent Consultant, former Technology Director at FT) Suraj Singh Dadwal (Team Lead, Incident & Problem Management, IG)

Event Intelligence Solutions Part Three: Best Practices for Successful Adoption

As Event Intelligence Solutions (EIS) move from early adoption to operational necessity, many enterprises are realizing that success depends on more than selecting the right technology. For Banking and Financial Services organizations, effective adoption requires a clear strategy, disciplined execution and a strong alignment to business priorities and regulatory demands and not least, customer expectations.

Transform IT major incident management with customizable AI Workflows from BigPanda

Enterprise Management Associates found that major IT service outages are increasing in cost, frequency, and duration, with unplanned downtime costing large enterprises nearly $25,000 per minute, or $1.5 million per hour. When every minute costs $25,000, you can’t afford to waste engineering time on coordination tasks like creating channels, paging experts, typing summaries, and posting updates. An agentic AI-powered incident assistant can eliminate that waste and reduce bridge call costs.

2-day vs. 4-day on-call rotations: Which one fits your team

Teams that find a weekly rotation too long and a daily rotation too short often end up choosing between 2-day and 4-day rotations. This guide compares both these rotations across three key criteria. For each criterion, we have discussed how it works for 2-day and 4-day rotations and recommended what to choose when. To make it easy, we also included a comparison table for a quick overview. This gives you all the information you need at a glance. Let’s dive in! Table of contents.

How to choose the right on-call rotation

Choosing an on-call rotation is about finding a rhythm that balances your team’s well-being and your system’s reliability. The right on-call rotation helps prevent burnout and makes on-call duties sustainable over the long run. This guide walks you through different on-call rotation patterns, from daily rotation to after-hours rotations. We’ll look at why you might choose a particular rotation and the challenges that often come with it.

Why a month is too long to be on-call

There is often a temptation to stretch on-call shifts to a month or longer, especially when incident volume is low. The logic seems sound. If the phone rarely rings, it feels unnecessary to hand off on-call duties every week. But looking strictly at incident volume often misses the human side of the equation. Being on-call isn’t just about answering pages. It is also a state of mind. Even when it is quiet, simply being on-call could create fatigue of its own.

EasyVista Service Manager + SIGNL4

Modern IT service management platforms excel at structuring work: tickets, workflows, approvals, SLAs, and reporting. But when a major incident occurs, success depends on more than clean processes – it depends on how fast the right people are reached and respond. This is where EasyVista Service Manager (EVSM) and SIGNL4 work exceptionally well together.

How HVAC Companies, Contractors and Property Management Firms Use OnPage for Emergency Response

Over the past couple of weeks, as snowstorms and extreme cold swept across much of the Northeast, something interesting started happening on our end at OnPage. Our phones lit up. Not from healthcare teams or IT operations/tech teams, which is where many people expect us to be used, but from HVAC companies, contractors, and property management firms scrambling to prepare for what they knew was coming.

When Minutes Matter, Records Aren't Enough

When critical systems go down, your business needs action, not another ticket. PagerDuty's Operations Cloud doesn't just track incidents; it resolves them. With AI-powered automation, intelligent routing, and real-time response, we turn alerts into outcomes while your competitors are still filling out forms. Deploy in days, not months. No complex implementation. No bloated services. Just faster resolution and lower total cost of ownership.

PagerDuty + Guide Integration: Never Schedule an Interview Over an Incident Again

For engineering organizations running on PagerDuty, on-call schedules are sacred. When P0 incidents happen, you need your best engineers focused and ready, not getting scheduled to conduct an interview they’ll have to decline. For years, recruiting teams have been playing a manual game of Tetris, cross-referencing on-call rotations against interviewer availability every single time they book a technical screen or panel.

Everything you need to know about ITIL 5, AI and incident management

ITIL 5 launched in January 2026, and for the first time in the framework's 40-year history, AI governance is front and center. If you're running incident management, on-call rotations, or building operational tooling, this matters: the gap between AI adoption and AI governance is about to become a compliance and operational risk issue. I’m not usually a big ITIL fan, but this guidance has some genuinely useful framing and questions.