Already a quarter of the way into 2024, we’re seeing a lot of shake-up in on-call best practices. We’re excited to see AI in the mix, but we’re also seeing a renewed focus on existing and neglected best practices. Some current topics in on-call best practices include: In this article, we’ll review some best practices and explore the 2024 trends.

Incident response and management (IRM) doesn’t happen in a vacuum. Your ability to respond to issues in a timely manner depends greatly on how well your on-call engineers can use their IRM tooling and observability tools together to understand what changed and why.

SRE and DevOps teams are the backbone of system uptime and reliability. But managing On-Call schedules, alerts, and communication during incidents can quickly turn resolution efforts into burnout. This blog explores the top On-Call management tools in 2024, designed to streamline Incident Response and keep your team ready for action.

Rootly On-Call streamlines incident management with automated scheduling, noise reduction, and centralized documentation. It mitigates on-call fatigue with features like flexible overrides, shift visibility, and shadow rotations, enhancing team well-being and preventing burnout.

On your bedside table sits a piece of software designed to wake you up. It loves bothering you when something goes wrong — and making it your responsibility to sort it out Meet the new incident.io On-call app. We designed it this way: to be as interruptive as possible. Whether you’re watching telly, at the gym, or as mentioned, fast asleep, it’ll get you. Got called even though you’re in silent mode? Great! We’ve done our job properly.

The on-call experience is sometimes a dreaded one for software engineers. Those late-night alerts and frantic Slack messages, after all, don’t exactly sound pleasant. But what’s an on-call shift really like? Is that perception of constant fire-fighting and 3 AM wake-up calls actually realistic? Michael Mandrus and Owen Smallwood, both senior software engineers here at Grafana Labs, wanted to set the record straight.

Dealing with a flood of incidents, each with varying degrees of urgency, can be a daily struggle for Incident Response teams. Suppose a low-priority alert pings while you're tackling a critical incident. This pulls your focus away from the urgent issue. This constant alert bombardment can: How do engineers ensure that high-severity issues take precedence? Don't they want to avoid being bothered or bombarded with notifications while addressing critical matters? They sure do.

Incidents are a stressful time for your team: your service isn't working the way you expect and your customers/stakeholders want to know what's going on. The last thing you want to do is let your team improvise everything when it comes to responding to incidents. Google's own SRE book has great overall tips for incident management, part of which involves "develop(ing) and document(ing) your incident management procedures in advance", which this article dives into.