Incident response and management (IRM) doesn’t happen in a vacuum. Your ability to respond to issues in a timely manner depends greatly on how well your on-call engineers can use their IRM tooling and observability tools together to understand what changed and why.

Rootly On-Call streamlines incident management with automated scheduling, noise reduction, and centralized documentation. It mitigates on-call fatigue with features like flexible overrides, shift visibility, and shadow rotations, enhancing team well-being and preventing burnout.

On your bedside table sits a piece of software designed to wake you up. It loves bothering you when something goes wrong — and making it your responsibility to sort it out Meet the new incident.io On-call app. We designed it this way: to be as interruptive as possible. Whether you’re watching telly, at the gym, or as mentioned, fast asleep, it’ll get you. Got called even though you’re in silent mode? Great! We’ve done our job properly.

The on-call experience is sometimes a dreaded one for software engineers. Those late-night alerts and frantic Slack messages, after all, don’t exactly sound pleasant. But what’s an on-call shift really like? Is that perception of constant fire-fighting and 3 AM wake-up calls actually realistic? Michael Mandrus and Owen Smallwood, both senior software engineers here at Grafana Labs, wanted to set the record straight.

Dealing with a flood of incidents, each with varying degrees of urgency, can be a daily struggle for Incident Response teams. Suppose a low-priority alert pings while you're tackling a critical incident. This pulls your focus away from the urgent issue. This constant alert bombardment can: How do engineers ensure that high-severity issues take precedence? Don't they want to avoid being bothered or bombarded with notifications while addressing critical matters? They sure do.

Incidents are a stressful time for your team: your service isn't working the way you expect and your customers/stakeholders want to know what's going on. The last thing you want to do is let your team improvise everything when it comes to responding to incidents. Google's own SRE book has great overall tips for incident management, part of which involves "develop(ing) and document(ing) your incident management procedures in advance", which this article dives into.

It’s 3am. You wake up to a blaring alarm, the sound burned into your soul from countless sleepless nights. You reach for your phone, ‘press 4 to acknowledge’ and bleary eyed, you open your laptop, grab a coffee and get to work. The next hour is a whirlwind—bringing services back online, keeping colleagues in the loop, maintaining a list of action items, updating a status page that will be seen by millions of customers. Potentially for the fifth time this month.