Operations | Monitoring | ITSM | DevOps | Cloud

July 2022

Sponsored Post

Classifying Severity Levels for Your Organization

Major outages are bound to occur in even the most well-maintained infrastructure and systems. Being able to quickly classify the severity level also allows your on-call team to respond more effectively. Imagine a scenario where your on-call team is getting critical alerts every 15 minutes, user complaints are piling up on social media, and since your platform is inoperative revenue losses are mounting every minute. How do you go about getting your application back on track? This is where understanding incident severity and priority can be invaluable. In this blog we look at severity levels and how they can improve your incident response process.

Setting up Runbooks in Squadcast | SRE Best Practices | Squadcast

A Runbook is a compilation of routine procedures and operations that are documented for reference while working on a critical incident. Sometimes, it can also be referred to as a Playbook. From this video, learn to create, attach, reference and mark progress for incident resolution using Runbooks.

What's New: Updates to Incident Response, PagerDuty Process Automation, Integrations, and More!

Following another successful PagerDuty Summit, development continues across several areas of the product. We’re excited to announce a new set of updates and enhancements to the PagerDuty Operations Cloud. Recent updates from the product team include Incident Response, PagerDuty® Process Automation and PagerDuty® Runbook Automation, Partner Integrations & Ecosystem, as well as Community & Advocacy Events updates.

Introducing Our Newest Integration with ServiceNow

Blameless just released a new integration to ServiceNow’s incident management ticketing solution. If you are a modern DevOps team moving towards SRE practices and you want to speed the time to incident resolution through streamlined, automated workflows, this is worth investigating.

Release Notes: Process Automation and Rundeck OSS 4.4.0

Product managers Forrest Evans and Jake Cohen show off new features and enhancements in PagerDuty Process Automation and Rundeck Open Source version 4.4.0. Version 4.4.0 features two new plugins for #AWS:#Lambda Custom (ephemeral) scripts#ECS/#Fargate Commands For more details on other improvements in this release, see the full Release Notes.

3 common pitfalls of post-mortems

Small confession: we currently use the term 'post-mortem' in incident.io despite preferring the term 'incident debrief'. Unless you have particularly serious incidents, the link to death here really isn’t helping anyone. However, we're optimising for familiarity, so we're sticking to the term 'post-mortem' here. Ask any engineer and they’ll tell you that a post-mortem is a positive thing (despite the scary name).

Zero Trust Security: Key Concepts and 7 Critical Best Practices

Zero trust is a security model to help secure IT systems and environments. The core principle of this model is to never trust and always verify. It means never trusting devices by default, even those connected to a managed network or previously verified devices. Modern enterprise environments include networks consisting of numerous interconnected segments, services, and infrastructure, with connections to and from remote cloud environments, mobile devices, and Internet of Things (IoT) devices.

Automating Common Diagnostics for Kubernetes, Linux, and other Common Components

This is the second piece in a series about automated diagnostics, a common use case for the PagerDuty Process Automation portfolio. In the last piece, we talked about the basics around automated diagnostics and how teams can use the solution to reduce escalations to specialists and empower responders to take action faster. In this blog, we’re going to talk about some basic diagnostics examples for components that are most relevant to our users.

What Is a Secure SDLC?

The Software Development Lifecycle (SDLC) framework defines the entire process required to plan, design, build, release, maintain and update software applications, including the final stages of replacing and decommissioning an application when needed. A Secure SDLC (SSDC) builds on this process, integrating security at all stages of the lifecycle. When migrating to DevSecOps (collaboration between Development, Security, and Operations teams), teams typically implement an SSDLC.

StatusCast Top Picks: 10 More Awesome Customer IT Status Pages

IT services are a critical backbone to the operations and functioning of most every business and organization. As more and more IT departments have embraced the need for good governance, this has driven greater transparency. From the perspective of IT service management, this has manifested itself as much greater openness when communicating about IT service availability.

Remote Actions for IT Remediation, IoT Actions and more

SIGNL4 supports the remote execution of automated tasks or workflows in IT or IoT systems using Remote Actions. These remote actions offer a wide range of applications. You can execute remote actions in response to an alert to trigger some kind of remediation action. But there are many more possible use cases. This article provides some examples and ideas about what is possible.

DevOps Tools

A tool that aids in automating the software development process is called DevOps Tool. It largely concentrates on interaction and cooperation between experts in product management, software development, and operations. A DevOps solution also enables teams to automate the majority of software development procedures including build, conflict management, dependency management, deployment, etc. and lessens human labour.

Cultural Adoption of Automation

You're convinced that automation of standard, basic tasks is the right way to go for your team. You want to get your teams """"out of the muck"""" and into more value-added tasks, such as the creation of new features, or launching new clients, or expansion into new territories. How do you build consensus with your teams? With your leadership? To convince your executive team that automation is the way to go, they'll insist on metrics to capture their return, whether through increased agility, improved service stability, and so on.

Incident Review & Postmortem Reports: 8 Best Practices

People make mistakes, technology breaks down, and processes aren’t infallible. But, when incidents happen, what can we do about it? What can we learn? As with all things, learning isn’t a binary action, it’s a process. And, when an incident occurs, organizations typically conduct a post-mortem analysis and generate a post-incident review to uncover what went wrong and why.

Sponsored Post

How to Spot the Effects of Alert Fatigue

Imagine being part of an overactive group chat that causes your phone to buzz every few minutes. In the beginning, you open every message but soon realize that most of them aren't important-or at least are not relevant to you. So, what do you do next? Maybe you let the messages pile up and check them later. Or perhaps, you mute the group chat and ignore the incoming messages altogether. You can blame this tendency to ignore or avoid incoming messages or notifications on one culprit: alert fatigue.

How Retrospective Data Enhances Reliability Insights

When things go wrong, we try to learn for the next time. Every incident should be a learning opportunity to make your system more reliable for the future. Luckily with Blameless Reliability Insights, you can see patterns in incidents at a glance, right out of the box. In fact, the ability to tag incidents makes reliability data even more helpful by allowing you to collect granular details about reliability, especially as they pertain to your unique business needs. ‍

FireHydrant Tasks provide turn-by-turn navigation during an incident

An incident has been declared and your runbook has fired. Everyone is gathered in your Slack channel, the tickets are opened, and roles are assigned. Now what? This is when most teams manually update status pages and kickoff investigation streams using a patchwork of tribal knowledge and supporting playbook documents.

Why SREs Need to Embrace Chaos Engineering

Reliability and chaos might seem like opposite ideas. But, as Netflix learned in 2010, introducing a bit of chaos—and carefully measuring the results of that chaos—can be a great recipe for reliability. Although most software is created in a tightly controlled environment and carefully tested before release, the production environment is harsher and much less controlled.

Episode 5: Mooving to... Practical Postmortems

Episode 5, Mooving to… Practical Postmortems covers how to leverage postmortems to effectively learn from failure. Postmortems are a commonplace reference and are now considered a best practice in most modern engineering teams. However, there’s still a lot of confusion on what postmortems should be – and more importantly, what they should NOT be. Thom Duran, Senior Manager of Productivity from Panther walks us through all that and more in the latest Mooving To.. episode!

What should you choose? Docker Swarm vs Kubernetes

Since the introduction of containerisation by Linux many years ago, maturity has shifted from the traditional virtual machine to these containers. These tools have made application development much easier than the initial process. Docker Swarm and Kubernetes came into action when the number of containers increased within a system, they helped orchestrate these containers. A question that arises is, which one is the better option?

Top Incident Response Metrics & How to Use Them

Two categories a software organization should always strive to improve in are: Data analysis is one way that your organization can improve the efficiency of incident management and overall application quality. However, the questions remain – which metrics should be collected? How can analysis of these metrics facilitate these improvements? Read on to hear about five key metrics essential to incident response.

Our fully-redesigned incident response experience delivers a more intuitive workflow

Today we’re releasing fully redesigned Slack and Command Center experiences for FireHydrant so anyone on your team can intuitively navigate the incident response process — in the app or on the web. There are many things you can do ahead of an incident to help things run smoothly: design and document your process, automate predictable steps, train the team, and run drills.

Don't Let Outages Ruin Your Reputation - Prevent Them With AIOps

The world is increasingly digital. The U.S. Census Bureau estimates e-commerce grew 14.2% from 2020 to 2021, for a total of $870.8 billion in sales. And just look at the trends in remote work. According to a FlexJob and Global Workplace Analytics report, remote work has grown 44% over the last five years and an astonishing 159% over the last 12. Indeed, much of America relies on a slew of digital apps and services to get business done every day. So what does this mean for businesses?

SecOps tools - SecOps & incident management for 2022.

Importance of secOps tools – The threats in the cyber world are becoming more and more complicated and sophisticated with each passing day, while the rapid expansion of digital operations, with more nodes, networks, and servers has resulted in more vulnerabilities. This situation demands efficient SecOps teams as well as practices so that threats are thwarted, and networks and data are always protected. What is SecOps & Best SecOps tools?

AWS outage? A better way to monitor outages in Amazon Web Services

Amazon Web Services (AWS) needs no introduction. It's one of the most popular services in the world. Or actually, the most popular cloud infrastructure provider (34%) according to this study. Like in any other service, there are outages. For people running their infrastructures, there's a good chance that outages have impacted your business in the past. And the reality for AWS (or any other service) is that there's a good chance it will happen again.

A deeper dive into the Rogers outage

Beginning at 8:44 UTC (4:44am EDT) on July 8, 2022, Canadian telecommunications giant Rogers Communications suffered a catastrophic outage taking down nearly all services for its 11 million customers in what is arguably the largest internet outage in Canadian history. Internet services began to return after 15 hours of downtime and were still being restored throughout the following day.

360º Fireside Chat with PagerDuty, Lisbon's Newest Tech Employer

Joining João Freitas, GM & Engineering Site Lead in Lisbon for this 360º Fireside Chat about PagerDuty’s projects, challenges, the technical parts of PagerDuty and how everything comes together and where we are today and where we’re going in the future.

Outage Alert: Top 5 Outages of Q2 2022

We are halfway through 2022 and one thing is certain – downtime is here to stay. In fact, trends are showing the frequency of downtime is increasing, along with the severity and wide-spread impact. Consumers and businesses are more interconnected and reliant on technology and software than ever, from remote business communication to simply listening to your favorite podcast on your way to work.

Promoted to SRE Advocate: A Dream Turned Reality

I get chills thinking about a line from the first film adaptation of Roald Dahl's Charlie and the Chocolate Factory, Gene Wilder as Wonka nearly whispers it to Charlie, as if it is secret information: We are the music makers, and we are the dreamers of dreams. For me, the quote (taken from a poem by Arthur O'Shaughnessy) is austere: We are the creators of what we create, and what we create becomes what we are.

We've raised $34M to help organisations be resilient in the face of failure

TL;DR: We’ve raised $34M to bring increased resilience to organisations around the world. With this latest round of investment we’re expanding internationally in the US, accelerating our product plans, and growing our amazing team 🎉 As technology becomes more complicated and runs an ever greater part of our lives, failure becomes more inevitable, and more costly.

What IT Pros Can Learn from the Marriott Data Breach

Despite the best efforts of individuals to protect their own data, they cannot always account for the cybersecurity shortcomings of larger organizations such as their employers, financial institutions, and healthcare providers entrusted with their personal information. Hotels should also be added to this list of vulnerable entities, as was made painfully apparent in the most recent Marriott data breach.

How MSPs Can Provide Irreplaceable Value in Uncertain Times

If you have been following the financial news lately, you have surely become all too familiar with the challenging economic conditions that have emerged in 2022. As rising inflationary concerns put pressure on the bottom line, decision makers within businesses of all sizes are suddenly having to re-evaluate strategies, forecasts, and expenses. This pivot to a more conservative outlook is not unlike the approach adopted by businesses at the onset of COVID-19.

Combining AIOps with Service Intelligence: Critical for Digital Service Uptime

For years, Artificial Intelligence for IT Operations (AIOps) applications have helped organizations streamline and improve their IT processes for better business results. But today, with rising disruptions and BCG research noting that 70% of digital transformations fail, these incidence response applications alone are no longer enough to maintain digital service uptime and ensure customer satisfaction.

Amazon OpenSearch + Squadcast Integration: Routing Alerts Made Easy

Developers often find comfort in embracing open-source software for numerous reasons. One of the most important reasons is the freedom to use that software anywhere and how they wish to. Amazon OpenSearch is an open-source search and analytics suite derived from Elasticsearch. It lets you perform interactive log analytics and real-time application monitoring with ease.

The Improved xMatters Group Experience: Product Feature Updates

We’re constantly looking for new ways to help DevOps, SREs, and operations teams automate operations workflows, secure infrastructure and applications, and rapidly deliver their products at scale. This commitment to our customers — and yours! — led us to redesign the way you experience groups in xMatters.

7 ways tagging incidents can teach you about system health

One of the most powerful ways to prepare for future incidents is to study and learn from patterns in past incidents. Blameless Reliability Insights highlights these patterns for you, with out-of-the-box dashboards that automatically collect and present all types of statistical information about your incidents.

Making the wrong choice on build vs buy

A few years ago I’d just moved to London and started out at my first software job. I was having a great time building things and making new friends, and one evening a friend and I decided there was a new problem we wanted to solve: we really didn’t like the expenses software. We thought it was confusing and over-complex, and decided we could do better.

July 2022 update - Remote actions for super-fast incident remediation

Our July update ships a very powerful new feature – remote actions. Remote actions are available for execution – once configured – in the SIGNL4 mobile app and allow you to quickly perform remediation actions without having to fire up a notebook and VPN or without using a desktop PC. So, genuine anywhere remediation comes true. As always, you can find all the details in this blog article.

More Powerful than Ever: PagerDuty's Revamped Mobile App is Primed for Even Better Incident Response

2020 revolutionized how we work. Many went from full-time office work to 100% remote overnight. And now that in-office is once again on the horizon, companies are thinking of ways to continue to work flexibly. However, this comes with increased challenges, and a need for tools that match this working style. The PagerDuty mobile application is well recognized, with a 4.8 stars rating on the App Store and Google Play.

The Importance of Next-Gen 911 Technology for Emergency Dispatchers

There has been a tremendous amount of change around public safety over the last few years, especially regarding 911. Keeping up with Next-Generation 911 technology (NG911) and the wealth of information can certainly be overwhelming. To make matters worse, emergency callers are more mobile than ever before.

Custom Reliability Insights Reports: Follow Up Action Items

Engineering teams use the Reliability Insights feature in Blameless to understand reliability in a holistic way. In addition to tracking incident data, you can keep a pulse on how well teams and workflows are operating. For example, some of the best ways to maximize value from Reliability Insights is to build reports that reflect how your team stays on task, communicates, and assigns responsibilities. In this series, we'll walk you through the most common reports we see reliability teams using and referring to regularly.

Blameless Reliability Insights: FUA (Follow Up Action) Statuses

Engineering teams use the Reliability Insights feature in Blameless to understand reliability in a holistic way. In addition to tracking incident data, you can keep a pulse on how well teams and workflows are operating. For example, some of the best ways to maximize value from Reliability Insights is to build reports that reflect how your team stays on task, communicates, and assigns responsibilities. In this series, we'll walk you through the most common reports we see reliability teams using and referring to regularly.

Blameless Reliability Insights: How to Build Custom Reports

Engineering teams use the Reliability Insights feature in Blameless to understand reliability in a holistic way. In addition to tracking incident data, you can keep a pulse on how well teams and workflows are operating. For example, some of the best ways to maximize value from Reliability Insights is to build reports that reflect how your team stays on task, communicates, and assigns responsibilities. In this series, we'll walk you through the most common reports we see reliability teams using and referring to regularly.

What I learned from leading my first incident

A few weeks ago we had a major incident. We were releasing our Practical Guide to Incident Management, and after posting about it online an incident.io employee noticed that the page wasn’t loading. Just to set the scene, I’ve been at incident.io for 3 months and don’t have any experience of incidents in my previous role. When the team got paged I expected this to be one of those “follow along and learn how the wizards work their magic” exercises.

AlertOps is in the ConnectWise's 2022 PitchIT Accelerator Program!

PitchIT is a competition for MSP innovators. The program is designed to showcase potential offerings that could be built or integrated into the ConnectWise platforms. It’s a 16-week accelerator program where AlertOps and the other participants will go through a rigorous business assessment, gain coaching from industry experts, earn placement on the ConnectWise marketplace, engage in co-marketing and more.