Operations | Monitoring | ITSM | DevOps | Cloud

Before we started using Datadog Real User Monitoring (RUM), we relied on frontend logging to gather data about the user experience. Logs gave us some helpful information about exceptions and errors but didn't provide any insight into issues directly related to the user’s perspective.

If you’re a Datadog customer, you’re likely using our platform to gain visibility into your infrastructure and applications and to troubleshoot using logs, metrics, and traces when issues arise. To support these efforts, you’ll want access to the most granular telemetry signals and intuitive workflows that streamline your investigation.

Delivering software quickly and reliably is the main focus of modern DevOps. But to improve your delivery performance, you need to understand it, and that starts with measurement. Teams primarily measure performance in this area by using DORA metrics—deployment frequency, change lead time, change failure rate, and time to restore service*. These metrics help teams understand trends in their software delivery practices in quantifiable terms that they can track and improve over time.

As your applications scale across cloud and SaaS providers, allocating costs and optimizing workloads become increasingly important—and challenging. Without access to cost data in their daily workflows, engineering teams can’t easily understand the cost of their resources and identify where they can reduce their spend. And while FinOps teams have access to cost data, they often review this information in silos.

Airflow is a popular open source platform that enables users to author, schedule, and monitor workflows programmatically. Airflow helps teams run complex pipelines that require task orchestration, dependency management, and efficient scheduling across many different tools. It’s particularly useful for creating data processing pipelines, orchestrating task-based workflows such as machine learning (ML) training, and running cloud services.

In Part 1 of this series, we looked at key Airflow metrics to monitor. Now, we’ll explore how you can collect those metrics, along with logs and traces, using Airflow’s native tooling. We’ll also look at a few key ways you can monitor this data from within the Airflow webserver interface.

In Part 1 of this series, we discussed key metrics for monitoring Airflow. In Part 2, we discussed strategies for collecting Airflow metrics, logs, and lineage. Finally, we will look at how you can use Datadog to monitor all this telemetry in a single consolidated view alongside telemetry from the rest of your infrastructure and services.