Operations | Monitoring | ITSM | DevOps | Cloud

Cribl recently introduced Lakehouse, a powerful new feature within Cribl Lake that enables fast queries on the freshest data. But it’s so much more than just speedy searches. Lakehouse redefines how organizations collect, store, manage, and analyze telemetry data at scale, ensuring a future-proofed, cost-efficient, and flexible approach to data management.

Learn what SOC 2 compliance is, why it matters, and how to achieve it. Discover key Trust Services Criteria, a step-by-step compliance checklist, and how Qovery automates and simplifies SOC 2 certification for modern cloud-native businesses.

In Part 1 of this series, we discussed key metrics for monitoring Airflow. In Part 2, we discussed strategies for collecting Airflow metrics, logs, and lineage. Finally, we will look at how you can use Datadog to monitor all this telemetry in a single consolidated view alongside telemetry from the rest of your infrastructure and services.

In Part 1 of this series, we looked at key Airflow metrics to monitor. Now, we’ll explore how you can collect those metrics, along with logs and traces, using Airflow’s native tooling. We’ll also look at a few key ways you can monitor this data from within the Airflow webserver interface.

Airflow is a popular open source platform that enables users to author, schedule, and monitor workflows programmatically. Airflow helps teams run complex pipelines that require task orchestration, dependency management, and efficient scheduling across many different tools. It’s particularly useful for creating data processing pipelines, orchestrating task-based workflows such as machine learning (ML) training, and running cloud services.