Operations | Monitoring | ITSM | DevOps | Cloud

I’ve been getting some version of this question a lot lately, mostly in our Seer preview webinars. Different audiences, same handful of questions: Worth answering all three in one place. Honestly, I needed to write this down for myself too. Things are moving fast around all of us and answers seem to get more nuanced by the week. This is an attempt to codify the difference: what each option is and when it makes sense to reach for one over the other.

Databricks is a platform for analytics, AI, and data intelligence that has pioneered the lakehouse paradigm in modern data systems.

In the first part of this series, we cataloged key metrics for Databricks data engineering, analytics, and Model Serving workloads. In this post, we’ll discuss how to collect those metrics and other telemetry data from Databricks and Apache Spark, which powers Databricks under the hood. We’ll cover collecting and querying telemetry data via system tables, as well as the other primary sources of visibility into.

Earlier in this series, we covered key metrics for monitoring performance in Databricks and discussed Databricks’ native resources for accessing those metrics and other key observability data, such as logs and data lineage. In this post, we’ll cover using the Databricks integration to bring that data into Datadog and monitor your Databricks analytics and AI/ML workloads alongside the rest of your end-to-end data pipelines and distributed infrastructure. We’ll show you how to.

With GitOps, you can roll back your cluster with git revert and explain each approved change through a commit log. Routing normal changes through Git and running an in-cluster agent that detects drift from the repository gives infrastructure changes the same review and rollback discipline as application code, and Git preserves their history. This guide covers the four GitOps principles, the pull-based reconciliation workflow, and the tools and practices you need to get started.