Operations | Monitoring | ITSM | DevOps | Cloud

We recently wrapped up a large-scale, multi-month Kafka migration project. We used to run self-hosted Confluent Platform and ZooKeeper as clusters of AWS EC2 instances, and now all of our Kafka clusters run open-source Apache Kafka 4.1.1 running in KRaft mode and deployed to AWS EKS.

The world is especially hard right now. The future of the software engineering profession looks more uncertain than ever. Execs are under heavy pressure to turn AI into magic results, and teams are fighting product competition and AI-induced burnout on one side, melting mental models and hellish oncall on the other side. Observability was supposed to be a solved problem by now.

One of the hardest challenges facing platform teams is wrangling the rising volume of PRs looking to add drift to the systems we've invested in. It's impossible to catch them all, so it's more important than ever to invest in building stronger guardrails so our product teams can keep building quickly and catch issues before they merge to main. Linters are a great tool to reach for first.

This post was co-written with Staff Software Engineer Martin Holman. Honeycomb Canvas is a collaborative investigation environment. When something goes wrong in production, multiple engineers might join the same Canvas to debug it together. Each person has their own AI agent, so they can pursue their own conversation thread and line of inquiry. This creates an opportunity for coordination.

"Hello world, this is your agent speaking!" The agent loop! The LLM is calling tools, the answers are sensible, and the sky's the limit. Now, as you look forward to production, you look for a composable toolset, something that can grow with your use case and system needs. That's what we created with Honeycomb Canvas: a collaborative investigation space where AI agents help you understand, fix, and learn about your system.

As of today, I’ve drafted this post upwards of 10 times – it’s old enough that the version I first started working on was called “Reflections on 1 Year of SRE Management” (I’m currently at 2.5 years). But everything I learned during that first year became critical for the next.

At O11yCon, we talked to engineering teams across the industry, and the numbers are starting to get genuinely wild: Mixpanel DevOps Engineer Eddie Bracho told us their engineering team is generating 50% more PRs than before AI came into the mix (sorry). That kind of velocity is exciting, but it's also a pressure test for every part of your stack that isn't writing code, including your observability practice. Here's what we're hearing from customers about how that's playing out.

AI agents are nondeterministic, multi-step, and opaque. When one fails in production, "the model said something weird" is the cheapest, most useless line in your incident postmortem. To debug agents the way they actually run, you need telemetry that captures all of it, in order, with enough context to reconstruct what happened. The OpenTelemetry GenAI Semantic Conventions give you a vendor-neutral way to do exactly that.

IT’S HERE it’s here it’s here it’s here!!!! The second edition of Observability Engineering is available for download, and since Honeycomb is the sponsor, you can now download it from our website (the dead tree version will take another month). This is a strange time to be writing a book.