Operations | Monitoring | ITSM | DevOps | Cloud

Software companies are in a constant pursuit to optimize their delivery flow and increase release velocity. But as they get better at CI/CD in the spirit of “move fast and break things,” they are also being forced to have a very sobering conversation about “how do we fix all those things we’ve been breaking so fast?” As a result, today’s cloud-native world is fraught with production errors, and in dire need of observability.

Kafka is a distributed, highly available event streaming platform which can be run on bare metal, virtualized, containerized, or as a managed service. At its heart, Kafka is a publish/subscribe (or pub/sub) system, which provides a "broker" to dole out events. Publishers post events to topics, and consumers subscribe to topics. When a new event is sent to a topic, consumers that subscribe to the topic will receive a new event notification.

The current landscape of what our customers are dealing with in monitoring and observability can be a bit of a mess. For one thing, there are varying expectations and implementations when it comes to observability data. For another, most customers have to lean on a hodgepodge of tools that might blend open source and proprietary, require extensive onboarding as team members have to learn which tools are used for what, and have a steep learning curve in general.

After we’d fixed Aparna’s network issue, James came to see me at my desk. Masks on, socially distanced and all that, but it was nice to have some face-to-face time. James is cool – that dry British humor and not your classic IT Ops dude. He’s been here forever and mentored me when the CIO, Charlie, hired me as the first SRE here a year or so ago. I lucked out really.

Monitoring has always been a crucial operation in a software development cycle. This is mainly because of the complexity of industry-level IT and consumer-facing product development. Additionally, there is an ever-growing demand for rapid upgrades in products. To meet these requirements, streamlined performance and stability have become more important than ever; and without effective monitoring practices, they appear difficult to achieve.

Metrics measuring user engagement on your website are crucial for observability in marketing. Metrics will help marketing departments understand which of your web pages do not provide value for your business. Once known, developers can look at the web page’s technical metrics and determine if updates are required. Typically user engagement statistics, like the average time required to load your page, are stored separately from technical site logs.

Recently, we added a new derived column function to Honeycomb, INGEST_TIMESTAMP(), which can help customers debug event latency and/or inaccurate timestamps. A meaningful minority of the events sent to Honeycomb are already old when they arrive, and a very special few claim to have been sent from the future. Has this happened to you? Let’s do an experiment.

We are thrilled that Elastic has been recognized as a Leader and a Fast Mover in the 2021 GigaOm Radar Report for Cloud Observability. GigaOm Radar reports offer a forward-looking view of the market and are designed to help IT decision makers evaluate solutions with an eye to the future. Their analysts consider not just where the solutions are today, but also how the market is evolving and where the solutions are headed relative to that trend.