Over the last decade, log management has been largely dominated by the ELK Stack – a once-open source tool set that collects, processes, stores and analyzes log data. The ‘k’ in the ELK Stack represents Kibana, which is the component engineers use to query and visualize their log data stored in Elasticsearch. Sadly, in January 2021, Elastic decided to close source the ELK Stack, and as a result, OpenSearch was launched by AWS as an open source replacement.
OpenTelemetry (also abbreviated as OTEL) is an increasingly popular open-source observability platform under the Cloud Native Computing Foundation (CNCF), which is currently the most active project in the CNCF after Kubernetes. It was created to establish a unified and vendor-agnostic way for instrumenting, collecting, and exporting telemetry data for your system and application across traces, logs, and metrics.
The basic goal of log management is to make log data easy to locate and understand so that users can identify how their services are performing and troubleshoot more quickly. Logging as a Service, or LaaS, takes log management a step further by providing a solution that seamlessly scales and manages your log data via cloud-native architecture.
Today’s IT systems are ever more fragmented. It is commonplace to see polyglot systems, written in multiple programming languages, and using a plethora of tools and cloud services as infrastructure building blocks, whether data stores, web proxy or other functions. In this dynamic cloud-native realm, open standards and open specifications have become integral drivers of compatibility, collaboration, and convergence – the Three C’s of Open Standards, if you will.
SIEM has been a crucial component of security systems for nearly two decades. While there’s ample information on operating SIEM solutions out there, guidance on evaluating and managing them effectively is lacking. We’ve noticed many SIEM vendors are taking advantage of this dearth of knowledge and not providing customers with needed value for what they’re buying.
While log parsing isn’t very sexy and never gets much credit, it is fundamental to productive and centralized log analysis. Log parsing extracts information in your logs and organizes them into fields. Without well-structured fields in your logs, searching and visualizing your log data is near impossible.
When our CEO and co-founder Tomer Levy delivered his “Observability is Broken” presentation at last year’s AWS re:Invent, he highlighted numerous challenges faced by today’s organizations as they seek to advance their observability practices. Of the six individual points that he noted, two specifically dealt with the current shortage of available engineering expertise, with another two focused on data overload.
Technology juggernauts–despite their larger staffs and budgets–still face the “cognitive load” for DevOps that many organizations deal with day-to-day. That’s what led Spotify to build Backstage, which supports DevOps and platform engineering practices for the creation of developer portals.
The platform development team at Endeavor Streaming has a critical mission — from balancing operation of the company’s leading digital video platform, at scale, to ensuring everything in their complex cloud environment is performing as expected. Enabling the company to confidently build on top of its platform and continue to evolve their product delivery is thereby also dependent on maintaining detailed visibility into its supporting cloud applications and infrastructure.
At Logz.io, we’re seeing a very fast pace of adoption for Kubernetes–at this point, it’s even outpacing cloud adoption, with companies running on-prem fully adopting Kubernetes in production. Why are companies going in this direction? Kubernetes provides additional layers of abstraction, which helps create business agility and flexibility for deploying critical applications. At the same time, those abstraction layers create additional complexity for observability.
