When our CEO and co-founder Tomer Levy delivered his “Observability is Broken” presentation at last year’s AWS re:Invent, he highlighted numerous challenges faced by today’s organizations as they seek to advance their observability practices. Of the six individual points that he noted, two specifically dealt with the current shortage of available engineering expertise, with another two focused on data overload.

Cribl Search is a powerful tool that is designed to enhance your data search efficiency, irrespective of the location of your data. This blog will explore how this tool seamlessly integrates with numerous Cribl Edge Nodes in real time, simplifying the process of discovery and troubleshooting. An integral part of Cribl Search is the “teleport” feature, which enables users to access specific Edge Nodes for in-depth analysis, simply by clicking on a host field.

Technology juggernauts–despite their larger staffs and budgets–still face the “cognitive load” for DevOps that many organizations deal with day-to-day. That’s what led Spotify to build Backstage, which supports DevOps and platform engineering practices for the creation of developer portals.

In a previous blog post, we built a small Python application that queries Elasticsearch using a mix of vector search and BM25 to help find the most relevant results in a proprietary data set. The top hit is then passed to OpenAI, which answers the question for us. In this blog, we will instrument a Python application that uses OpenAI and analyze its performance, as well as the cost to run the application.

In the rapidly evolving digital landscape companies are facing an increasing number of challenges in maintaining their IT infrastructure, and ensuring application stability. It is critical to stay on top of all the information to ensure the health of the organization and the business side of it. One of the ways to achieve visibility is to use a log monitoring tool to centralize the log data coming from each application and infrastructure element.

Firewall systems are critical for protecting your network and devices from unauthorized traffic. There are several types of firewalls that you can deploy for your environment via hardware, software, or the cloud—and they all typically fall under one of two categories: network-based or host-based. Network-based firewalls monitor and filter traffic to and from your network, whereas host-based firewalls manage traffic to and from a specific host, such as a laptop.

Across industries and geographies, businesses rely heavily on Systems Applications and Products (SAP) systems. These powerful and versatile systems streamline operations and manage critical data spanning areas like finance, human resources, and supply chain. However, the real-time monitoring of these systems, with an in-depth understanding of performance metrics and quick anomaly detection, is paramount for smooth operations and business continuity. It's here that our unique offering steps in.

In the data business, we often refer to the series of steps or processes used to collect, transform, and analyze data as “pipelines.” As a data scientist, I find this analogy fitting, as my concerns around data closely mirror those most people have with water: Where is it coming from? What’s in it? How can we optimize its quality, quantity, and pressure for its intended use? And, crucially, is it leaking anywhere?

Understanding Metrics, Logs, Events and Traces - the key pillars of observability and their pros and cons for SRE and DevOps teams.

As the affordable choice for cloud computing, Google Cloud Platform (GCP) is catching up to its competitors, like AWS and Microsoft Azure. As a business, you need the speed and scalability that the cloud provides, but you want to limit your costs to ensure you hit revenue targets. With GCP, you found a digital services business partner to help you meet your business objectives, a technology that gives you the service availability you want at the speed you need.

Follow these best practices to learn how to get the most out of your Google Cloud and third-party logging infrastructure for the lowest cost.

More platform teams owning multi-tenant systems need a full-stack observability solution that aggregates volumes of data into logs, metrics and traces. In tandem, there’s a growing number of major players in the observability industry, including New Relic. This post will compare some key features between Coralogix vs. New Relic. We will also go over what customers are looking for when choosing a complete observability platform.

As more organizations leverage the Amazon Web Services (AWS) cloud platform and services to drive operational efficiency and bring products to market, managing logs becomes a critical component of maintaining visibility and safeguarding multi-account AWS environments. Traditionally, logs are stored in Amazon Simple Storage Service (Amazon S3) and then shipped to an external monitoring and analysis solution for further processing.

Choosing an excellent application performance monitoring tool is a challenging task. Nowadays, there are dozens of instruments, and it can be problematic to pick the right one. However, when looking into every given “top ten list”, New Relic vs. Datadog will always be there. At this point, instead of focusing on dozens of log management tools, let’s focus on some key ones. Comparing New Relic vs. Datadog offers a distinct perspective on how infrastructure monitoring should look.

Datadog and Splunk are among the most popular performance monitoring tools available on the market. If you’re looking for such a solution and looking to scratch one off your shortlist, look no further than this article. In this Datadog vs Splunk comparison, we will take a deep dive into everything each tool has to offer. We will point out their similarities and differences to help you decide which tool can meet your needs better.

Meta, the parent company of Facebook, has been fined a record €1.2 billion ($1.3 billion) by the European Union for violating its data privacy laws. The fine was issued by Ireland’s Data Protection Commission, which is Meta’s lead regulator in the EU, and is the largest ever levied under the EU’s General Data Protection Regulation (GDPR), which went into effect in 2018.

Several years ago, there was little choice among performance monitoring tools. You had to deal with what the market offers. Datadog is one of the oldest solutions available and, thus, well-known. Yet, it is not without flaws, which might make people look for alternative solutions since the market is booming and new tools emerge regularly.

DevOps teams and security engineers use monitoring tools like Prometheus and Datadog to search for bugs and find any issues that might put an app or the entire IT infrastructure at risk. Better monitoring capabilities and aspects like event monitoring mean users can log data more effectively and engage in data collection leading to data visualization. These actions lead to infrastructure metrics, which allow experts to conduct timely analysis and prevent an app from crashing.

With Cribl Stream, our customers are experiencing choice and control over their data that would have been a pipe dream (or maybe I should say a pipeline dream) before. The ability to get the right data to the right destination in the right format is extremely powerful. Stream can optimize the data being sent to expensive destinations; you can remove unnecessary or redundant fields, drop unnecessary events, or even pull valuable metrics from verbose logs. Optimizing your data has a few benefits.

Growth of cloud computing and the preference for data-driven decision-making have led to a steady increase in investments in observability over the years. Telemetry data is recognized as not only critical for maintaining a company’s infrastructure, but also for aiding security and business teams in making informed decisions. However, just increasing investment in observability technology is not enough.

Remember the first time you were at a wedding, or a party and you learned about dances like The Electric Slide? You know, those dances with a clear structure and steps to follow, which were a huge help to someone who was slightly challenged on the dance floor, like me? All you had to do was learn a few simple steps, and you could hang with even the best dancers.

While more observability vendors are providing tracing ingestion and visualization as part of their core service, only Coralogix, the leading in-stream observability platform, supports a set of data optimization features that drive down cost, maximize insights and create a scalable tracing strategy unlike others.

The Elasticsearch reindex API copies data from one index to another. You can use reindex to change the index mapping, copy data to another cluster, or copy only a subset of data to another index. For example, suppose you want to reindex all the data in index1 into index2. In that case, you run the following example in Kibana dev tools: In this article, we dive into some common issues solved by reindexing as well as troubleshooting issues with reindexing itself.

Dr. Anton Chuvakin, a noted warrior/poet/security cybersecurity expert, sums up my thoughts about RSAC 2023 marketing messaging perfectly with this post on Twitter. For those who are new to the vendor hall, the amount of just bad marketing can be overwhelming and confusing. . There’s only one chance to get your message across to your prospects, so make it short and sweet. Anton’s guess of “zero click zero trust” is closer than you think to the truth.

Correlation in monitoring and observability refers to the process of analyzing different types of data to identify and understand relationships between application, network, and infrastructure behavior. Correlating these data sets can help IT teams identify all technology components contributing to or impacted by a performance or reliability issue, thereby empowering them to identify root cause and troubleshoot faster.

Because of where you’re reading this post, I’m going to assume you already know that Grafana is a great tool for visualizing and presenting metrics, and persisting them on dashboards. Ever since the Grafana Loki query builder for LogQL was introduced in 2022, it’s been easy to display and visualize logs, too.

As we’ve shown in previous blogs, Elastic® provides a way to ingest and manage telemetry from the Kubernetes cluster and the application running on it. Elastic provides out-of-the-box dashboards to help with tracking metrics, log management and analytics, APM functionality (which also supports native OpenTelemetry), and the ability to analyze everything with AIOps features and machine learning (ML).

AppSignal Logging gives you 360-degree insights into your application's performance. To help give you those insights, we wanted to ensure our logging solution allowed you to send AppSignal your logs your way. You can now use Winston transport to send your Node.js application's logs directly to AppSignal and take advantage of having access to all of your application's performance logs and metrics in one place.

Coralogix is a full-stack observability platform that effortlessly processes logs, metrics, traces, and security data. More specifically, logs in Coralogix are processed in larger volumes than almost any other observability provider out there, making a log’s life cycle unique. This article will examine the different stages of logs and help you better understand one of the most sophisticated telemetry processing architecture on the market.

As technology evolves in the enterprise, oftentimes the processes and tools used to manage it must also evolve. The increased adoption of Kubernetes has become a major inflection point for those of us in the monitoring and management side of the IT operations world. What has worked for decades (traditional infrastructure monitoring) has to be adjusted to the complexity and ephemeral nature of modern distributed systems where Kubernetes has a prime role.

The platform development team at Endeavor Streaming has a critical mission — from balancing operation of the company’s leading digital video platform, at scale, to ensuring everything in their complex cloud environment is performing as expected. Enabling the company to confidently build on top of its platform and continue to evolve their product delivery is thereby also dependent on maintaining detailed visibility into its supporting cloud applications and infrastructure.

We are excited to announce the release of Graylog 5.1! A follow-up to our 5.0 release, Graylog 5.1 brings updates across our entire product line, including changes to infrastructure, Security, Operations, and our Open offerings.

Adoption of Azure Functions in cloud-native applications on Microsoft Azure has been increasing exponentially over the last few years. Serverless functions, such as the Azure Functions, provide a high level of abstraction from the underlying infrastructure and orchestration, given these tasks are managed by the cloud provider. Software development teams can then focus on the implementation of business and application logic.

Network Data Analytics Function (NWDAF) is a key component in 5G networks, designed to collect, analyze, and deliver valuable insights to service providers. NWDAF provides an unbiased, vendor-vendor agnostic view of the network, expanding telco visibility beyond traditional use cases. As network complexities grow, service providers require unbiased and accurate data to make informed decisions, driving the demand for vendor agnostic data analytics.

Enterprises have enough data, in fact, they are overwhelmed with it, but finding the nuggets of value amongst the data ‘noise’ is not all that simple. It is bucket’d, blob’d, and bestrewn across the enterprise infrastructure in clouds, filesystems, and hosts machines. It’s logs, metrics, traces, config files, and more, but as Jimmy Buffett says, “we’ve all got ’em, we all want ’em, but what do we do with ’em”.

Cribl has been named to the 5th annual Enterprise Tech 30 (ET30) – a definitive list of the most promising, private enterprise tech companies. This is our first time on the ET30 list, ranking number four on the list of ten companies in the late stage category. The recognition highlights the value our innovative products deliver to our customers and partners as we work together to unlock the value of all observability data.

In the eternal hunt for elusive bugs, logging is an indispensable aid. By recording the events and messages that occur during the execution of your program, logging opens the door to unparalleled debugging and performance monitoring capabilities. It all starts with Python’s built-in logging module. However, the true power of Python logging is unlocked not merely by using it, but by mastering it.

There are many advantages to using dashboards that are powered by open-source technology that make them a compelling choice for many organizations. Below we will discuss some of the major benefits of using dashboards that are built with the help of open-source technology, along with examples of some of the leading use cases for which open-source technology has been utilized.

InfluxDB is a powerful tool for managing time-series data. It is widely used in industries such as IoT, finance, healthcare, and more. Using InfluxDB, you can query and store large amounts of data in real-time, making it easier to identify patterns, trends, and anomalies. InfluxDB dashboards provide a comprehensive overview of your system performance, metrics, and KPIs in real-time. You can customize these dashboards to meet your specific requirements.

Enterprises are accumulating more and more observability and security data in isolated silos, not much different than the dust and spare change under couches and chairs in your grandparent’s rarely-used living room. There is something of value in both examples, but the nature of the value is very unclear and hard to measure without a lot of effort.

Today's applications are incredibly intricate and interconnected, often relying on numerous third-party services and libraries. With this complexity comes an increased likelihood of things going wrong. However, an error doesn't usually announce itself with great fanfare and a detailed explanation. More often than not, it shows up as an unexplained crash, a suspicious slowdown, or a surprising output. Error logging shines a spotlight on these problems.

At Logz.io, we’re seeing a very fast pace of adoption for Kubernetes–at this point, it’s even outpacing cloud adoption, with companies running on-prem fully adopting Kubernetes in production. Why are companies going in this direction? Kubernetes provides additional layers of abstraction, which helps create business agility and flexibility for deploying critical applications. At the same time, those abstraction layers create additional complexity for observability.

As a big proponent of open source and all things open, I jumped at the opportunity to expand on Cribl Stream’s OpenTelemetry implementation. I’m happy to report that as of Cribl Stream 4.1, both our OpenTelemetry source and destination now support OTLP over HTTP!

Rich content like videos and graphics used to cause network congestion and long load times when all the content was stored on a centrally located server. Fortunately, Content Delivery Networks (CDNs) came to the rescue in the late 1990s, letting users load rich content from a location geographically closer to them and reducing load times by distributing a cached version of content across servers worldwide.

Third-party APIs and cloud based software as a service (SaaS) tools have become a cornerstone of modern enterprises. It is essential to monitor log data and optimize API performance. This will ensure that development teams provide the desired advantages to clients and users. To address this challenge, businesses can use an observability pipeline. It is a set of tools and processes that monitor and analyze data from various sources. That includes third-party APIs and SaaS tools.

The observability landscape is changing fast, as organizations look to deploy applications and separate themselves from competition at a breakneck pace. What are the trends organizations need to be aware of as they make sense of the landscape? Every year, we at Logz.io set out to answer this question by going right to the DevOps and observability practitioners on the front lines.

Managing observability data can feel like a juggling act. Modern cloud applications generate vast amounts of data, and quickly accessing the most important data is a fundamental step toward quickly gaining unobstructed visibility into your infrastructure and applications. Yet, when data volumes grow, complexity follows. Many observability users find it overwhelming to assess the critical data generated from their complex infrastructure and applications.

In this live stream discussion, angel investor Ross Haleliuk joins Cribl’s Ed Bailey to make a big announcement about his new fund to shape the future of the cybersecurity industry. Ross is a big believer in focusing on the security practitioner to provide practical solutions to common issues by making early investments in companies that will promote these values.

Over the past several years, one topic that has become of increasing importance for DevOps and site reliability engineering (SRE) teams is AIOps. Artificial intelligence for IT Operations (AIOps) is the application of artificial intelligence (AI), machine learning (ML), and analytics to improve the day-to-day operational work for IT operations teams.

In part 1 of this post, we talked about how Cribl is empowering security functions by giving our customers freedom of choice and control over their data. This post focuses on their experiences and the benefits they are getting from our suite of products. In a past life, I was in charge of security and operational logging at Transunion — around 2015, things started going crazy.

Elasticsearch is used for a wide variety of data types — one of these is metrics. With the introduction of Metricbeat many years ago and later our APM Agents, the metric use case has become more popular. Over the years, Elasticsearch has made many improvements on how to handle things like metrics aggregations and sparse documents. At the same time, TSVB visualizations were introduced to make visualizing metrics easier.

Event-driven architecture is an efficient and effective way to process random, high-volume events in software. Real-time monitoring of event-driven system architecture has become increasingly important as more and more applications and services move toward this architecture.

Enterprises are getting increasingly tired of feeling locked into vendors, and rightfully so. As soon as you put your observability data into a SaaS vendors’ storage, it’s now their data, and it’s difficult to get it out or reuse it for other purposes. As a result, strategic independence is becoming increasingly important as organizations decide what data management tools they’re going to invest time and resources into.

Cost optimization has been one of the hottest topics in observability (and beyond!) lately. Everyone is striving to be efficient, spend money wisely, and get the most out of every dollar invested. At Logz.io, we recently embarked on a very interesting and fruitful data volume optimization journey, reducing our own internal log volume by a whopping 50%. In this article, I’ll tell you how exactly we achieved this result.

We are caught in a whirlwind of rapid data change. As more engineers, services and sophisticated practices are helping generate an astronomical amount of digital information, there’s a growing challenge of the data explosion. Coralogix offers a completely unique solution to the data problem. Using Coralogix Remote Query, the platform can drive cost savings without sacrificing insights or functionality.

With governments doubling down on logging compliance, many public sector organizations have been focusing on optimizing their log management, especially to ensure they retain logs for required periods of time. Logs — though seemingly straightforward — are the backbone of many mission-based use cases and therefore have the potential to accelerate mission success when centrally organized and leveraged strategically. In public sector, logs are instrumental in.

Log events come in all sorts of shapes and sizes. Some are delivered as a single event per line. Others are delivered as multi-line structures. Some come in as a stream of data that will need to be parsed out. Still, others come in as an array that should be split into discrete entries. Because Cribl Stream works on events one at a time, we have to ensure we are dealing with discrete events before o11y and security teams can use the information in those events.