Coralogix can be configured to automatically and dynamically archive logs to an S3 bucket. This saves Coralogix customers money, but of course there are times when the data needs to be reindexed. This operation counts the reindexed logs against the daily quota. Many times customers would like to search and focus on the exact logs to be reindexed or even query the logs outside of Coralogix all together.

Today I’m happy to announce that observIQ’s Stanza Log Agent will become a key part of the OpenTelemetry project. This has been in the works for many months and the team at observIQ is thrilled to see it becoming a reality. We’re particularly pleased to see it happening just as we launch our log management platform which will be the first platform to take full advantage of the log agent technology now incorporated into OpenTelemetry.

Even before new hybrid workforce models, many companies already moved a lot of services to the cloud. COVID-19 digital transformation strategies instantly increased the number of access points and endpoints. This led to a rapid increase in event log data followed by all kinds of other issues -- performance, availability, security, and ultimately increased IT costs amongst other things. A centralized log management solution for your cloud environment can help you manage the above and more.

Syslog takes its name from the System Logging Protocol. It is a standard for message logging and has been in use for decades to send system logs or event messages to a specific server, called a Syslog Server.

In a previous post, we went through a few input plugins like the file input plugin, the TCP/UDP input plugins, etc for collecting data using Logstash. In this post, we will see a few more useful input plugins like the HTTP, HTTP poller, dead letter queue, twitter input plugins, and see how these input plugins work.

So you've set up a Google Cloud Logging sink along with a Dataflow pipeline and are happily ingesting these events into your Splunk infrastructure — great! But now what? How do you start to get meaningful insights from this data? In this blog post, I'll share eight useful signals hiding within Google Cloud audit logs that will help you uncover meaningful insights. You'll learn how to detect: Finally, we’ll wrap up with a simple dashboard that captures all these queries in one place.

Intercept X is Sophos’ endpoint security solution, including anti-ransomware, zero-day exploit prevention, plus managed endpoint defense and response. It employs a layered approach reliant on multiple security techniques for endpoint detection and response (EDR). Those tactics include app lockdown, data loss prevention, web control and malware detection.

Recently, I presented at .conf20, Splunk’s annual user conference, on link analysis, where I promised more technical details on the topic in the coming weeks. To keep my promise, I’ve started a three-part series to show you how to use Splunk for link analysis.

Mediatonic is a sprawling video game studio based in the UK, with a number of successful titles to their name: Heavenstrike Rivals, Gears POP!, and Murder by Numbers among them. In 2020, they struck gold again with Fall Guys: Ultimate Knockout. But this game would be special, and the need of handling these kinds of gaming logs at this kind of scale would be, too. This battle royal-style fighting game pits 60 players against each other until one reigns supreme.

The recent changes to the Elasticsearch license could have consequences on your intellectual property. On the 14th of January 2021, Elastic announced through their blog that Elasticsearch and Kibana will be moving over to a Server Side Public License (SSPL). This license change, effective from Elasticsearch version 7.11, has business owners that rely on the ELK stack rightly concerned.

Earlier this week, I wrote a blog stating our intention to fork Kibana and Elasticsearch. This was a huge decision on our end, one that we did not take lightly. A few days have passed since this announcement and I wanted to share how humbled and excited we are with the responses from companies and individuals who are eager to participate and contribute.

Although AWS Lambda is a blessing from the infrastructure perspective, while using it, we still have to face perhaps the least-wanted part of software development: debugging. In order to fix issues, we need to know what is causing them. In AWS Lambda that can be a curse. But we have a solution that could save you dozens of hours of time. TL;DR: Dashbird offers a shortcut to everything presented in this article.

Three years ago, when we released Embedded Views into beta, we were excited to enable customers to share log lines in a customized way outside of our web application. However, it's become clear that the vast majority of our users prefer having the full functionality of our web application when exploring their logs rather than using an Embedded View.

Let’s say you get an alert that one or more queries is slow. Or that your users complain, whichever comes first 🙂 We’ve all been there… How do you find the root cause for this slowness and then fix it? In this article, I’ll go through my usual thought process: first, I’d try to find which queries are slow. Then, I’d dig deeper: Let’s take a specific example and run through each step.

A couple of days ago, Elastic announced that it will change the licensing of Elasticsearch and Kibana as of the 7.11 release to a proprietary dual license (under the SSPL license) and away from the open-source Apache-2.0 license. This move has caused extensive turmoil and frustration in the open-source community, especially with organizations that rely on Elasticsearch. Let me start with the end in mind.

As the Trump Administration comes to a close, there is no better time than the present to reexamine the Department of Defense Digital Modernization Strategy and its potential sustainment beyond January 2021.

Centralized Log Management offers the visibility you need to optimize your cloud usage to keep infrastructure costs down. Cloud-first infrastructures are the future of modern business operations. As organizations like Google and Twitter announce long-term plans for enabling a remote workforce, maintaining a competitive business model includes scaled cloud services adoption. While the cloud offers scalability that can save money with pay-as-you-need services, managing the costs is challenging.

Network security has changed a lot over the years, it had to. From wide open infrastructures to tightly controlled environments, the standard practices of network security have grown more and more sophisticated. This post will take us back in time to look at the journey that a typical network has been on over the past 15+ years. From a wide open, “chewy” network, all the way to zero trust networking. Let’s get started.

In a previous post, we explored the basic concepts behind using Grok patterns with Logstash to parse files. We saw how versatile this combo is and how it can be adapted to process almost anything we want to throw at it. But the first few times you use something, it can be hard to figure out how to configure for your specific use case.

We recently announced a license change: Blog, FAQ. We posted some additional guidance on the license change this morning. I wanted to share why we had to make this change. This was an incredibly hard decision, especially with my background and history around Open Source. I take our responsibility very seriously. And to be clear, this change most likely has zero effect on you, our users. It has no effect on our customers that engage with us either in cloud or on premises.

AWS Lambda is a serverless compute service that runs your code in response to events and automatically manages the underlying compute resources for you. The code that runs on the AWS Lambda service is called Lambda functions, and the events the functions respond to are called triggers. Lambda functions are very useful for log collection (think of log arrival as a trigger), and Coralogix makes extensive use of them in its AWS integrations.

Do you have an application that’s a little… sluggish? Cloud Profiler, Google Cloud’s continuous application profiling tool, can quickly find poor performing code that slows your app performance and drives up your compute bill. In fact, by helping you find the source of memory leaks and other errors, Profiler has helped some of Google Cloud’s largest accounts reduce their CPU consumption by double-digit percentage points.

With Kinesis Firehose being Splunk’s preferred option when collecting logs at scale from AWS Cloudwatch Logs, we’ve seen plenty of posts on setting this up, automation and examples on transforming event content. But what about when things go wrong?

Logz.io has recently launched its Smart Tiering solution, which gives you the flexibility to place data on different tiers to optimize cost, performance and availability. Our mission has been to make Smart Tiering a multi-cloud and multi-region service. As part of this launch, we are glad to announce that the Historical Tier now supports Microsoft Azure Blob Storage, alongside AWS S3.

In this article I’m going to discuss table joins and the let statement in Log Analytics. Along with custom logs, these are concepts that really had me scratching my head for a long time, and it was a little bit tricky to put all the pieces together from documentation and other people’s blog posts. Hopefully this will help anyone else out there that still has unanswered questions on one of these topics.

In this article, I’m going to discuss custom logs in Log Analytics. Along with table joins and the let statement that I discuss in another blog, custom logs is a concept that I struggled to wrap my head around for a long time, as there don’t seem to be very many comprehensive guides out there as of yet. Here is a summary of everything I have managed to piece together from documentation and other people’s blog posts.

Grafana Cloud is the easiest way to get started observing metrics, logs, traces, and dashboards. When we say “easiest,” we mean it: Grafana Cloud is designed so that even novice observability users can use it. As a new user, you are not required to dive into the complexity of setting up Prometheus and figuring out how to create Grafana dashboards from scratch. Integrations are the reason why.

Elastic solutions solve many different business challenges from powering search bars to creating observable systems to detecting and responding to threats. And with the amount of capabilities each offers, learning how to maximize the power of our solutions for enterprise search, observability, and security is critical to realizing Elastic's full value. But finding the time to build new skills can be challenging.

I hope I’m not alone in starting 2021 with some sense of optimism. While several hard months remain ahead of us, I am hopeful and also expecting that some sense of normality will return by the summer months. Either way, this gives us an opportunity to reflect on the challenges we have faced. 2020 was testing. We learnt a lot about ourselves and our businesses in the most challenging of circumstances.

Syslog is a popular standard for centralizing and formatting log data generated by network devices. It provides a standardized way of generating and collecting log information, such as program errors, notices, warnings, status messages, and so on. Almost all Unix-like operating systems, such as those based on Linux or BSD kernels, use a Syslog daemon that is responsible for collecting log information and storing it.

Oftentimes users of open source are told to go download it and figure it out… or pay for a managed solution in the cloud. So the typical choice is free and do-it-yourself or expensive and easy. With our new changes to Grafana Cloud, we are making it both free and easy to have a real, composable observability solution.

In my ongoing Loki how-to series, I have already shared all the best tips for creating fast filter queries that can filter terabytes of data in seconds and how to escape special characters. In this blog post, we’ll cover how to use metric queries in Loki to aggregate log data over time.

As with many things in 2020, this year’s AWS re:Invent was quite different from any previous iterations. For starters, instead of a week of live talks, face-to-face sessions, and a room full of booths, this year the event was fully online and stretched out for three weeks. As sponsors of this year’s event, we were excited to participate and continue to make an impact on the AWS community.

Fraud rates for Unemployment Insurance Benefits (UIB) and Pandemic Unemployment Assistance (PUA) are out of control. In May 2020, Brian Krebs of Krebsonsecurity published two articles detailing fraud that was occurring in several different state’s UIB portals. These states had been warned by the US Secret Service to be on the lookout for this. Reading the articles, the common theme is that many states are missing rudimentary controls for combating fraud.

A large amount of data requires special tools. Apache Cassandra is one of those databases that can handle a large amount of data spread among many commodity servers, providing high availability and fault tolerance without a single point of failure. Developed under the umbrella of Apache Software Foundation, it ensures full visibility into the code base and being free of charge.

In our last blog, "What's New in Splunk Cloud: Part 1," we reviewed a host of new Splunk Cloud features that we have delivered through our accelerated releases since the beginning of 2020. A large part of this effort focused on empowering Splunk Cloud admins and making their experience as self-service as possible. In this blog, we will examine our latest effort to continue this empowerment: Splunk Cloud’s Admin Configuration Service (ACS).

Now, you can ship ModSecurity logs to Logz.io to automatically surface high-priority attacks identified by ModSecurity. Logz.io will automatically parse those logs to project a greater bird’s-eye-view of your security situation within dedicated dashboards. ModSecurity is a prolific web application firewall (WAF) popularly used to help secure web servers. It supports Apache HTTP, IIS, and NGINX. It can deploy either as a proxy server or within a web server itself.

Elastic Cloud puts the power of the Elastic Stack in your hands within minutes. Whether you’re trying to add search capabilities with Elastic Enterprise Search, monitor critical systems and applications with Elastic Observability, or protect your organization from cyber threats with Elastic Security, taking the first step is easy.

Choosing whether to go with PostgreSQL or MySQL depends on your needs as they are both great databases to use under different circumstances. In this article we will run through a few of the top reasons and use cases to help you choose between these choices for database creation. Note: As a matter of fact, MySQL is so popular it became part of the LAMP stack (Linux, Apache, MySQL, PHP) used for building many web servers.

When you’re troubleshooting an app or a deployment, every second counts! Cloud Logging helps you troubleshoot by aggregating logs from across Google Cloud, on-premises or other clouds, indexing, aggregating logs into metrics, scanning for unique errors with Error Reporting and making logs available for search, all in less than a minute. And now, we’ve built two new features for streaming logs to give you even fresher insights from your logs data.

We created the Elastic Contributor Program to encourage knowledge sharing in our community and to recognize and reward the hard work of our awesome contributors. There are six different contribution types accepted in the program: event organization, presentation, written content, video, translation, and code. In this blog post, we’ll cover how to contribute code in the many free and open projects that Elastic maintains.

With 2020 dominated by a global pandemic, organizations expedited their digital transformation strategies. (According to TechFirst podcast, COVID19 accelerated digital transformation by an average of 6 years.) One of the most significant changes was the rapid move to a remote workforce. This required stopgap measures to keep the business running. While these measures met the company’s immediate needs, the measures also introduced anticipated and unanticipated issues.

In my ongoing Loki how-to series, I have already shared all the best tips for creating fast filter queries that can filter terabytes of data in seconds. In this installment, I’ll reveal how to correctly escape special characters within a string in Loki’s LogQL. When writing LogQL queries, you may have realized that in multiple places you have to write strings delimited by double quotes.

This article was originally published on InfoQ at December 3rd 2020. If you’ve migrated from a monolith to a microservices architecture you probably experienced it: Modern systems today are far more complex to monitor. Microservices combined with containerized deployment results in highly dynamic systems with many moving parts across multiple layers.

Increasingly, we are seeing on-prem workloads being moved onto the cloud. Elasticsearch has been around for many years with our users and customers typically managing it themselves on-prem. Elasticsearch Service on Elastic Cloud — our managed Elasticsearch service that runs on Amazon Web Services (AWS), Google Cloud, and Microsoft Azure across many different regions, is the best way to consume the Elastic Stack and our solutions for enterprise search, observability, and security.

Today we are excited to announce Graylog Illuminate v1.4. This release includes the addition of Office 365 content, which provides deeper visibility into Azure Active Directory and Exchange Online logs along with new alerts for a more granular level of notifications.

In days gone by, highly regulated industries like pharmaceuticals and finance were the biggest targets for nefarious cyber actors, due to the financial resources at banks and drug companies’ disposal – their respective security standards were indicative of this. Verizon reports in 2020 that, whilst banks and pharma companies account for 25% of major data breaches, big tech, and supply chain are increasingly at risk.

One of the keys to CloudWatch’s success is its no bang, no buck billing system. The pricing structure has been designed from the outset to ensure that CloudWatch users only pay for what they actually use. In addition, the CloudWatch Free Tier allows first time users to test the waters without shelling out. The downside of this flexibility and adaptability is complexity.

Prometheus metrics are an essential part of your observability stack. Observability comes hand in hand with monitoring, and is covered extensively here in this Essential Observability Techniques article. A well-monitored application with flexible logging frameworks can pay enormous dividends over a long period of sustained growth, but Prometheus has a problem when it comes to scale.