Operations | Monitoring | ITSM | DevOps | Cloud

In this lesson, we’ll learn how we can use Elasticsearch Hadoop to process very large amounts of data. For our exercise, we’ll use a simple Apache access log to represent our “big data”. We’ll learn how to write a MapReduce job to ingest the file with Hadoop and index it into Elasticsearch.

Prometheus is the cornerstone of many monitoring solutions, and sooner or later, prometheus federation will appear on your radar. A well monitored application with flexible logging frameworks can pay enormous dividends over a long period of sustained growth. However, once you begin to scale your prometheus stack, it becomes difficult to keep up with your application’s demands. Prometheus is an extremely popular choice when it comes down to collecting and querying real-time metrics.

Modern day Security Information and Event Management (SIEM) tooling enterprise security technology combine systems together for a comprehensive view of IT security. This can be tricky, so we’ve put together a simple SIEM tutorial to help you understand what a great SIEM provider will do for you. A SIEM’s responsibility is to collect, store, analyze, investigate and report on log and other data for incident response, forensics and regulatory compliance purposes.

Availability and quality are the biggest differentiators when people opt for a service or product today. You should be aware of the impact of your customers alerting you to your own problems, as well as how to stop this from becoming the norm. To make sure you don’t become an organization known for its bugs, understanding the organizational changes required to deliver a stable service is key.

Across the technology and IT infrastructure domain, log files are recognized as often time-stamped files that can virtually record all critical information about events occurring within the purview of your IT network, OS, or other software applications. Some log files are humanly interpretable, while others are largely meant for machines to consume.

While all cybersecurity professionals agree that log management is integral for robust proactive and reactive security, managing the enormous amount of data logs can be a challenge. While you might be tempted to collect all logs generated from your systems, software, network devices, and users, this “fear of missing out” on an important notification ultimately leads to so much noise that your security analysts and threat hunters cannot find the most important information.

We’re excited to announce the public beta of the LogDNA Terraform Provider, allowing organizations to manage Views and Alerts programmatically via Terraform. Today, more teams than ever are adopting Infrastructure as Code (IaC) to reduce human error and create efficiently scaled workflows for their infrastructure. Additionally, teams are looking to bring the same benefits of scalability and predictability into their SaaS-based observability stack.

Today is the last day of ObservabilityCON 2020! We hope you’ve had the chance to catch the talks so far, and will tune in live for today’s sessions. View the full schedule on the event page, and for additional information on viewing, participate in Q&As, and more, check out our quick guide to getting the most out of ObservabilityCON. If you aren’t up-to-date on the presentations so far, here’s a recap of day three of the conference.