Apache Hive is an open source interface that allows users to query and analyze distributed datasets using SQL commands. Hive compiles SQL commands into an execution plan, which it then runs against your Hadoop deployment. You can customize Hive by using a number of pluggable components (e.g., HDFS and HBase for storage, Spark and MapReduce for execution). With our new integration, you can monitor Hive metrics and logs in context with the rest of your big data infrastructure.

Starting this week, we will do a series of four blogposts on the intersection of Spark with Kubernetes. The first blog post will delve into the reasons why both platforms should be integrated. The second will deep-dive into Spark/K8s integration. The third will discuss usecases for Serverless and Big Data Analytics. The last post will round off with insights on best practices.

A recent report by Gartner casts light into the world of AIOps, and the need for deploying it in organizations today. AIOps is a modern approach to DevOps which is based on recent AI technology. Gartner’s vision of the AIOps platform is one that enables continuous insights across IT operations management.

Today we are releasing the next public beta of Graylog v3.1. This release brings a whole new alerting and event system that provides more flexible alert conditions and event correlation based on the new search APIs that also power the views. In addition, some extended search capabilities introduced in Graylog Enterprise v3.0 are now available in the open source edition in preparation for unifying the various search features.

Apache Flume helps organizations stream large log files from various sources to distributed data storage like Hadoop HDFS. This article focuses on the features and capabilities of Apache Flume and how it can help applications efficiently process data for reporting and analytical purposes.

Apache Ambari is an open source management tool that helps organizations operate Hadoop clusters at scale. Ambari provides a web UI and REST API to help users configure, spin up, and monitor Hadoop clusters with one centralized platform. As your Hadoop deployment grows in size and complexity, you need deep visibility into your clusters as well as the Ambari servers that manage them. If issues arise in Ambari, it can lead to problems in your data pipelines and cripple your ability to manage clusters.

Monitoring is an essential aspect of any IT system. System metrics such as CPU, RAM, disk usage, and network throughput are the basic building blocks of a monitoring setup. Nowadays, they are often supplemented by higher-level metrics that measure the performance of the application (or microservice) itself as seen by its users (human beings on the internet or other microservices in the same or different clusters).

Five worthy reads is a regular column on five noteworthy items we’ve discovered while researching trending and timeless topics. This week, we define what data sprawl is and how organizations can cope with it effectively. Data sprawl—defined as the proliferation of data into endpoints, servers, applications, BYODs, operating systems, network environments, and even other geo-servers—can be a challenge to monitor and control.