Operations | Monitoring | ITSM | DevOps | Cloud

The way we handle massive volumes of data from multiple sources is about to change fundamentally. The traditional data processing systems don’t always fit into our budget (unless you have some pretty deep pockets). Our wallets constantly need to expand to keep up with the changing data veracity and volume, which isn’t always feasible. Yet we keep doing it because data is a commodity.

We often get questions like: And while the 14-year-old in me is proud to say that we’ve done 24/7 support for clusters of 1000+ nodes holding many PB of data, I am quick to add that.

As we venture deeper into the data-driven era, the traditional systems we have employed to store, search, and analyze data are being challenged by revolutionary advancements in Artificial Intelligence. One such groundbreaking development is the notable advent of Large Language Models (LLMs), specifically those with Multi-Mod[a]l abilities (e.g., Image & Audio).

OpenSearch is a powerful, open-source analytics and search engine that can be utilized to construct custom search solutions for a broad variety of applications, from websites to enterprise-level systems. It enables flexible search and indexing abilities, making it suitable for a range of uses, a great example of this is scalability. OpenSearch is designed for horizontal scalability, enabling organizations to input additional nodes to their cluster as data volumes and query loads increase.

In today’s world of relentless data growth, security-relevant logs represent a small snapshot of an organization’s overall environment. Teams are beset with a variety of data types, including performance metrics and traces, asset configuration and state, audit logs, and much more. On top of that, teams are expected to scan all of this to compare against industry best practices and join this data with logs and metrics for added context.

Splunk is pleased to announce the general availability of Federated Search for Amazon S3, a new capability that allows customers to search data from their Amazon S3 buckets directly from Splunk Cloud Platform without the need to ingest it. Enterprises rely heavily on cloud object storage services as the de facto destination for their new data to leverage the cost, compliance, security, scalability and manageability benefits that cloud platforms can offer.

If you’re thinking of running OpenSearch on Kubernetes, you have to check out the OpenSearch Kubernetes Operator. It’s by far the easiest way to get going, you can configure pretty much everything and it has nice functionality, such as rolling upgrades and draining nodes before shutting them down. Let’s get going 🙂