Operations | Monitoring | ITSM | DevOps | Cloud

January 2020

Automate all the things: Terraform + Ansible + Elastic Cloud Enterprise

A sequel to our first post, Automating the installation of Elastic Cloud Enterprise with Ansible, this blog shows how to extend automation to cloud provisioning with Terraform. In the first post, we detailed how to deploy and configure Elastic Cloud Enterprise (ECE) across three availability zones in AWS using Ansible. However, the provisioning of the underlying EC2 instances and configuration of the security groups was all manual.

Elastic Common Schema .NET library and integrations released

The Elastic Common Schema (ECS) defines a common set of fields for ingesting data into Elasticsearch. A common schema helps you correlate data from sources like logs and metrics or IT operations analytics and security analytics. Further information on ECS can be found in the official Elastic documentation, GitHub repository, or the Introducing Elastic Common Schema article.

High availability Elasticsearch on Kubernetes with ECK and GKE

Elastic Cloud on Kubernetes (ECK) is an operator that allows you to automate the deployment of the Elastic Stack — including Elasticsearch, Kibana, and Elastic APM, Elastic SIEM, and more — using Kubernetes. By using this ECK, you can quickly and easily deploy Elasticsearch clusters with Kubernetes, as well as secure and upgrade your Elasticsearch clusters. It is the only official Elasticsearch operator.

Using the Elastic APM Java Agent on Kubernetes

Elasticsearch and the rest of the Elastic Stack are commonly used for log and metric aggregation in various environments, including Kubernetes. In addition, the Elastic Stack is frequently being used for uptime tracking, with Heartbeat, as well as Application Performance Monitoring (APM), with agents supporting common programming languages, including Java.

Getting AWS logs from S3 using Filebeat and the Elastic Stack

Logs from a variety of different AWS services can be stored in S3 buckets, like S3 server access logs, ELB access logs, CloudWatch logs, and VPC flow logs. S3 server access logs, for example, provide detailed records for the requests that are made to a bucket. This is very useful information, but unfortunately, AWS creates multiple .txt files for multiple operations, making it difficult to see exactly what operations are recorded in the log files without opening every single .txt file separately.

Bi-directional replication with Elasticsearch cross-cluster replication (CCR)

Elasticsearch cross-cluster replication (CCR) was released as a beta feature in Elasticsearch 6.5, and as a Generally Available (GA) feature in Elasticsearch 6.7. CCR allows multiple indices to be replicated to one or more Elasticsearch clusters. Replicating indices to additional Elasticsearch clusters solves several use cases, including high availability (HA) across datacenters, disaster recovery (DR), and CDN-like architectures to co-locate data closer to application servers (and users).

Elastic SIEM for home and small business: Beats on Mac

Hey, there. This is part six of the Elastic SIEM for home and small business blog series. If you haven’t read the first, second, and third blogs, you may want to before going any further. In the Getting started blog, we created our Elasticsearch Service deployment and started collecting data from one of our computers using Winlogbeat. In the Securing cluster access blog, we secured access to our cluster by restricting privileges for users and Beats.

Elastic on Elastic: Embracing our own technology

When making investments in our tech stack, we tend to have doubts about companies that don’t use their own products and services. At Elastic, we deploy the full suite of our technology across the enterprise. We do so because our technology not only works, but it makes us more efficient and flexible on so many levels. And it can do the same for you and your business, too.

Support ending for TLS 1.0/1.1 and unencrypted HTTP traffic to Elasticsearch Service on Elastic Cloud

Starting April 21, 2020, all requests to Elasticsearch Service on Elastic Cloud must use HTTP over TLS (HTTPS) with support for TLS 1.2. We’ve decided to make this change in the best interest of our users so we can ensure the security of data in transit and stay up to date with modern encryption, security protocols, and practices.

Elastic SIEM for home and small business: Beats on CentOS

Hey, there. This is part five of the Elastic SIEM for home and small business blog series. If you haven’t read the first, second, and third blogs, you may want to before going any further. In the Getting started blog, we created our Elasticsearch Service deployment and started collecting data from one of our computers using Winlogbeat. In the Securing cluster access blog, we secured access to our cluster by restricting privileges for users and Beats.

Mac system extensions for threat detection: Part 2

In the previous post, we covered some of the frameworks accessible by kernel extensions that provide information about file system, process, and network events. These frameworks included the Mandatory Access Control Framework, the KAuth framework, and the IP/socket filter frameworks. In this post, we will go into the various tips and tricks that can be used in order to obtain even more information regarding system events.

Embracing offensive tooling: Building detections against Koadic using EQL

This year at BSidesDFW, my local security conference, I highlighted a continuing trend of adversaries using open source offensive tools. The talk reviewed one of these post-exploitation frameworks named Koadic and walked through different ways defenders can build behavioral detections through the use of Event Query Language (EQL).

Control the phase transition timings in ILM using the origination date

As part of Elasticsearch 7.5.0, we introduced a couple of ways to control the index age math that’s used by index lifecycle management (ILM) for phase timings calculations using the origination_date index lifecycle settings. This means you can now tell Elasticsearch how old your data is, which is pretty handy if you’re indexing data that’s older than today-days-old.

A tour of Go concurrency patterns via the new Heartbeat scheduler

Curious about how to write more idiomatic concurrent code in Go? It’s not always easy or intuitive, even if you’ve done lots of concurrent programming in other languages. I’ve been lucky to have worked in a well-written code base, and had the expert advice of Beats core area lead Steffen Siering along the way. In this post I’ll walk you through how we implemented a new scheduler for Heartbeat that is part of our upcoming 7.6.0 release.

How KeyBank used the Elastic Stack to build an enterprise monitoring solution

KeyBank is one of the largest banks in the United States. And as the bank has grown, so has their end-to-end monitoring system. With more than 1,100 branches and 1,400 ATMs stretching across 15 states, KeyBank’s infrastructure had evolved into a “Noah’s Ark of design,” says Mick Miller, Senior Product Manager, Cloud Native at KeyBank. In other words, they had two of everything, resulting in 21 different data islands.

Mac system extensions for threat detection: Part 1

When it comes to having visibility and detecting threats on macOS, one of the best sources of information for file system events, process events, and network events is the kernel. MacOS kernel extensions provide the ability to receive data about these events in real time with great detail. This is good for providing quick visibility into detecting anomalies and identifying possible threats.