Apache Kafka is a distributed messaging system that can be used to build applications with high throughput and resilience. It is often used in conjunction with other big data technologies, such as Hadoop and Spark. Kafka-based applications are typically used for real-time data processing, including streaming analytics, fraud detection, and customer sentiment analysis. There are many derivatives such as Confluent Kafka, Cloudera Kafka, and IBM Event Streams.
Back in November 2021, Grafana Labs released version 2.4 of Grafana Loki. One of the new features it included was a Promtail Kafka Consumer that can easily ingest messages out of Kafka and into Loki for storing, querying, and visualization. Kafka has always been an important technology for distributed streaming data architectures, so I wanted to share a working example of how to use it to help you get started.