Hulu, the entertainment streaming platform, needed a solution to scale up its internal application and infrastructure monitoring platform as it grew beyond 1 million metrics per second. The solution it created combines two open source tools— InfluxDB, a time series database, and Kafka, an event-streaming platform. It’s not just global enterprises like Hulu that have access to world-class tools and infrastructure to achieve their business goals.

Apache Kafka is known for its ability to handle real-time streaming data with speed and efficiency. It’s also known for being scalable and durable, which makes it ideal for complex, enterprise-grade applications. Of course, those new to the concept behind Kafka may find that it takes some time to understand how it works. Thanks to its unique combination of messaging, storage, and stream processing features, Kafka is well suited for both real-time and historical data analysis.

What if Aiven doesn’t provide the Apache Kafka® connector you want? Read on to learn how to use an external connector, to gather Twitter messages into Kafka.

Aiven for Apache Kafka® What if Aiven doesn't provide the Apache Kafka® connector you want? Read on to learn how to use an external connector, to gather Twitter messages into Kafka.

We noticed this new announcement for Kafka users which may be worth looking into: “Instaclustr is pleased to announce the immediate availability of a new Kafka add-on, Karapace Schema Registry, in public preview, on our managed platform. This follows our earlier announcement of Karapace as part of our comprehensive Apache Kafka support solutions in mid March.

Aiven for Apache Kafka® Moving the data infrastructure of your service or app from an on-prem model into a managed cloud service comes with risks. These ones you can't ignore.

The world lives by processing the data. Humans process the data – each sound we hear, each picture we see – everything is data for our brain. The same goes for modern applications and algorithms – the data is the fuel that allows them to function and provide useful features. Even though such thinking is not new, what is new in recent years is the requirement of near-real-time processing of large quantities of events processed by our systems.

At Grafana Labs, we’re continuing to expand our platform of Grafana Cloud integrations that make it easier than ever to connect and monitor external systems. These integrations enable you to answer the big picture questions in your organization and tell your observability story.

In Part 1 of this series, we started to compare the uses of Kafka and MQTT within an IoT infrastructure. It was concluded that in a basic publish-and-subscribe model of an IoT device, Kafka might simply be overkill.