Operations | Monitoring | ITSM | DevOps | Cloud

Apache Kafka has become the go-to platform for organizations handling high-throughput, real-time data streaming. Its ability to manage massive data volumes while ensuring reliability is second to none. However, as businesses grow and demand for data increases, scaling Apache Kafka isn’t always a walk in the park.

Diskless topics are proposed in KIP-1150, which is currently under community review. The examples in this article use "Inkless", Aiven's implementation of KIP-1150 that lets you run it in production. I joined Aiven as a Developer Advocate in May, shortly after the Kafka Improvement Proposal KIP-1150: Diskless Topics was announced, which reduces the total cost of ownership of Kafka by up to 80%!

Diskless Kafka splits storage from compute, delegating replication to cheap object storage and turning Apache Kafka Brokers into a stateless compute layer. It’s 100% Kafka, and 80% cheaper. But in the cloud, a cheaper underlying technology does not always mean you pay less. The cost varies significantly depending on the deployment model - SaaS or BYOC. In this article, we will learn why.

Your Kafka producer pushed a million messages yesterday. Nice. But can you tell if they all made it? Or why did latency spike at 2 PM? Producer metrics help you determine that. They expose how long messages take to send, whether messages are getting stuck, and whether retries are piling up. Let’s go over which ones help while debugging and how to monitor them.

In this article, we will be continuing our series of deep dives into KIP-405. Previously, we covered: Now, we turn our attention to the internals of the read and delete paths. Just like we did for the write and metadata, here we will also be focusing on Aiven’s battle-tested Apache-licensed KIP-405 plugin. What makes the read path particularly interesting is how it delivers latency comparable to local disk or memory systems despite leveraging external object storage—let's dive in!

KIP-1150 proposes a new class of topics in Apache Kafka that delegates replication to object storage. With disks out of the hot path, the usual pains— cluster rebalancing, hot partitions and IOPS limits are gone, while cross‑zone network fees and pricey local disks drop by as much as 80%.

In this post, we summarize the major changes in the recently officially released Apache Kafka 4.0.0 version. We will look at the most notable features compared to the previous versions and explain what these changes mean in real production environments and what improvements they can bring to your streaming infrastructure.

The idea behind KIP-405 is to simply store most of the cluster’s data in another service. As we covered in detail in the last article - it’s a simple-sounding idea that goes a very long way. This other server where the data gets stored is pluggable. KIP-405 was designed in such a way to make Kafka seamlessly extensible to store its data in any kind of external store through a solid interface.

Have you ever looked at your monitoring dashboard and wondered, "Why is my Kafka consumer lag spiking again?" It’s a common frustration. Consumer lag isn’t just an inconvenience—it’s a sign that something’s wrong with your data pipeline. When lag builds up, you're facing delayed data processing and the risk of system failures.

Apache Kafka is the backbone of modern data streaming architectures, enabling real-time data movement, stream processing, and event-driven applications at scale. It enables high-throughput messaging between data sources and analytics platforms, supports log aggregation, and facilitates scalable extract, transform, load (ETL) pipelines for continuous data transformation and storage.