Operations | Monitoring | ITSM | DevOps | Cloud

Finding yourself migrating and weighing the options when it comes to moving your entire Apache Kafka setup to a new cloud? This blog will point you to a solution to efficiently migrate your workloads to an Aiven for Apache Kafka cluster circumventing the headache that usually comes with cloud migration.

Diskless topics reward high-throughput workloads with large batches but can struggle with low-throughput patterns. Note: This analysis is based on testing with Diskless Kafka 4.0.0-rc15. Diskless topics are available for you to start experimenting with via the Inkless fork but the feature is still in development, and performance characteristics may change significantly as the technology matures. If you're: This post is for you!

You're running Kafka on Confluent Cloud. You care about lag, throughput, retries, and replication. But where do you see those metrics? Confluent gives you metrics, sure, but not all in one place. Some live behind a metrics API, others behind Connect clusters or Schema Registries. You either wire them manually or give up. What if you could stream those metrics to a platform built for high-frequency, high-cardinality time series, and do it in minutes?

In the cloud, Kafka’s promise of “never lose a byte” quietly morphs into “always pay for two.” Every time the leader syncs followers across zones, you get hit with premium egress charges that can dwarf compute costs. Diskless Kafka turns that upside-down: brokers replicate data straight into S3, so the pricey cross-zone hops vanish. Yes, object storage is slower than a local SSD, but the swap buys you on-demand elasticity and a bill that finally makes sense.

Apache Kafka has become the go-to platform for organizations handling high-throughput, real-time data streaming. Its ability to manage massive data volumes while ensuring reliability is second to none. However, as businesses grow and demand for data increases, scaling Kafka isn’t always a walk in the park. It often comes with its own set of challenges that can throw even the most seasoned teams for a loop.

Apache Kafka has become the go-to platform for organizations handling high-throughput, real-time data streaming. Its ability to manage massive data volumes while ensuring reliability is second to none. However, as businesses grow and demand for data increases, scaling Apache Kafka isn’t always a walk in the park.

Diskless topics are proposed in KIP-1150, which is currently under community review. The examples in this article use "Inkless", Aiven's implementation of KIP-1150 that lets you run it in production. I joined Aiven as a Developer Advocate in May, shortly after the Kafka Improvement Proposal KIP-1150: Diskless Topics was announced, which reduces the total cost of ownership of Kafka by up to 80%!

Diskless Kafka splits storage from compute, delegating replication to cheap object storage and turning Apache Kafka Brokers into a stateless compute layer. It’s 100% Kafka, and 80% cheaper. But in the cloud, a cheaper underlying technology does not always mean you pay less. The cost varies significantly depending on the deployment model - SaaS or BYOC. In this article, we will learn why.

Your Kafka producer pushed a million messages yesterday. Nice. But can you tell if they all made it? Or why did latency spike at 2 PM? Producer metrics help you determine that. They expose how long messages take to send, whether messages are getting stuck, and whether retries are piling up. Let’s go over which ones help while debugging and how to monitor them.