Engineer Gleb Lesnikov on Why Splitting a Monolith into Microservices May Sink Your Business

Gleb Lesnikov has been developing the system behind Dodo Pizza, an international fast-food chain, since joining the company in 2015. Over the years, the local business turned into a company running over 1,600 restaurants. When Dodo expanded rapidly, Gleb had to take key technical decisions that allowed the company to scale and succeed. Looking back, he reflects on the period when the company moved to a microservices architecture and explains how he managed to solve problems that could have caused the company to collapse.

Q: How big is Dodo Pizza, and when did you decide to break up the original monolith architecture of the system?

We started building our own information system - Dodo IS back in 2011. It began as a small module for taking orders in a single pizzeria, built as a monolithic application in a single repository. Today it keeps running 1,500+ pizzerias, 160+ coffee shops, 84 backend services, and 8 mobile apps. It's written in C#/.NET, we use MySQL, and we host on both Microsoft Azure and Yandex Cloud — that's the only way to keep the system operational across all our markets. It matters because Dodo IS is the technological core of our business: it digitizes every stage of every order, from being taken at the website, kiosk, cash register or app, through preparation, delivery, and even ingredient storage and movement.

But after a series of major outages, it became clear the monolithic application just couldn't handle the load anymore. So we started rewriting it in 2016.

Q: What exactly was wrong with the monolith's architecture?

The architecture itself was actually good, but the implementation wasn't great: a lot of things were coupled, modularity was low, and the technology was outdated. There were many domains that had nothing to do with each other inside one application. For example, order taking and employee payroll were united, and any failure brought all of them down together. Feature delivery was slow, releases were painful, and the database was a single point of failure.

Q: 2016 was the peak of the trend when companies were switching from monolith to microservices. Did this shape your decision, and how the transition went?

Yes, you can say so: we decided to "saw apart" our monolith when it was fashionable to move from a monolith to microservices, or SOA — Service-Oriented Architecture.

We split the system into domains and separate services following a database-per-service model and immediately ran into a synchronization problem! For example, the state of an order had to propagate reliably and consistently. But we couldn't just pull out our Tracker service, which handles order tracking. Even if we moved the order status into its own database along with it, the services still living inside the monolith would still need to know what was happening with that order. At first we synchronized statuses over HTTP RPC, but we soon decided to move to an event-driven architecture with a message broker instead. In this architecture, each service publishes events when its own data changes, and the rest subscribe to those events and update their own storage, achieving eventual consistency.

The event-driven architecture was the fix. What happened next and what obstacles did you encounter?

After we moved from a monolithic architecture to a service-oriented architecture, we adopted RabbitMQ into our stack. There were three basic reasons for that: fan-out state transfer between services, internal queues for asynchronous commands inside applications, and asynchronous unit-of-work workflows.

Over time, as the platform grew, fan-out state transfer became the dominant load, which led us to our first problems. In RabbitMQ, a publisher sends a message to an exchange, and queues subscribed to that exchange each get their own physical copy of every message. We had a separate durable queue per service and per consumer, so the number of queues, N, became a write amplification factor — one published message gets multiplied and written to disk N times. Our busiest exchanges had around 30 consumers, so the load on the broker was effectively multiplied by 30.

Q: How did the outages actually happen?

There was one recurring cause behind many of our outages: one or two consumers slowing down. It played out like this: a couple of consumers slow down or hang because of database, code, or network issues. The backlog in the queue grows — and at roughly 1,000+ messages a second, that backlog explodes fast, since fan-out turns that into 30,000 messages a second. Failed messages get requeued, so we're essentially just shuffling our own messages around inside the broker. Millions of messages pile up, and the broker runs out of memory and switches to read-only mode. And then everything falls down.

Q: Apart from this specific problem, did a deeper architectural mismatch exist?

Yes! Our core mistake was that we used RabbitMQ not as a queue for distributing work, but as a bus for streaming. RabbitMQ's queues give you message-level semantics, ack, nack, reject, requeue, dead-letter, which were optimized for work-queue scenarios. Brokers like it are, in fact, a server-side cursor with a lot of sophisticated features, like routing. This is what made our use case so expensive.

Kafka works differently. In Kafka, a topic is an append-only log split into partitions. Within a consumer group, partitions are assigned to consumers without overlap: each partition is owned by at most one consumer, while a consumer may own multiple partitions. This model allows you to have as many consumers as you want, because they're just reading a read-only log. This is much cheaper.

Q: What were the results of the transition to Kafka?

We had lots of events reflecting individual transitions or actions in RabbitMQ. We call them keyed events, and it could be things like MoveToOvenStage or MoveToPreparationStage. They increased the number of event types, making reordering things and reconstructing state on the consumer side hard.

With Kafka, we moved toward entity events instead — one topic per entity's event stream. The idea comes from Adam Belamare's book Building Event-Driven Microservices. If we, for example, take a tracker.order event, the partition key matches the entity's ID — the order, in our case. Events with the same key land in the same partition, and consumers see them in write order within that partition. Each event is a full snapshot of the entity right after it changed, not just a transition. With that shift, we cut down a lot of the "straightening" logic our consumers needed — they no longer have to reconstruct state from a pile of transition events, they just subscribe to the stream.

Q: If you could go back to 2016, right before you started splitting the monolith, what would you tell yourself?

The major lesson is this: the messaging model has to match how you actually use it. RabbitMQ isn't a bad broker, but it is built for message-level work-queue semantics. We were using it as a streaming bus for fan-out state transfer to dozens of consumers and this mismatch nearly sank us operationally.

The same applies to splitting a monolith: SOA and microservices solved our coupling problem, but they immediately created a distributed state synchronisation problem. Following an architectural trend without understanding the operational model it forces on you is exactly how good intentions turn into repeated outages.