Welcome back dear readers! It’s time for the 10th episode of the Biweekly Engineering newsletter!

It feels good that I have been running this newsletter for 10 episodes. Hopefully, hundreds more to come. :)

Let’s begin!

Pitfalls to avoid while building event-driven systems

#wix #eventdrivensystems #kafka #microservices #distributedsystems

While designing systems, specially in system design interviews where we don’t need to think in terms of actual implementation, it is very common to encounter this-

The service will persist that piece of data into a MySQL database, and emit a message to a Kafka cluster.

This sounds very simple and obvious. But here lies the biggest pitfall - writing a piece of data in MySQL, and emitting the same data as a message to Kafka are not atomic. One of them might fail, and you will end up in an inconsistent state in your system!

In this article, from Wix Engineering, the author discusses pitfalls that they faced while building even-driven systems. The article mentions 5 common pitfalls-

• Write to db and then fire event without atomicity

• Using event sourcing everywhere

• No context propagation

• Publishing events with large payloads

• Not handling duplicate events

The best part of the article is that it not only discusses the pitfalls, but also shares solutions for them. A great read without any doubt, specially if you have had exposure to event-driven systems.

Storing billions of messages at Discord

#discord #cassandra #distributedsystems

Discord is a very well-known platform for online communication, particularly famous among gaming communities. The platform grew quickly since its launch. It now has millions of users all over the planet with trillions of messages stored in its systems.

Today, I am sharing this old but popular blog post by Discord. In this interesting article, the authors discussed why they decided to use Apache Cassandra for storing messages, how they did it, and what are the future plans. As you can expect, Discord needed to scale its systems with the growing number of users and messages, which is why having a highly scalable database storage became super critical for the platform’s performance and scalability.

Discord started with MongoDB knowing that it would require to migrate to a different database at some point. When they hit 100M messages, expected issues started to appear. But how to decide which database to use? This is where engineers needed to look at their data, its query patterns, and the requirements for performance and scalability.

The article precisely discussed the story of migration to Cassandra for storing billions of messages. One thing I particularly loved - the authors showed us the data modelling for storing the messages. This was definitely one of the critical parts of designing the solution.

From good to better code reviews

#codereviews #pragmaticengineering

As software engineers, doing code reviews is a part of our daily responsibilities. More or less, every company has a pre-defined set of code review practices. Doing it well is super critical - failing to do so can have disastrous consequences.

Personally, when I started to do code review at the beginning of my career, it felt overwhelming. As a junior engineer, I was only familiar with reading code starting from some obvious entry point, and following along the chain of function calls. But in code reviews, you don’t always have an entry point to start reading from. And frequently, you don’t have the context. This is why it gets tricky, and I needed to learn doing code reviews over time.

In this article, Gergely shared his personal experience of how to make good code reviews better. There are also some resources linked in the post which give readers scope to explore more.

With that, we conclude this week’s episode. Thanks for reading.

Please do share the newsletter with your network if you find it useful. This is the subscription link: https://biweekly-engineering.beehiiv.com/subscribe 

Take care, and have a great week ahead!