Hey there dear readers! Hope life has been going well for you all. I am glad to be back once again with yet another episode of Biweekly Engineering!

In today’s episode, we have engineering blog posts from Agoda and Myntra, along with an article on stream processing.

Let’s read!

How Myntra Used Dynamic Programming at Checkout

As a software engineer or a computer science student, you surely have heard of Dynamic Programming. No wonder it crossed your mind - do we really need dynamic programming in real-life engineering?

Well, in general, algorithms and data structures are always used in software engineering. But as an engineer, you will not always need to be hands on with them. There are always some libraries or frameworks taking care of all the nitty-gritty stuff. But for sure, it would be really nice to use different algorithms and data structures to solve real-life problems, wouldn’t it?

Turns out, that is exactly what Myntra did for its checkout offers. Myntra is an India-based e-commerce platform for fashion and lifestyle. They have an offer called Buy-n-Get-x, where users, just before checkout, can choose to pay for n products from the cart in non-discounted prices and get x products for free. They can also choose to buy all the products in the discounted price without taking the Buy-Get offer.

So during checkout, every product has two prices - discounted price and original price. Now, users need to carefully pick n products to buy in original prices so that they can get x products for free, while making sure they are saving money. As you can possibly tell, this is a problem where users have many combinations to choose from, and they would want to take the most optimal combination which would save the highest amount of money for them. This is where Myntra used Dynamic Programming to pick the best cost-saving combination for its users.

For details, don’t hesitate to check the original article. It is refreshing to see dynamic programming in action in real-life!

Trillion-Scale Kafka Platform at Agoda

Like many other companies in the world, Agoda uses Kafka for its events streaming. According to the article, in 2021, they have processed 1.5 trillion events daily in its Kafka platform.

So how did Agoda achieve this?

In short, the design has a 2-step logging mechanism:

• Every application has a client library that logs the events to the local disk.

• Every node runs a daemon process (the Forwarder) that reads the events from the disk and sends them to Kafka.

The advantage of this 2-step process is that it is resilient, reliable, and customisable by the platform team. As a result, application developers don’t need to bother about the management of the Forwarder. They just write the events using the client library, and the Forwarder takes care of the rest.

Of course, there is also a disadvantage. Being a 2-step mechanism makes it slower to write events. The latency for 99th percentile is 10 seconds. But this can be further tuned to 1 second.

Given that it is slow, 2-step logging is only used for analytical events which don’t need sub-second latency. For use cases where sub-second latency is required, the write to disk is skipped. Such applications can directly write to Kafka using Kafka clients.

The article also discusses how the team handled monitoring, auditing, scaling, and cost reduction.

Introduction to Stream Processing

Have you heard of stream processing, but not sure what this is about? Now it’s time to learn! In this wonderful article, the author gives us a thorough introduction to stream processing.

Being a big data technology, stream processing enables us to react to continuous stream of data in very short time. As the author mentions-

The opposite of stream processing is batch processing, where data is at rest. A batch job triggers on schedule, reads the data, processes it, and then outputs results for downstream systems to consume. More and more companies are using stream processing now to build real-time systems.

But as you can expect, don’t throw stream processing for every use case you have. The article also nicely discusses where stream processing makes sense and where it doesn’t.

If you have always been curious of stream processing but don’t know where to start, this article is a perfect starting point.

And that’s a wrap! As always, I hope you have learnt something new in this episode. Make some room from your daily schedule to go through the articles shared in the newsletter. Don’t forget that reading engineering blogs is one of the best ways to grow as an engineer. And Biweekly Engineering was created to help you, the readers, to pursue the growth!

Till next time, adios! 👋