Hello there good people! We are back with the second issue of our Biweekly Engineering Blog Reading List! Hope you had fun reading the articles shared two weeks ago. If you missed that, find it here-

In this week's issue, we will read articles from Medium, Netflix, Grab, and Monzo. Let's go! 🚴‍♀️

The Medium way of monolith to microservices migration

#medium #microservices #monolith #nodejs

The popularity and usefulness of microservices cannot be denied. Having a monolithic system is okay, as long as things are not going out of hands. But when things do, it becomes a race against time to move your systems into a microservices-based architecture.

Briefly, microservices must have the following three properties.

• Single purpose - a microservice only does one thing, it has a single purpose. If you are adding a new purpose to the service, chances are you actually need to design a new microservice.

• Loose coupling - microservices are independently changeable, change in one shouldn't impact the other. Communication between them should only be done through well-defined APIs.

• High cohesion - a microservice should own and encapsulate relevant features and data together.

Medium started with a Node.js monolithic app. Eventually, the engineering teams at Medium started to feel the pain of a monolith. There was also the performance issues of Node.js on top of the monolithic-caused pain.

The article first discusses how monolith became a bottleneck for the engineering teams, and why they needed to go for microservices. There are seven strategies that Medium followed which are discussed in the article in good detail-

• Build new services with clear value.

• Monolithic persistent storage considered harmful.

• Decouple “building a service” and “running services”.

• Thorough and consistent observability.

• Not every new service needs to be built from scratch.

• Respect failures because they will happen.

• Avoid “microservice syndromes” from day one.

Overall, this is a wonderful read for anyone willing to learn about microservices from the experience of a tech company. Definitely recommended!

The Netflix stream processing platform that supports massive scale

#netflix #streamprocessing #apacheflink #bigdata

Keystone is a real-time stream processing platform at Netflix. It provides stream processing as a service, which is a huge convenience for service owners. As a service owner, you only need to write the business logic for your stream processing use-case. Keystone takes care of everything else.

Given the scope of Netflix systems, it is expected that the platform is capable of handling massive scale. As a whole, it processes trillions of events per day which amounts to petabytes of data.

Supporting this level of scale was not the only challenge the team faced. They had to support a wide variety of stream processing jobs with complicated states, complex DAGs, different levels of traffic patterns, deduplication and ordering of events, etc. There was also the requirement to design the platform as a multi-tenant one, meaning, many stream processing jobs from many different teams can run in isolation, without disrupting or depriving other jobs in terms of resources and runtime infrastructure.

From the perspective of a service engineer, you use Keystone as an out-of-the-box solution for your stream processing needs. The platform provides a UI, tools to write jobs, deployment orchestration, backfilling, rewinding, and self-healing mechanisms, monitoring and alerting, and of course, the underlying stream processing engine. Right now, Netflix is using Apache Flink as the processor engine.

Just like many other Netflix blog posts, this is domain specific article with very good detail. Readers who have familiarity or experience in stream processing systems will be benefited from this well-written post.

Storing millions of orders at Grab

#grab #mysql #dynamodb #dataingestion

Grab is a South-East Asian giant that provides rides, deliveries, financial, and enterprise services to its customers. Using the Grab app, consumers can order ride, food, and groceries seamlessly. The company has around two hundred millions users, which naturally means there are millions of orders per day. So how do they store these orders?

Data is stored in two ways based on two use cases, OLTP and OLAP. For OLTP use cases, the backend is powered by DynamoDB whereas OLAP use cases are handled by a MySQL layer. In general, depending on the scale, MySQL may or may not be suitable to facilitate analytical queries.

Ongoing or live orders are primarily stored in DynamoDB (OLTP), whereas historical orders are moved to MySQL (OLAP) for analytical purpose. This is why there is a short retention period for DynamoDB and much longer for MySQL. The rationale behind this is simple - once an order is complete, it can leave the OLTP database and remain in the OLAP one for analytical and any future queries.

An ingestion pipeline is used to ingest data from the order platform to the OLAP database.

This article is a good example of using separate databases for distinctly separate use-cases.

How Monzo built a crowdfunding system

#monzo #cassandra #go

Monzo is a UK based bank providing digital banking services to its customers. It is a neobank that doesn't have any branches. All banking services are provided online.

In 2018, the bank crowdfunded 20 million GBP. This was also done online, where customers used the Monzo app to invest.

While designing a backend system to support such a crowdfunding, the team had to consider the following constraints

• At peak, the system needs to handle 1000 requests in a second.

• No more than 20 million GBP can be raised.

• Users must have enough money in their balance to successfully invest.

• The bank mustn't go down.

At high scale, every problem becomes difficult. It might sound that it should be fairly easy to design and implement such a system, but in reality, things are more difficult and constrained given each company's scenario.

Monzo decided to design the system in two layers: the pre-investment layer, and the final investment layer.

• Pre-investment layer - in this layer, investment requests from the funders are persisted and enqueued in a queue.

• Final investment layer - here, the final investment is done by actually moving the money.

Overall, this is an example of a complex design for a seemingly simple problem.

That's all for this week's issue. See ya'll soon!

Want to share the newsletter with a friend? Share this link.