Any design is a trade-off

Irrespective of any area in the world (software or otherwise), every design is a trade off. A design cannot be the 'one and all'.

A few years ago, I was part of a team that developed a set of data engineering tools with the intent to develop modular components that would address and enable faster deployment of common tasks like ingestion, data validation and machine learning.

My team and I had three modules - data quality checks, data profiling and data reconciliation. We implemented these modules using PySpark. We also developed the modules as micro-services.

For a customer implementation, the team came back to me stating that the pipelines that were being orchestrated from Airflow were timing out. To resolve the issue, we dug into details and realized that this was being caused due to a slow network. And it was related to our design decision.

As we were using a metadata based approach, it was decided to use MongoDB to store the application metadata. For example, for the reconciliation module, we stored source and target information. We stored data store type, table name, path to file (if data store is file) and more. We also stored execution information - things like date of execution, start time, end time, result of execution and more.

As mentioned, we had three components - data quality, data profiling and data reconciliation. Each was developed as a micro-service. Each micro-service had to interact with MongoDB database to fetch metadata and also store execution information.

This is pretty straight forward. Where is the design element? As we have three services that interact with the metadata database, the relevant question to ask is, how does each component interact with the database?

For this seemingly simple ask, we can have two approaches.

The first approach is to have each component interact with the database directly. The second approach is have an intermediate component to handle the interaction with the database and then have each component connect with the intermediate component. In other words, in the second approach, we can define another micro-service to handle the interactions with the database.

While the first approach is simple, the drawback is that each component is directly coupled with the backend database. If we decide to change the database from MongoDB to say PostgreSQL, then each component has to be updated to work with the new database.

In the second approach, as the connection to the database is handled by a dedicated micro-service, if we change the database, we can substitute the MongoDB micro-service implementation by a PostgreSQL micro-service implementation, while ensuring that the interface remains the same. If the interface is not changed, the three components will not be aware of the database change.

But now we have one more micro-service in the picture. Even if a micro-service is deployed in the same data center as the component micro-services, the fact that micro-services use the network (we were using REST APIs), means that we add a small communication overhead a compared to the execution of a function. If the network is nor performant, this communication becomes the bone of contention. Additionally, each network connect goes through authentication and authorization, which adds to the time taken for execution.

This example shows that we have a dilemma. On the one hand, we have components where code is duplicated, while on the other hand, we have components that are flexible, but where we will spend some additional time.

As for code duplication, you will correctly suggest that we should implement the common code as functions / methods. While this is one approach, it does not take away the fact that we no longer have 'seamless replacement' of components. Each time we make a change, we have to update, re-test and re-deploy each component.

If you are not convinced with this, you can watch a few YouTube videos related to aircrafts from the second world war (WW2) (of all things). The design of each aircraft was all about trade offs. The Supermarine Spitfire - one of the well known aircrafts from WW2 - was not able to accommodate cannons for a long time due to its wing. While its wing design was superb, it was much thinner than that of the Hawker Hurricane. Hence it was easier to accommodate cannons in the Hurricane as compared to the Spitfire.

#design #trade_off #micro_service #microservice