Resume Driven Development (or Architecture) for Data Infrastructure

Resume driven development or Resume driven architecture is a common phenomenon where the developers or the architects follow a practice of choosing technologies or approaches for a project primarily to enhance one's resume or CV, rather than based on what's most appropriate for the project's needs.

This is not only common for the Software architectures but also to some of the niche areas like Data Infrastructure.

Some of the mostly commonly observed patterns include:

a. Kubernetes workloads - Using Kubernetes to orchestrate a simple data pipeline that could be run on a single server

b. Microservices dilemma - Implementing a complex microservices architecture for an application that doesn't require that level of scalability (recall Amazon prime going back to 'Monolith'?)

c. Fancy NoSQL - Choosing a NoSQL database like MongoDB for data that fits well in a relational model, just to gain NoSQL experience

d. Data lake or Lake house or Delta Lake or Data Warehouse or Data Marts or Data Mesh or some other *house - Building an overly complex data lake using technologies where a simple data warehouse could just work perfectly fine

e. Forced Machine Learning - Adopting the latest machine learning frameworks for basic analytics tasks that don't require ML. What about holy grain called GenAI?

Well, we all have been there and done that (to an extent at least), but why? Probably, one or many from below.

Carry forward from your previous project

or

Desire to remain marketable and relevant (Ouch!)

or

Trendy tech to excite your future employer

or

Remain up to date with the buzzwords

or

Forcing change readiness?

Does it end here? Well, no! For every action there is a consequence or some tradeoff.

What are the consequences of 'Resume Driven Development'?

Needless to say that it is an opportunity to create an 'over engineered' solution which not only is difficult to maintain but also expensive (Increase complexity, provide solution which is expensive and call yourself a superman!)

and/or

Potential mismatch between actual business needs and technology

and/or

Key man dependencies when your north star developer leaves the organisation

ok, enough of this.

How do we avoid pitfalls of Resume Driven Development for data infrastructure?

Focus on solving business problems rather than picking your favourite technology (Horses for courses!)

Consider long term maintainability and Team skills. Learn from past, evaluate future based on what is known at present and opt for solutions that has stood the test of time.

Emphasise on delivering medium to long term value rather than chasing technology.

If you've read till here, do you think data products today are the same as APIs were probably a decade ago?