You may not need that technology

Written by: Anish K Manandhar , Principal Architect

Things are always changing when it comes to tech and the allure of new technologies get tempting by the day. Especially when every new tool or framework claims to change how we code, making it easy to get carried away.

However, as we face innovation, it's important to remember that the value of technology lies not in how new it is, but in how useful it is.?

This blog is a sincere message to software engineers, urging them to look past the excitement of new technologies and base their choices on the solid foundation of numbers, computer science, and the basic rules of designing systems.

The illusion of the ‘new’

The realm of software development is often dazzled by the latest frameworks and technologies, each boasting a lot of features.

However, in reality, we often utilize just a small portion of these capabilities. This leads to a technological maze— difficult to navigate and maintain.

The truth is, technology should not only resolve business problems but find the right balance between user experience, development experience, cost effectiveness, and scalability.

It's about creating solutions that are as elegant in their simplicity as they are effective in their function.

Opting for LLMs where simple heuristics would suffice, employing microservices unnecessarily, or adopting new javascript frameworks every year, can be some instances that illustrate the trend towards complexity over practicality.

The role of numbers

At the core of system design lies a profound yet frequently overlooked element: numbers and mathematics. Before chasing the newest tech trend, answer these quintessential concerns.?

1. What is the application's maximum capacity for concurrent users?
2. During which periods do you observe peak usage of the application?
3. How do you anticipate the volume of data evolving over time? Will it follow a linear, exponential, logarithmic, or decaying pattern?
4. What are the computational resource requirements for processing a standard transaction?
5. What is the estimated cost for implementing this solution?
6. How do potential security vulnerabilities increase with the integration of additional technologies?
7. How is data interrelated within your application? Are the relationships one-to-few, one-to-many, many-to-many, or many-to-one?
8. How is user data segmented within your system? Are there quantitative limits on data access levels?

These aren’t mere theoretical considerations; they are the foundation upon which we can build efficient and scalable systems.

Testament of fundamental principles

In my experience, I’ve encountered situations where countless requests were efficiently handled using appropriately chosen data structures. And it's functioned well for years.?

This wasn’t the result of pursuing the newest technologies but rather a testament to the power of fundamental principles thoughtfully applied.?

Some popular examples:

1. Medium uses the Bloom filters, probabilistic data structure, in its recommendation module to avoid showing those posts that have already been seen by the user.
2. Meta leverages HyperLogLog within Presto for efficient cardinality estimations, streamlining massive dataset queries from days to minutes with minimal memory use.

Had these been implemented with multiple advanced technologies, the costs would have been substantially greater.

Balancing innovation and fundamentals

This is not to imply new technologies should be avoided. They often bring valuable innovation and enhancements. However, the key is finding the right balance. A mix of new and time-tested technologies, wisely chosen, can lead to solutions that are efficient, cost-effective, and deliver enduring value.

The goal is not to reject innovation but to temper it with insight and foresight.

Conclusion

As we navigate the landmine of emerging technology, let's remember:

“Technologies will come and go, but the foundations remain steadfast.“?

It’s these principles that have endured the test of time, guiding us to make choices that are not just technologically sound but also deeply rooted in the real needs of the products and users.

In the end, the true art of software engineering lies in discerning what is truly necessary, finding beauty in simplicity, and always keeping in mind that at the heart of technology lies the unchanging essence of problem-solving and innovation.