Past, Present, and Future of Software Engineering

I recently completed 15 years in my career. I naturally looked back and wondered how much I have evolved over this time. But what intrigued me more was how much the environment has evolved over the same time. I sat back and structured my thoughts on this matter, and that gave me a powerful mental model about the larger trend playing out and what the future will likely look like.

Understanding trends is a powerful tool to prepare for future.

I am sharing that mental model here in the hope that it gives young software engineers a perspective on how things will evolve and helps them make good career decisions for rapid growth.

The Past

15 years back, the expectations from a software engineer were very specific. For example, people were expected to know one programming language. The job titles also were accordingly like Java Developer, PHP Developer, etc. On the frontend side of things, designers used to create designs in Photoshop, and then an HTML/CSS developer would convert that into a static HTML document. A web developer would then add logic (Javascript) to it to implement desired behaviour. Relational databases ruled the world and only very few people knew NoSQL as it was too cutting edge. Further, TechOps was a separate skillset to manage servers. Developers were not expected to monitor performance and manage uptime of their code in production. Cloud was in its infancy, so every tech company needed people with specific skills to operate physical servers in data centers, e.g. networking specialist.

The focus was on depth of knowledge and career paths were clear.

Accordingly, people used to focus on a few things and went deep into them as they progressed in their careers. Knowledge of clever CSS hacks for IE 6 browser, SQL query optimization, Programming language internals, etc. were skills young developers aspired for. This meant that career paths were very clear for most people. They needed to become more knowledgeable in their specific domain and get promoted to senior roles with bigger paychecks.

The Present

Today, a software engineer is expected to posses a great breadth of skills. Number of popular programming languages has proliferated and it's common for backend engineers to (need to) learn a few. On top of that, full stack developers are in great demand and many people know both backend and frontend. Even further, DevOps movement has brought traditional ops responsibilities like monitoring to developers' plate. Many teams even put the responsibility of testing on developers. As it it were not enough, most software development teams work as small mission-oriented groups wherein software engineers are expected to have some understanding of product and business as well.

The focus is on breadth and career paths are not straightforward.

Ergo, software engineers today focus on learning new skills. They may be less on depth, but can get started quickly and learn on the go. The number of skills on a developer's resume today can be anywhere between 5 to 50! Accordingly, the career paths seem very random today. Many young software engineers are confused (even anxious) about what to do and what not to do to build a good career. As if the old playbook has been thrown out of the window but a new one hasn't been written yet.

Why did this happen?

The most fundamental change that has happened over these years is that the predominant software development methodology has firmly changed from waterfall to agile. This has changed many more things that most people think at first. Let's see how.

Why is a very powerful question. It can lead to profound answers.

The waterfall model of software development is a multi-stage process with unidirectional flow of work from the first stage (requirement gathering) to the last (shipping). In such a process, it is desirable to have clear contracts on inputs expected by and output expected from each stage. Ensuring that output specification of one stage exactly matches the input specification of next stage ensures that the overall process runs smoothly. Under such a process, it is natural (and desirable) for workforce to also align accordingly, i.e, specialize in one stage. Ergo, Product Managers gathered requirements and produced PRDs. Engineers consumed PRDs and produced code. Ops took code and produced working software. It was therefore very natural for people to seek depth (and not breadth) over their careers and the career paths were also very clear.

Notice that I had highlighted the word unidirectional when describing waterfall. It's because that is a fundamental assumption of waterfall model, i.e., the waterfall model is designed to optimize the flow of work in one direction. And it is this very assumption that the real world did not uphold kindly. Bugs flow back from QA to dev, deployment failures and production issues flow back from ops to dev. And the arch nemesis of waterfall is requirement change, which brings everything back to the first stage.

Generalists have a strong advantage over specialists in the agile paradigm.

Agile attempted to solve this problem by embracing the volatility instead of fighting it. It accepted that work will flow backwards and sought to provide a mechanism to operate optimally under such condition. One profound impact of accepting backward flow was that each stage could no longer be optimized in isolation, and broader perspective became necessary. This necessitated much greater collaboration, thereby mandating that people cross the sharp boundaries and understand what and how of other people's work. Thus people had to become more horizontal than before. Another profound impact was that the frequent back and forth added a lot of handoff overhead, and therefore multi-skilled people who could work on adjacent stages delivered far greater productivity as they could simply do multiple things themselves saving the handoff time. Full stack developers are more productive because they can do both things on their own without context switching, instead of a backend developer and a frontend developer having to schedule time to decide contract and do integration testing, etc. Thus, generalists have a strong advantage over specialists in the agile paradigm. Is is for this reason that software engineers today are seeing more and more responsibilities (and thus opportunities) coming up their way, and are getting nudged further down the path of generality.

The Future: Coding at the Speed of Thought

The trend of generalism is creating more opportunities for software engineers and making the discipline more intellectually stimulating. But it is also creating challenges, of which two I feel are big.

First, lack of clarity in career progression is causing anxiety in young professionals. The way out for this seems to be to adopt the agile mindset for careers themselves - don't plan and commit very long term, take short steps and keep improvising. Careers do not have to be unidirectional, taking a step back at times is perfectly okay. It does not much matter exactly what you are doing, as long as one is learning along the way and adapting accordingly.

Second, generalism cannot be sustained if the underlying complexity of software development does not get increasingly abstracted. This is happening by tools. Evolution from low level languages to high level languages to frameworks to low code platforms and AI assisted code companions is just one example. Wherever we see complexity today, there lies an opportunity to build tools to abstract it. Thus DevTools ecosystem is proliferating the way it is.

The endgame of software engineering is in building products with mere thought. Every step in future will take us closer.