When the rush to adopting new technologies strikes back

I've spent over 30 years of my life in the software industry, working on small to large projects, so I have seen enough stories to learn some lessons and share them with you.

Last night, I was thinking on a couple of troubled big projects I led, where the problems boil down (mostly) to the same problem: using the latest technologies.

Now - I want to be clear: I am a technical person, and I love innovation. So, don't be sidetracked: what I am writing is not because I don't like new technologies.

Established technologies (e.g. relational databases) today have plenty of tools that support a proper software engineering process - generating code, documentation, reverse engineering. This is not new: since the late 80s the need of such tools has been discussed and acknowledged as key to develop and maintain quickly high-quality software. Something that, sadly, new technologies (because they are new) seldom have.

So, here’s the story.

Both these two projects had the same problem: a data model (XML and JSON, respectively) that, because of the iterative development, was subject to frequent changes.  As the data model underlies all the required data transformation, from input to the persistence layer and from the persistence layer to the output, whenever it changed all the transformations had to be changed - manually.

The lack of tools to write and maintain such code causes a very long, cumbersome and error-prone process: understanding the changes in the data model, changing the code of the data transformation, changing the test scripts, changing the test data, re-running the tests, propagate the changes to all dependencies. Moreover, there is the risk of introducing defects or side-effects in the code already developed, which adds insult to injury.

And while this problem is already important with a small team, in large projects with multiple sub-teams changing the data model concurrently, often in different geographical locations with the ensuing difficulty of communicating the changes, it becomes a formidable one, taking a high toll on the project budget and timescale.

In fact, in both projects, the time  spent changing the transformation code to adapt it to the changes in the data model outweighed the time spent to implement new user stories.

So, what’s the lesson?

In my view, if you want to adopt new technologies (they do have advantages!), be sure you also get the software engineering tools necessary to develop quickly high quality software. If you don’t, those advantages will be quickly offset by the problems I highlighted - and in the end, it will take longer to develop using the new technologies than staying with old ones.