Alignment (or not) between database technology and software development methodologies
Rik Van Bruggen
Helping the world Operationalise Machine Learning and AI in a meaningful, efficient, managed and effective way
How data technologies and software development have evolved in parallel and synergistic ways
When I think about both the database technology industry and the software development community, and try to take stock of the evolution that we have gone through in them in the past 25 years, I can’t help to see that there are some really interesting and important trends, and that these trends are actually really well aligned between both industries. No surprise?
Relational databases and waterfall development dominating the earth
When I started my career in tech, the world was reigned by Relational Database Management Systems (RDBMSs). My university graduation thesis was actually about a sizing study of an early version of MS SQL Server – of all things J … And: at that time, we were developing software in a very well-structured, very well-governed fashion, using waterfall development methodologies. Every step had its clear and defined purpose, and would follow another step. It was all about analysing, designing, developing and testing software on significant chunks of functionality, that would all take months to deliver.
Things started to shift…
I remember at the time, feeling that there was starting to be a shift. We were starting to see some understanding of how we would need to become more agile – that the mythical manmonth was indeed, mythical. That we would need to break up problems into smaller ones. That use cases were going to have to drive our alignment with business stakeholders.
In my experience, the advent of many new technologies (object orientation, open source operating systems and development frameworks/infrastructures, to name a few), this started to change significantly, and we started to develop software in a truly more agile, iterative way. And forgive me if I get the chronology wrong here, but just around that time, we also started to look at data in a different way, and our industry developed an entire range of new (NOSQL, for lack of a better category name) data technologies. All of a sudden the normalised, table-based data model that had famously overthrown the navigational and network databases of the mainframe, was deemed less opportune for modern software development practices. Document databases, key-value stores, graph databases, and wide-column stores each had their unique traits that suited specific use cases – and therefore drove their adoption to sky-high levels. We are still seeing that today.
Parallels between database tech and software development?methodologies
So in my mind, we have a really interesting, and parallel set of shifts:
领英推荐
About alignment and misalignment
It seems to me that there is an interesting alignment and misalignment here between these shifts:
I tried to express this in the following bigger mindmap, from which I have pasted above:
Interesting, not? Clearly, my overview and articulation of this evolution is incomplete and even inexact, but the broad lines of the argument hold true. I would love to hear it if you think otherwise. Note that I do not mean to say that you cannot do agile with RDBMSs, or that you cannot do waterfall with a NOSQL database – but I do not think they are well aligned.
It feels important to point out these types of higher level shifts so that people understand why a modeling tool like Hackolade, does what it does, and doesn’t do what it doesn’t do. After all: data modeling
And in any case: let’s hope that it encourages the conversation and discussion – we can only collectively get better by doing that.
All the best
Rik
PS: I thought about using a graph method and "close the triangles". Kind of like the old adage "an enemy of my friend is my enemy", one could argue that "misalignment of a concept to an aligned concept will misalign that concept with the original concept". Therefore: NOSQL/Waterfall and RDBMS/Agile would be implied to be misaligned. But I think that is too simplistic, and therefore I did consciously not include it in the article.
Senior AI/ML/Data Serverless Specialist Solution Architect
1 年Ja, jij bent wel van bruggen.
Driving transformation, innovation & business growth by bridging the gap between technology and business; combining system & design thinking with cutting-edge technologies; Graphs, AI, GenAI, LLM, ML ??
1 年Adding to the discussion, I've noticed an intriguing trend where organizations are moving beyond established frameworks like SaFE, Lean, and Scrum. We're entering a new paradigm where the focus is shifting toward creating unique, tailor-made methodologies that cater to an organization's specific needs and culture. This shift is a testament to the dynamic nature of our industry.
Driving transformation, innovation & business growth by bridging the gap between technology and business; combining system & design thinking with cutting-edge technologies; Graphs, AI, GenAI, LLM, ML ??
1 年Absolutely insightful read, Rik Van Bruggen! I appreciate your deep dive into the evolution of database technology and software development methodologies over the past 25 years. It's fascinating to see how these two industries have grown and shifted in parallel, from the dominance of relational databases and waterfall development to the rise of NoSQL databases and agile methodologies. Your point about the alignment and misalignment between these shifts is particularly thought-provoking. It's true that while waterfall/agile methodologies and relational/NOSQL technologies are opposed and therefore misaligned, there's a clear historical alignment between relational/waterfall and NOSQL/agile. I also agree with your assertion that it's not about whether you can do agile with RDBMSs or waterfall with a NOSQL database but rather about the alignment and synergy between the methodologies and technologies. Your perspective on how Hackolade fits into this landscape is enlightening. Hackolade is designed to support the agile, iterative approach to software development and the heterogenous nature of modern data architectures.