Evolutionary Architecture

As Martin Fowler points out in the foreword of the book “Building Evolutionary Architectures”, as we moved from a waterfall oriented approach, where software architecture should be completed before designing and writing code, to an Agile approach, the role of architecture was questioned. Through numerous failed projects we have learned that we cannot design and build agile software without a solid architectural foundation. This led Martin Fowler and his team to propose an approach where architecture and programming work closely together, that allows architecture not only to follow changing requirements, but also to benefit from feedback from programmers and endcustomer. They called this approach ‘Evolutionary Architecture’, the core of which is to incrementally change software, learning by feedback from these changes. And actually, incremental change led by feedback is exactly what Charles Darwin described in 1859, when he published his book “On the Origin of Species by Means of Natural Selection”, popularly known as “evolution theory”. That was the main reason why the name ‘Evolutionary Architecture’ was chosen. Unfortunately, Darwin’s ideas were universally misinterpreted as “evolution is an incremental process of change” (a.k.a. phyletic gradualism), and, up until today, people think of evolution as a process of gradual, incremental change.

In 1972, paleontologists Niles Eldredge and Stephen Jay Gould published their landmark paper “Punctuated equilibria”. In this paper, they propose that most evolution is characterized by long periods of evolutionary stability, infrequently punctuated by swift periods of branching speciation. The theory was based on lineages in Devonian trilobites (Eldredge) and Plio-Pleistocene snails (Gould). My own research in Cretaceous foraminifera corroborated their findings, as did many other studies, and currently the “punctuated equilibria” postulate is generally accepted among evolutionary biologists.

Eldredge and Gould did not elaborate on what caused the equilibria to be punctuated. It was not until after Per Bak published his ideas on self-organized criticality in complex adaptive systems in his 1996 book “How Nature Works” that the evolutionary biology community started to realise that the idea of “punctuated equilibria” is a perfect example of self-organized criticality in the paleontological record. Without specifically referring to these scientific principles, this is also exactly what Sean Caroll outlines in his 2017 book “The Serengeti Rules”. What it boils down to is that equilibria get punctuated when an ecosystem cannot respond in time to changing conditions. Ecosystems are dynamical systems, and changes in or affecting ecosystems must be met by changes in the participants of the ecosystem. In an ecosystem in natural equilibrium, small, incremental changes will be met with small, incremental changes in the participants. These are the long periods of evolutionary stability that Eldredge and Gould mentioned. But sometimes, something disruptive happens that throws off an ecosystem, for instance the outbreak of an infectious disease, a climatic change, or the impact of an asteroid. That is when the equilibrium gets punctuated, when the point of self-organized criticality is passed: the ecosystem collapses, and the participant species either are able to weather the change, they change as fast as possible to keep up with their changing environment, or they perish. The last two options are Eldredge and Gould’s moments of swift branching speciation. The pace of change in an ecosystem must be met by the pace of change of all affected participants; if that pace of change cannot be met, the ecosystem will collapse, and a new ecosystem will emerge. So evolution is not only a slow process of gradual, incremental change, it also encompasses rapid, abrupt, comprehensive change, and even annihilation and starting all over again.

One of the best known disruptive change events in history is the asteroid impact at the end of the Cretaceous period, known as the K-T event. As much as 80% of all plant and animal species went extinct. In the meantime, groups like crocodilians and snakes where not affected, and groups like mammals were able to adapt fast enough to weather the effects of the impact. The extinction event freed up many ecological niches, which gave rise to a rapid evolution and diversification of the remaining animal and plant life. As said, this was the best known extinction event in geological history, but there have been at least 5 other major extinction events in the last 530 million years, and countless minor extinction events. The K-T event wasn’t even the biggest catastrophe to hit the earth. These abrupt disruptive events have been far more decisive in the evolutionary record, than the periods of stability.

The patterns, guidelines and best practices advocated in evolutionary architecture are specifically meant to allow gradual, incremental change of software solutions. These are sound principles in an environment of relative stability. But when facing abrupt, disruptive change, the incremental approach of evolutionary architecture will fall short. Maybe the proponents of the original evolutionary architecture concept were not aware of the Punctuated Equilibria postulate. Fact is that abrupt, disruptive change is as much part of the evolutionary record as incremental change, a fact that is overlooked in the original evolutionary architecture concept. We could avoid addressing disruptive change altogether and start using a better fitting name like Incremental Architecture, but that will not help us much. IMHO the evolutionary architecture approach should, like the evolutionary record, incorporate abrupt, disruptive change as well. It would entail adding guidelines on how to decide whether a change is disruptive enough to warrant a major overhaul of your existing software solution, or even scrapping it in favor of a new solution. It might also delve into ways to better facilitate and manage major overhauls and retirement of your software solutions.

All this does not invalidate the patterns, guidelines and best practices that allow gradual, incremental upgrade of software solutions. In fact I strongly recommend them, as minor changes are common. Also will they prove to be helpful in major overhauls.?