The Death of Systems and the Agile Fallacy

‘Everything that has a beginning has an end’

Computer systems experience a life cycle that is not dissimilar to living organisms. They are created brand new and shiny, go through a youthful period of exuberance where everything seems possible, experience a middle age of slowly dawning realities, a maturity of resignation to the inevitable, a decrepitude of quiet fragility and ultimately they are no more.

This arc of vitality is well known to anyone who has ever worked on a green field project or been condemned to nurse an ancient legacy system. It is part of the reality of technology. There are brand new Android babies issuing forth every few minutes and there are prehistoric leviathans languishing in the darkened server rooms of financial institutions and booking systems the world over.

Given this truth, it seems strange to me that we go about building software as if this were not the case. We build our software, like we live our lives, with scant regard for the end game. We never consider the death of the system except as some distant abstraction that we may know exists but which we refuse to engage with at a deep level. We build brand new systems and everything is great!

Only it's not.

We often build systems that are far more complex than the problem they are trying to address. We have all seen nightmarish legacy code that you wouldn't want to work on if it were the last system on earth! Organisations struggle under the operational burden of these ancient monoliths and fail to replace them in a timely way.

But why do systems become legacy? Is it something that we can avoid? Why can't we make systems that last longer than a few years? To answer this we need to consider the words of Ivor Jacobson who referenced the 2nd Law of Thermodynamics in relation to software systems.

'When a program is modified, its complexity will increase, provided that one does not actively work against this.'

This means in evolving software systems, entropy builds. That is to say, over time the amount of effort to maintain the system increases and the amount of work that the system can do decreases.

In addition Leyman’s Law of Continuing Change states

'an E-type system must be continually adapted or it becomes progressively less satisfactory'

These two principles infer why systems become redundant. A system must change to stay useable but, as entropy increases, the system gets harder to change. Intuitively, there exists some point in time where the cost of modifying the system in order to exploit some value from it exceeds that value. At this point the system is no longer useful and must be replaced. It has become legacy.

The measurement and control of the entropy in the system is the primary technical challenge of an incremental delivery program.

It is not enough to create a system that works today! You have to ensure sufficient lifespan in the software such that the business can extract value from it tomorrow to justify the investment. Only careful design and anticipation of future requirements can help to slow the rate at which the system becomes unchangeable.

Refactoring can help because it allows us to improve the internal quality safely and if refactoring were free we could limit the entropy indefinitely. However, refactoring is not free for all but the most trivial of changes. In fact it is very costly particularly at the architectural level.

This illuminates a fallacy at the heart of Agile planning.

We cannot continue delivering business value at a constant rate because we have to expend an ever increasing amount of effort to keep the software in good shape.

Velocity inevitably falls. You can slow the rate at which it does so by careful design and technology choices and by trying to anticipate future changes but you cannot stop the cost of ownership increasing. This throws up a rather unsettling question for managers of incremental delivery programs.

How do we know that the software isn't atrophying at a rate that will cause the program to fail?

I have wrestled with this question for a good many years. Sadly, we lack any clear indications of the rate of fall in productivity that can warn us early enough to be able to do something about it. Often we spot emerging design limitations far too late to be able to address them which is very frustrating. Fortunately many projects are of sufficiently short duration and scope (at least officially) to be confronted by this stark reality. However, it does exist.

The rate at which the software becomes unchangeable is a critical factor to be considered in the life span of the system. If I am building a demo for a conference in two months time and the code is to be thrown away after that conference, then there is little point in investing effort in a highly adaptable design. By contrast if I am building a financial platform that will support a business for the next five years then I'd better evolve an architecture that keeps the software supple for that period.

This leads to the first major headache. When was the last time you heard any business stakeholder or product owner define the expected life span of the system? We just don't consider it. Presumably it's enough of a challenge even getting the system up and running without worrying about how long it will last.

But how are we designing our systems if we don't know how long they need to live? Are we assuming that systems will last forever? Because they most certainly do not.

I'm not suggesting that businesses are oblivious to the lifespans of their systems. However, this awareness normally occurs when a system reaches old age and is already failing in some way. The proposed life span of the product is a vital KPI for the system architecture and needs to inform the design from the beginning.

It is impossible to make reasoned design decisions without knowing how long a system is expected to last and in what ways it is expected to adapt.

You can certainly ignore the future but it is highly inefficient to do so when even the simplest of longer term considerations can radically improve your chances of extending the system life span.

This kind of long term view and discipline is very hard to achieve and presumably that is one explanation for the horrifying dumbing down of design that I see on many Agile projects. However without sufficient technical rigour we open ourselves up to high costs of ownership for the software and the processes that support it.

Agile programs following an emergent design paradigm will experience sky rocketing levels of entropy. This will be made worse if the team is big or if it has a flattened design authority. (Heaven help you if you have both!) I have seen codebases that are pretty much set in concrete before even achieving a first release. Principles like YAGNI and do the simplest thing that can possibly work are making the situation worse.

In summary then, incremental delivery is good. Businesses can now experience early return on investment, more fully engaged stakeholders and opportunities to change direction without incurring huge costs. However, existing Agile development practices do little to address the rate at which software becomes unchangeable and in some cases they exacerbate the problem.

Until we accept that software evolution is wickedly difficult and deserves the same level of design rigour as other engineering disciplines we will continue to create legacy systems at an alarming rate.

So what can be done? How can we protect our incremental delivery programs from the ravages of change?

Well, we need to get back to serious design work. Only highly skilled software designers can slow the relentless march of disorder. The informal design adopted by many Agile projects is fine for small teams and relatively short projects but if you want systems that don't quickly become a burden you better get serious about software design. (A quick conversation at the white board doesn't cut it I'm afraid).

We need a new theory of evolutionary architecture and a new breed of evolutionary architect who focuses on managing the cost of ownership of an evolving system. This is less about prescribing how to build a correct system and more about maintaining a correct system, in the face of change, at reasonable cost.

We need mechanisms for replacing old systems with new ones. Maybe automated tests, expressed in the language of the stakeholders and independent of the system internals could be used but it would have to go much further than our current approach and tooling. Most acceptance tests we build today are so dependent on the anatomy and language of the old system that they would not stand up as a contract for replacement. Specification by Example maybe a valid approach.

We need to refocus delivery teams and product owners on the importance of managing complexity in the software. We need a new culture in which we spend less time in stand ups, retrospectives and planning games and more time caring for the internal goodness of our systems.

Lastly we need to design and build our systems with their replacements in mind. As we move into an age of unprecedented complexity we can no longer approach software development as if systems live forever.