Transformation, Technical Debt and Toxicity... Episode 2

Episode 2 - Technical Debt

As part of a series of articles around undertaking business transformation, I will now consider technical debt. Episode 1 gave an introduction to the drivers, pitfalls and implications of doing a transformation. As the unsustainable weight of technical debt on a business is often a common reason for embarking on a large change programme, it merits is own investigation. That's the purpose of this article.

Technical Debt?

Common aspects of technical debt from a business perspective:

• The systems no longer work as expected
• Are getting slower, more unstable and can't handle business volumes
• More expensive to manage and operate than in the past
• Hindering rather than helping the business (what was once an asset is now a liability). Our technology is now a competitive disadvantage
• Delivery of software updates to clients is slower

From a technical perspective:

• Delivering software and systems are harder
• Always finding old bugs that need to be refactored (slowing us down)
• Platforms are both infrastructure and software limited
• Support is a nightmare and the systems are now considered unstable

How did we get here? How is it that our technical systems are platforms are now unloved by the entire company and bringing us down? Technical debt is a technology risk that if left unattended can quickly become a business risk.

Technical debt is Bad! ...but not completely so

The above are just some of the opinions and implications of technical debt so you can see how it is bad. But to get into the position you have to had delivered something to clients in the past (and create revenue). A lot of business literature describe technical debt as the cost of future reworking required of systems to meet the business needs at that future time. Taking an "easy/speedier" path today rather than the "right" one. Yes, technical debt is stealing from tomorrow to deliver today. But I like to think of it as:

Technical debt is an echo of past delivery that was achieved with finite resources on a tight deadline

Yes "The delivery/project/senior manager made me do it against my better judgment" is something I often hear. Having technical debt however, means you achieved delivery in the past. That delivery may have helped you survive to this point where you can now lament your past, lazy, compromising self.

Technical debt in itself is therefore not wholly bad. What was achieved was not perfect but good enough at the time. What will impact more is thinking that what was created as good enough for the past will remain good enough for today. Allowing technical debt to build up and not addressing the compromises you know were made at the time will only make matters worse. This debt needs to be repaid (remediate the technical defects) sooner rather than later. If the manager made you do it then, now time needs to be allocated to eliminate the technical debt that got you your past delivery. Ignoring this accumulation will store up problems for the future and at the end of that road is a hard transformation programme.

Red flags

The accumulation of technical debt is slow and inexorable, everything will seem alright (or at least manageable) ...right up until it isn't. Here are some red flags creators of technology need to be aware of and all result in:

A software system built on top of a weak architecture will sink due to the weight of its own success

Be wary when your non-technical colleagues take a delivery position of:

• "It is only an MVP / PoC"
• "It's just a throwaway for a demanding client to keep them happy"
• "Let's not boil the ocean on this"
• "Don't let perfect be the enemy of good"
• "This is top priority for management"

MVP (Minimum Viable Product) and PoC (Proof of Concept) systems are by definition poorly architected with fast delivery taking precedence over sustainability. "Throwaway" systems are rarely so. Systems created for these purposes have a very annoying habit of becoming "mission-critical" and placed "in production" without addressing the heavy baggage of technical debt they carry. As for multiple projects, in the context of finite resources, technical managers should note:

When everything is top priority; nothing is, to lead is to choose.

Technical debt cannot be fully solved by simply throwing more IT resources at the issues. While this is a common strategy in a cloud based world, this assumes that the platforms are infrastructure limited and more compute, disk and IO will solve the problems. To counter that, remember Moore's law will not save you every time, even in the cloud.

Software gets slower faster than hardware gets faster - Niklaus Wirth

Eventually the limitations of the software will come to the fore and cost control will prevent further infrastructure investment. Software architecture is not a dirty word, even in an agile SDLC[1].

A Question of Balance (for the entire organisation)

Of course technologists understand the business pressures their client facing colleagues have to manage. But all members of the organisation need to be aware that technology is a balancing act between:

• Stability
• Functionality
• Performance

Without infinite time or money, you can at best, only maximise 2 of the 3 competing factors. "Pick 2" is a good question to ask any colleague when discussing technology.

Stability - This is the result of delivering well architected systems and/or investing time and money into paying any technical debt accrued. It allows the business to retain clients they may have won in the past and facilitate sustainable growth.

Functionality - Is the result of delivery, with or without associated technical debt and drives growth in the form of new business. While this helps somewhat in retaining existing clients, it is more a driver for new client acquisition.

Performance - This is the ability of systems to scale as is, the ease with which it handles increased business volumes and the cost efficiency of their operation. This helps both with the retention of existing clients and if functional enough makes new client onboarding a painless process.

A better world

As also discussed in an article on transformation, it is important that all staff (delivery and management in particular) have an understanding of the current capabilities of its technical department and appreciate what is possible what is not possible and what is storing up problems for the future. Stretch targets are fine, but a chronic emphasis on rapid delivery without regard to the accrual of technical debt will lead to unpleasant (and costly) tipping points.

Strategically it is better to say no to unreasonable delivery targets to protect the sustainability of your organisation or platforms. No client has ever cared about YOUR technical debt and it should not just be left as a technologist headache. All members of an organisation should be aware of the implications of their promises to clients. Sometimes "it will be ready when it is ready" has to suffice.

Avoiding (or reducing) technical debt

Technology has a long and venerable history of failures which should be seen as a learning experience. You do not have to reinvent the mistakes of others. Excellent ways of avoiding technical debt include:

• Every delivery sprint should not be just about client delivery, Refactoring sprints should be regular. The proportion of these sprints should reflect the right balance for your particular organisation with refactoring a core element of developer time. Enforcement could mean insisting that refactoring artifacts are part of your release processes
• Make space and time for regular discussions about technical debt within your organisation including delivery, project and technology teams
• Architecture first, MVP second, re-architect where possible
• Treat regression issues seriously as they feed refactoring work
• Document everything, take knowledge transfer seriously. This reduces key-man dependencies and avoids relying on heroes for delivery and speeds up new hire impact. Technical institutional knowledge should not be solely in the heads of your technical staff.
• Measure as much SDLC metrics as you can and make time to analyse them. How can you manage something if you cannot measure it?
• Disciplines such as incident, problem, release and capacity management in a service operations department can produce KPIs that are a great signpost to targeted removal of technical debt
• There is tooling to identify software technical debt across many different SDLC frameworks, much of it automated. Current tooling goes way beyond the classic use of lint [2] tools in writing code. Allied to this is a shift left in all your testing operations.
• Appropriate automation tools in your development lifecycle can eliminate much non-architectural technical debt at it's source. You can automate the elimination of sloppy code, eliminating sloppy architecture is much more difficult.
• A shift-left mindset is embracing deep QA in your SDLC. Load testing in particular will give you a window on the future as unexpected volumes are a sure fire method of revealing both software and infrastructure technical debt.

In summary

The accrual of technical debt may be a slow process but it is a technological risk that can easily become a serious business risk. Eliminating technical debt is a costly process especially when left too long, where it invariably needs a large transformation to remedy. Technical debt results from the triumph (or rather the imbalance) of delivery focus over stability and performance on an extended timescale. While difficult to remove, there are now many tooling options to allow you to minimise this inside your SDLC. The best defense against the emergence of technical debt is well-architected, rather than an expedient systems. In the long term "good enough" is never enough.

[1] SDLC - Software Development Life Cycle, are all the process involved in the creation of software, from initial authoring to release to users.

[2] Lint - is a method of static code analysis that checks for obvious programming errors and can be easily included in any developer's editor or IDE