Software Development – How to Tame Tech Debt

Tech debt, short for technical debt, is commonly referred to as the accumulation of all the non-customer-facing technology work that needs to be done in future when shortcuts or suboptimal solutions are implemented to expedite the delivery of software. Just like financial debt, tech debt is a valuable tool that incurs costs and consequences if it is not paid off in a timely manner.

Sources of Tech Debt

In reality, tech debt can arise due to various reasons, not only because of shortcuts. These can be broadly classified into three categories: Intentional, Obsolescence and Unintentional, which are easier to remember as a mnemonic, IOU.

Intentional

It is the most common form of tech debt, resulting from a trade-off between time to market vs staying true to the definition of done or quality standards. Just like financial debt, it can be a valuable tool if it is an informed decision based on cost/benefit analysis, and it is explicitly captured in the backlog for future payoff. For example, foregoing test automation for a time-critical milestone or not fully meeting scale requirements because product adoption might take months, could be the right tradeoffs. Other common scenarios where it makes sense to take shortcuts are when technology or product direction are not clear. E.g., proof of concepts or A / B testing of a feature.

Intentional tech debt can be bad when rash decisions are made without even realizing how big a future liability is being created. Martin Flower classifies it as Deliberate-Reckless in his technical debt quadrant model.

Obsolescence

Even though the term software implies something that is soft and malleable, software systems tend to become more disorganized, complex, and brittle as they undergo changes over time. It is also called software entropy, rot or erosion which manifests itself as tech debt in the form of code refactoring.

In addition to internally developed code, a typical software system carries hundreds of external dependencies, such as operating systems, browsers, databases, programming languages, frameworks, and libraries. Today,?96%?of scanned applications contain Open Source Software (OSS) components, with an average of?528?components per application! Such use of commercial and OSS is absolutely essential for rapid time to market and cost considerations. New versions of these dependencies come out all the time while older versions become obsolete/unsupported. The effort required to keep such dependencies up-to-date and find substitutes when they are declared end-of-life (e.g., Internet Explorer and AngularJS) is another form of tech debt.

Technological advances and paradigm shifts (e.g., software as a service, public cloud adoption, microservices architecture, etc.) can cause massive tech debt in the form of major refactoring of code or a complete rewrite if the existing architecture and code become obsolete. Such disruptive changes require strategic decisions at the highest-level involving technology, business and customer facing teams to pay back the tech debt.

Unintentional

Software development is an iterative process. Product managers come up with product requirements that reflect their best possible understanding of user needs. There is no such thing as perfect understanding of requirements and how they will evolve in future. Developers translate their imperfect understanding of requirements into technical architecture and code.

As a new software release is used by customers, new understanding is gained by the entire software delivery team. The architects and developers may now view the initial architecture or code as sub-optimal. The delta effort to align / refactor the code to this new understanding, as Ward Cunningham explains in his tech debt metaphor video, is one of the many forms of unintentional tech debt. Martin Flower classifies it as Prudent-Inadvertent tech debt in his technical debt quadrant model.

The opposite of prudent-inadvertent is reckless-inadvertent tech debt. Poor development practices, ad-hoc or non-existent developer onboarding, and lack of checks and balances are the major contributors to this kind of tech debt.

Another major source of unintentional tech debt is lack of explicit non-functional requirements (NFRs), such as performance, throughput, scale, reliability, accessibility, localization, etc.

Consequences of Excessive Tech Debt

I am going to borrow the cooking analogy from this InfoQ article to get the point across. Imagine cooking in a restaurant kitchen under constant time pressure to deliver fast, with no time to clean prep area, pots, and pans. Soon, the kitchen will be a mess, the cooking process will slow down, the quality of dishes will suffer, and customer complaints will grow. How would you like working in such a kitchen?

It is the same story for developers working on a software system with excessive tech debt! The morale of developers goes down, productivity drops, the total cost of ownership goes up, security and compliance risks spike and the quality of deliveries suffers.

Managing Tech Debt

The biggest problem is that unlike a dirty kitchen, tech debt is mostly invisible to non-technical stakeholders. By the time its effects start showing up in business results, it’s often too late. If unchecked, it can turn into a debt spiral, an existential threat for the software business. Here are some best practices to prevent and manage tech debt.

1.????? Create a Shared Definition of Tech Debt

The first step in managing tech debt is to create a shared understanding among all stakeholders on what qualifies as tech debt. That becomes the basis for assessing and prioritizing future tech debt. From my experience, the two most effective practices in this regard are:

a)?????? Have a clear definition of done. You can tell if a shortcut was taken or not only if you have a standard to measure against. E.g., the unit testing code coverage or system availability targets.

b)????? Treat new or enhanced NFRs as new feature investments, not tech debt. Stepping aside into the hardware world for a minute, CPU, GPU, and memory chip makers spend tens or even hundreds of millions of dollars to build newer chips that are faster or offer more capacity / throughput. It is strange but not uncommon for non-technical folks in the software industry to assume that such non-functional enhancements for software systems can be met by just throwing more hardware at the problem. While additional or more powerful hardware can help with incremental performance / throughput gains, invariably major investments in code and architecture are needed to support a much larger scale of usage even for the existing functionality. E.g., it could be a multi-year effort to scale up a software system to meet the needs of large enterprises if the solution was previously designed for the mid-market segment. Then there are other classes of non-functional requirements, such as reliability, accessibility, localization, security, etc. which can’t be solved at all by just adding more hardware.

2.????? Treat Tech Debt as a Business Issue

Tech debt has become a mainstream but confusing term in the software industry. The word ‘tech’ in it has created a perception that it is an internal housekeeping chore for the technology teams. It is ironic that while most technology companies declare estimated useful life of software in the 3-7 years range, they don’t explicitly allocate that depreciation or amortization budget to software development teams. The IOU contributors to tech debt mentioned above should make it evident that it is a much broader, complex, and strategic issue. Therefore, business understanding, participation and buy-in is critical for all significant tech debt decisions.

3.????? Surface, Classify and Communicate It

The first step in solving a problem is to create visibility around it. Otherwise, it is out of sight and out of mind! It starts by having a process wherein every new intentional tech debt is added to the respective team’s backlog. To prevent obsolescence of dependencies, have a software artifact registry and a technology governance process that periodically reviews the support lifecycle for each artifact and adds dependency upgrade work to the backlog. The tech debt created by technology paradigm shifts tends to be strategic and large. E.g., migration from co-located data centers to public cloud creates a cost bubble during the transition period, impacting gross margin. Such strategic decisions should be part of the multi-year product and technology roadmaps blessed by the executive team. Finally, to surface unintentional tech debt, implement code quality tools, such as SonarQube.

The next step is to classify tech debt and assess its impact; both in terms of the risks it poses, and the efforts required to pay it off. These attributes later help with the prioritization process. Tech debt can be classified into various types, such as architecture, code, dependency, test, knowledge, infrastructure, and developer efficiency. While it can be hard to accurately estimate effort level for each tech debt item, T-shirt sizing is a reasonable approach.

Create awareness of tech debt by making it a part of the living backlog, along with functional, non-functional, and operational backlog.

4.????? Prioritize It

Not all tech debt is equal. Also, zero debt is not a goal. A healthy product would have a deep feature backlog and a manageable tech debt. One may ask, “How do you know if the tech debt is manageable, or it has reached a tipping point?” While the answer is contextual, it is worthwhile to define a tech debt ceiling for each module, similar to Service Level Objectives (SLOs) and Error Budgets used by Service Reliability Engineering (SRE) teams.

Armed with what, why and how much of tech debt, factor tech debt explicitly into the development process by including it in the conversations early on. Product and engineering leaders should review and prioritize the most detrimental tech debt together, along with everything else in the backlog. For large tech debt items, avoid the big-bang approach. Break it into smaller, manageable chunks. It allows teams to make steady progress and balance business needs of delivering new customer value.

5.????? Recognize and celebrate Tech Debt Payoff

Even though tech debt elimination is an internally facing effort vs shipping new functionality to customers, it is an equally hard job with equal business value. To build a culture of product and engineering excellence, it is important that tech debt initiative accomplishments are recognized and celebrated the same way as new product functionality.

6.????? Continuously Monitor It

Managing debt is a continuous endeavor. Simply addressing it once is insufficient. Consistently monitoring and evaluating the situation is crucial to ensure that debt remains manageable. It is important to establish metrics and key performance indicators (KPIs) to gauge progress in reducing debt. Regularly reviewing these metrics, tracking trends, and making informed decisions based on the data enables teams to remain vigilant and keep the software systems in a healthy, sustainable state.

In Summary

Tech debt refers to the accumulation of all the technological work that needs to be paid off in the future. Broadly speaking, there are three drivers for creation of tech debt, namely IOU. That is, Intentional decisions, Obsolescence of code over time and various Unintentional factors. It can manifest in different forms, including architecture, code, design, dependency, and infrastructure debt.

Just like financial debt, tech debt is a valuable tool that incurs costs and consequences. The consequences of excessive tech debt can include increased maintenance costs, decreased system performance, reduced scalability, difficulty in adding new features, and increased risk of bugs and errors. It can also hinder innovation, lower team morale and slow down development progress in the long run.

Managing tech debt as a business issue is crucial for software development teams. It involves recognizing and prioritizing areas of debt, allocating resources to address it, and making strategic decisions to balance debt reduction with new feature development. By actively managing and monitoring tech debt, teams can ensure the long-term health and sustainability of their software systems.