Technical Debt: A Pernicious Problem with Big Consequences

Businesses that rely on technology must deal with many threats that are common to all businesses: competition (both direct and indirect), supply chain problems, sales revenue shortfalls, shifting market dynamics, legal challenges, and more. But unlike those external threats, there’s an internal threat that’s unique to technology-dependent businesses—a threat they often fail to understand, recognize, and address—and it’s a threat that can slowly but surely kill the business. That threat is something called technical debt.

What is Technical Debt?

Technical Debt is work that should have been done but was not done. Most often, the kind of work involved is the kind that does not deliver immediate, obvious value, but is necessary for long-term sustainability.

A common example of this kind of work is software code refactoring, in which a developer edits (nominally) working code to make it more efficient, more robust, and easier to maintain and extend. Compared to the original code, the refactored code does not appear to deliver visible incremental value to customers in the short term; it seems to work just like the old code did. However, without regular refactoring, the code base tends to become slower, more fragile, and harder to modify without breaking existing functionality. The value of the investment in refactoring becomes evident only over a time period that’s far longer than the fiscal periods on which most company managers focus.

Technical debt isn’t limited to the software world; there are plenty of examples in other fields. If you don’t get around to fixing a small rip in your clothing, it will soon become a big tear. If you don’t fix a leak in a roof or a pipe when it’s small, you will likely end up with a much bigger leak, as well as rot that requires major reconstruction to remedy. If you fail to keep up the maintenance on your car, you increase the risk that it will fail in some expensive (and perhaps dangerous) way.

Let’s consider a few examples of technical debt that have made news headlines in recent years.

An Airline Meltdown

Just before Christmas 2022, severe winter storms swept across the U.S., forcing thousands of flight cancellations throughout the country, and stranding a million travelers. Though all major airlines canceled flights when the worst of the storms hit, most resumed near-normal service a day or two later. But one airline — Southwest— had some 80% of the flight cancellations (some 16,000 over the course of 4 days). Why did Southwest have so much trouble recovering?

One factor was the severe weather, which shut down several airports key to Southwest’s operations. Another factor was staff absences due to illnesses (such as COVID, flu, and RSV). Some people suggest that Southwest’s point-to-point routing makes recovery from disruptions more difficult than it is with the hub-and-spoke routing used by most airlines (though other people believe the opposite is true). But the majority of the blame has been assigned to Southwest’s SkySolver crew and airplane scheduling software, which dates back to the 1990s. The company admitted that SkySolver was unable to track and coordinate all the pilots, crew members, and airplanes under the rapidly changing conditions of nationwide disruptions, and with so many crew members and airplanes out of position, cancellations snowballed.

Southwest had not updated its computer systems since the 1990s, despite the airline’s changeover to point-to-point routing and significant growth in flight volume in the past 20 years. Its failure to modernize its computer systems left the company vulnerable and unable to recover quickly from major disruptions. Southwest estimates that total costs it will have incurred from the meltdown to be $750-850 million.

A summary discussion of the causes of the December 2022 Southwest Airlines meltdown is provided in this CNN Business article.

Another Aviation Disaster Story

For many decades, Boeing enjoyed its reputation as an innovative, world-class manufacturer of airplanes that dominated the commercial aviation market. Its 737 mid-size passenger jet became the most widely-sold airplane used by commercial airlines.

In 2017, Boeing’s 4th-generation of that plane, the 737 Max, began flying commercial routes. In 2018 and 2019, two mysterious fatal crashes caused the Federal Aviation Administration (FAA) to ground the planes while the causes were under investigation.

The investigation revealed that the crashes were caused by a flight stabilizing system that forced the plane nose down. The 737 Max used the same basic airframe as the original 737 series, but was fitted with larger, higher-efficiency engines that had to be mounted more forward and higher than engines for which the airframe was originally designed. This change increased the tendency of the airplane’s nose to pitch up at high angles of attack, which would create a risk of stalling.

Rather than undertake the long, expensive process of redesigning the airframe, Boeing modified its flight stabilizing system to force the nose down when a high angle of attack was detected. Pilots were not informed about these changes, so when false readings from faulty angle-of-attack sensors caused planes to mysteriously dive, the pilots did not know what was going on and how to deal with the situation.

The 737 Max debacle had costly consequences for Boeing. The company paid $2.5 billion to settle a fraud lawsuit related to the certification process for the airplane, and estimated that it lost some $18 billion in 2019 alone due to increased costs, lost sales, victim litigation, lowered credit rating, and lower stock value. Though 737 Max airplanes are flying again following corrective actions taken by Boeing, the company is still in a much worse position than it had been before the debacle.

Why did Boeing make choices it did with the 737 Max project? The company was facing competitive pressure from Airbus, which had introduced its A320 Neo series in response to market demand for better fuel efficiency. Seeking a way to bring a more efficient 737-class plane to market as quickly as possible, Boeing went with a plan that would involve making minimal modifications to the well-accepted 737 that pilots knew how to fly and airlines knew how to support. The compromises that were forced by this plan generated substantial technical debt, which has created huge losses for Boeing and severely damaged its reputation.?

Overwhelmed Government Computer Systems

I live in California, where outdated computer systems of several government agencies have been the topic of national news in the past few years. The California Department of Motor Vehicles was swamped with new transaction demands when the Real ID program was implemented, and its systems and processes were overwhelmed, leaving customers waiting weeks to complete all kinds of transactions, including renewals of licenses and registrations.?The state’s Employment Development Department (EDD), which distributes aid payments to citizens who are having trouble finding employment, was unable to effectively manage distribution of federal relief funds that were allocated soon after the onset of the COVID-19 pandemic, leaving millions of citizens at risk of eviction.

In both of these cases, the agencies were reliant on computer systems that had been designed and implemented decades ago—systems that were not capable of handling significant new demand loads, and could not be easily modified to add capabilities. Underinvestment in government computer systems is common, because it’s difficult to get funds approved for modernizing and updating systems until a crisis makes the need obvious. As is common with technical debt, the risks and costs are not seen and appreciated early.

The High Costs of Technical Debt

The costs of technical debt are truly staggering. According to a 2022 report by the Consortium for Information and Software Quality (CISQ), the total accumulated software technical debt in the U.S. has grown to about $1.5 trillion.?This figure represents only principal, namely the amount of money that would have to be invested to fix the software. Beyond this principal cost are many other costs, including:

• costs of operational failures and security breaches
• increasing difficulty of maintaining and extending products
• increasing risk of introducing new defects with every product change
• ever-slower development (decreasing development velocity)
• decreasing developer satisfaction, leading to poorer retention
• slower delivery of value to customers
• lower product quality
• increased risk of losing to competitors

The impact of technical has become so great that the topic has its own international conference. First held in 2018, the International Conference on Technical Debt brings together software researchers and practitioners to discuss approaches for managing various types of Technical Debt, to share experiences and best practices, and to identify key challenges for industry and academia. The 2022 conference was conducted online; the 2023 conference site is Melbourne, Australia.?

Why Do People Allow Technical Debt to Accrue?

There are many reasons people allow technical debt to accrue, including these:?

Lack of Awareness

Many organizational leaders do not know what technical debt is, why it’s important, and how to limit the risks it creates. As with any problem, the first step in mitigation is developing awareness and understanding.

Competing Priorities

Keeping a long-term perspective is difficult when you are confronted with many issues that demand your attention and action. It’s easy to get focus on near-term tasks that can be completed and measured; longer-term initiatives that don’t provide immediate payoff are much harder to undertake and sustain.

Bias Toward the Dramatic

Putting out a fire is always more dramatic than preventing a fire in the first place. We tend to place higher value on curative efforts that produce an obvious, dramatic improvement than on preventive efforts that mitigate problems so that they don’t become dramatic. Getting leaders to recognize the value of preventive efforts is challenging, but it’s essential for the long-term health of the organization.

Not All Technical Debt is Bad

There are times that taking on technical debt makes sense, just as there are times that taking on financial debt makes sense. If you’re buying a house or founding a company, you may need to assume some financial debt in order to get started. For that debt decision to make sense, you have to have reasonable confidence that you’ll be able to pay down that debt in an appropriate timeframe.

Similarly, taking on technical debt may be a good choice if, for example, you need to get a product to market in time to exploit a specific opportunity. You need to be aware of the risks that the technical debt creates, and have a plan for reducing that debt within an appropriate timeframe. (In some cases, you may decide that reducing the technical debt is not worth the investment, such as with products that will only be used for a short time before being retired.)

Martin Fowler classifies how organizations take on technical debt using a 2 x 2 matrix, with quadrants defined by Deliberate vs. Inadvertent, and Reckless vs. Prudent:?

Not all technical debt is introduced deliberately; a certain amount of technical debt cannot be foreseen and is thus unavoidable (represented by the lower right corner of the matrix). What’s important to avoid is the Reckless half of the matrix, because whether the debt is introduced deliberately or inadvertently, reckless behavior will likely lead to disastrous consequences.

Pay Attention to Technical Debt, and Manage It Appropriately

I hope this article has helped increase your awareness of technical debt, and given you some insights on how to think about it. As the use of technology continues to grow, the impacts of technical debt will also grow, and we all need to recognize the risks it brings so that we can make appropriate decisions to manage those risks.

Jim Schibler leads product management teams that deliver software experiences customers love, and he coaches professionals on job search and career management. He writes on a broad range of topics; see more of his articles at his website.

Image credits: Southwest Cancellations courtesy Matt Marton at Reuters; Boeing 737 Max courtesy Boeing; DMV courtesy Omar Bárcena at Flickr.com; Iceberg courtesy Wikimedia Commons.

Copyright ? 2022 Jim Schibler — All rights reserved