Practical Approaches to Tech Debt

"Tech debt" has become such a ubiquitous, yet commonly misunderstood term. In most circles it's treated like a four-letter word (or two four-letter words to be precise). People avoid uttering it out loud, as if it will jinx their software. Everybody likes to pretend that they're tech debt-free, yet everyone has it. A lot more than you'd imagine. I'll even wager that the software you rely on day-to-day runs on mountains of tech debt. If you Google the cost of tech debt, you'll find staggering numbers on the percentage of time engineers spend on it, and its multi-trillion dollar impact on GDP.

Despite how much has been written about it, I have not yet read about practical ways to address it. Ergo, this blog post. I try to introduce ideas that you may not have considered before, such as how to quantify the ROI of tech debt, who should implement it when it's unavoidable, and how to undo it more easily. Whether you're a junior engineer, a senior engineer, a product manager, or other, I hope you find some useful takeaways.

Background

What is tech debt?

This is my take on it:

Tech debt is the cost introduced from a suboptimal design decision that prioritizes seemingly short term gains and faster time to market over long-term sustainability.

If you have no idea what I'm talking about, it's possible you've been involved with tech debt before and didn’t even know it. If you’ve debated “short-term vs. long-term” or “tactical vs. strategic” trade-offs with engineers, you were probably discussing tech debt. If you heard that work will need to get undone or redone, you were probably discussing tech debt. If you’ve managed to piss off engineers during a design discussion, it’s likely you pressured them into creating tech debt (it's a huge pet peeve among engineers). There are other tech debt "smells", but you get the idea.

What isn't tech debt?

It's important to clarify that tech debt IS NOT the result of building some version of the software today that won't be able to do everything you want in the future. That's just building software incrementally, which is how all software should get built.

I often hear engineers push back on work requests for reasons such as, "if we build it this way, it won't be able to do XYZ next year, and that will be tech debt!", or "this design choice won't scale to infinity, therefore it's tech debt!"

My response to that is, "that's what all software development is about!" Knowing how to deliver something today and how to evolve it over time is the essence of good software engineering. And an engineer who can't build software incrementally and actually ship something is not worth a penny. The key is how to build something today such that next year it won't be a nightmare to add a new feature, or to scale it further.

Analogy Time

Imagine you need to get across a river. You and your friends construct a bridge made out of tree branches and rope in a few hours. It's hardly a robust bridge, but it does the job you need right now. This hack bridge may last for a long time. But is it ready to transport tens of thousands of humans, and heavy vehicles? Of course not. Is this tech debt? Nope. There was no conscious trade-off of quality vs. time to market, and there weren't expectations for this bridge to support heavier loads in the future.

Now, pretend you're constructing a massive steel bridge for commercial use. A year before the expected completion, the client suddenly demands that you open the bridge next month. This is an impossible feat, and you explain that one of the two decks hasn't even started getting constructed. Some wise guy then suggests to open up only the completed deck, and to construct a new temporary ramp that will lead to that deck. While the first deck is opened and being used, you'll finish construction on the second deck.

However, you soon realize that the temporary ramp interferes with where the main ramp was supposed to go, and due to it's hasty design, it's causing daily traffic jams. You now struggle to construct the main ramp, because the temporary ramp is in the way. You continue to come up with workarounds to support both ramps, which ultimately causes you more delays, more expenses, and a far from perfect bridge. You ultimately decide you need to close the first deck and demolish the temporary ramp, so you can properly complete construction on the bridge. Because of the extra work and complications, your project is now behind schedule and over budget.

In short, you implemented a poor design in order to achieve seemingly faster results. But the work to undo it and to redo it resulted in a net increase in time, cost, and complexity. This is the essence of tech debt.

There's a Real Cost

As alluded to before, tech debt has real financial consequences. It's not very different from financial debt. Many view tech debt as something that only idealistic engineers bitch about, but which isn't that important in reality. So if you're the type who only cares about the bottom line, and can't be bothered with good designs, or engineering happiness - I assure you that tech debt translates directly into a real financial cost.

Hopefully I have your full attention now. Here are a few ways that tech debt manifests into cost:

• Productivity - A system with tech debt is difficult to modify or upgrade. If it takes you 4x longer to build a feature because of its unnecessary complexity, you literally increased your costs by 4x. You wouldn't be ok with paying an engineer 4x the market rate as a salary, right? And you made your customers wait 4x longer, which resulted in additional financial losses, and negative impacts to your brand. By the way, 4x is an understatement - I've seen projects where something that should have taken one hour took one month, because of tech debt.
• Maintenance - software with lots of tech debt is fragile. One simple bug fix can accidentally introduce another 5 bugs. It is risky business to work with a fragile system. So on top of new features taking forever to build, your expensive engineers are spending most of their time fixing bugs instead of working on new functionality. Not exactly how you want to utilize the most expensive people on your payroll.
• Security - the more fragile your system becomes, the more security risk you introduce (i.e. you are exposing yourself to hacks and data breaches). Depending on the industry you're in, even one security incident can cost you your entire business.
• Turnover - no, I'm not talking about pastries. I'm talking about people quitting your company because you suck. Great engineers don't stick around to maintain software that's more of a shit show than a Jerry Springer episode. Great engineers want to work on great software, and they will eventually leave you if they're constantly patching up bad software. This goes for other team members too - product managers, sales people, designers, etc., won't enjoy working on a product that is on fire most of the time.
• Competition - if your competition is building features faster than you, and adapting to market changes faster than you, they will outpace you and you will lose. Tech debt is a guaranteed way to not be able to catch up to your competition.

All of these issues compound each other too, leading to the notorious death spiral. More maintenance -> lower productivity -> lower morale -> more incentive to keep adding tech debt instead of fixing it -> even lower morale -> employee turnover -> more new hires who don't understand your system -> more tech debt -> rinse and repeat. Eventually, this can lead to an entire product or company collapsing. I've seen first-hand how a team can fail to deliver one feature in an entire year because of the mounting tech debt.

Practical advice

Finally, the section you've all been waiting for. As per the title of this article, this would all be moot if I don't offer some practical advice. I split this up into three sections:

1. Tech debt mindset
2. How to avoid tech debt
3. How to address existing tech debt

Tech Debt Mindset

Don't Judge!

Despite the obnoxious / cocky tone of my writing, you must remember to never approach tech debt with judgement or ego. For starters, you genuinely don't know the context and history of the product you're working on. Tech debt is rarely implemented with bad intent. Rather, it's usually due to lack of understanding or awareness.

When you start addressing tech debt issues, be like Dale Carnegie - don't condemn, criticize, or complain. This is the golden rule. If you don't follow it, the rest of the points here are moot. If your first point of order is to call out how bad a code base is, nobody will listen to you or want to work with you. So try to be more like a doctor - diagnose the situation, and offer cures, in a non-judgmental way.

How to Avoid Tech Debt

Say no!

This may sound facetious, but I'm dead serious. Start adopting the mindset of not agreeing to tech debt, and start building software properly. Would a doctor cut corners because a patient demanded that a procedure be done faster? Hopefully not any doctor I'd go to. Would a chef serve undercooked food because the customer was yelling that he was hungry NOW? No. Then why should an engineer cut corners just because someone who doesn't know any better is insisting that you need to deliver things faster? The pressure may also come from your own engineering peers, who should know better. It's your duty to educate and influence others to take the right approach (which in my experience, will usually also be the fastest approach).

Just say no to tech debt (without getting yourself fired), and start saying yes to quality, modular, extensible code. Don't let others suggest to you how to perform proper engineering, especially if they're not engineers. Don't even fall into the trap of entertaining the idea and getting into long discussions about it. This doesn't mean you'll always successfully avoid it, but it sets the right tone with your colleagues and business partners.

You may be thinking "that's not advice, we have to in order to please stakeholders and get products shipped". Believe me, you won't please any customers with rushed, half-assed software. And while you may please your boss today, you won't be for long. But mostly, it boils down to ethics, and your identity as an engineer. Some engineers are artists. Others are code monkeys. The choice is yours.

Talk about it

Being aware you have a problem and discussing it is already a step in the right direction, even if your advice ends up falling on deaf ears. It's possible that your peers are not even aware of the situation or of the consequences. So bring it to others' attention. Present options, evaluate trade-offs, and explain your concerns to your peers and managers. Communication alone goes a long way.

Formalize your process

Compile a list of questions around tech debt, and incorporate them into a template that you use for tech specs / feature requests. When new work comes up, you'll naturally have a forum where you can discuss tech debt before any software work begins. It'll become a routine part of planning and engineering design, which makes things much easier.

Some sample questions:

• What are potential features / capabilities we'll want to add in the near future? Does our current design enable those, or prevent those?
• What is the time difference between Design Option A (tech debt) and Design Option B (proper design) in terms of time to market, resourcing, ROI, etc?
• If we choose Option A (tech debt), what effort is required to undo it, and to build Option B (proper design)? Does the total time to implement Option A, undo option A, and build Option B exceed the time to just implement Option B in the first place? The key is to include the time to undo option A. Once you do, you'll usually find that you're not saving yourself any time, and it becomes a no-brainer to choose Option B.
• How long can Option A last? (for example, if you know a poor design choice won't scale to 10 million users, and you plan to reach that number in 6 months, you may be in trouble. If you plan to reach that number in 5 years, you may be fine).

Get sign-off

I also recommend to formalize a sign-off process, and have engineering and non-engineering stakeholders acknowledge the following in writing (or via an email or JIRA ticket):

• An understanding that eventually the work will need to get undone and redone, and the estimated time it'll take to do so.
• A commitment on when you'll devote resources to addressing technical debt. Devoting time means that that's what you're doing that week, or that month. It does not mean you'll try to squeeze it in on the side "when you'll have time". We all know that means it'll never happen.
• List the mileage you'll get from the current design choice, so things don't come as a surprise to anyone later (i.e. when you scale to 10 million users and your servers are crashing daily).
• List the features that would not be able to get developed until this tech debt is addressed.

Put your most senior engineers on it

A senior engineer should know how to implement tech the right way and the fast way. It's often possible, but you need the right person. They will evaluate all aspects (speed, scale, security, time to rebuild) and make a sound judgement. A senior engineer will often be able to get you the shortcut you need today whilst keeping the door open for the longer term work. Again, it is sometimes necessary to do a "hack job", but a strong engineer will implement it in a way that it will be trivial to undo.

Build to the interface you want

A senior engineer will find this tip obvious, but not everyone reading this is a senior. Tech debt is often the result of improper architecture, or messy code. However, if you wrap your crap in the interface you'll plan to have in the future, most of your problems go away.

For example, you may need to display data from a new source in your app. You don't have time to design a full schema, deploy a database, wrap it in an API, and have your front-end call it. I've seen hack jobs where engineers will have the front-end load a CSV file directly. Before you know it, this monstrosity grows, and you find you have 5 apps loading an entire collection of CSVs.

If you had built a light-weight API that behind the scenes loaded the CSVs, at least your front-end could have called a simple API. Later, when you had to undo this mess, at least the front-end wouldn't have to change, and the tech debt would have been isolated to what's behind the API. Same goes for any interface, whether it's a function call, a class, etc. Limit the blast radius of your bad designs by wrapping it in a clean interface. This small extra effort is going to buy you a ton of mileage, and can be a good balance between shipping code faster, and not horribly ruining your code base.

I said it before, and I'll repeat it here - knowing how to build modularly and evolve the code is the essence of good software engineering. And a trick to doing that is to always think about the interface you're exposing, even if behind the interface is a big piece of crap.

Document it to death

It's not always easy to understand your own code after enough time has passed. It's harder to understand someone else's code, even when it's good. It's nearly impossible to decipher bad code, especially without the context of why the decisions were made. So be a good samaritan, and document the context, the decision, and the tech debt implementation. Litter your code with notes about where you'll need to undo or redo things. Not just for others, but for yourself. I guarantee you won't remember all the details when you revisit this code in a few weeks.

How to Tackle Existing Tech Debt

Now that I shared some ideas on how to avoid tech debt, let's discuss how to address existing tech debt. It may be your own, or it may have existed before you joined the project you're on.

Most of the tips from the previous section still apply - talk about it, evaluate trade-offs, get others to commit time and resources, etc. And definitely don't continue the cycle of mounting more debt upon the existing debt.

The biggest hurdle I've seen to addressing tech debt is getting the green light to invest the time and money into it. Business stakeholders may want more features, and not have an interest in (or understanding of) the importance of tech debt. Maybe even your engineering peers are not inclined to take on the challenge. It could be out of fear, lack of motivation, or lack of understanding (e.g. I used to work at a financial firm that hadn't upgraded its Fortran code because "it works and it's risky to touch it". P.S. It didn't f*ing work). So in addition to tech tips below, there are psychological and political tips.

Strangler Fig Pattern

This is similar to the prior tip of "wrap your crap" in the interface you want, but in reverse. Martin Fowler coined this term as an analogy to rewriting software. I won't do it justice by describing it in detail here, so feel free to Google it. The general idea is you wrap existing tech debt behind the proper interface, and then you have the freedom to change the messy code under the hood layer by layer without downstream consumers being affected (similar to how this plant wraps itself around the branches of an existing tree, and gradually works its way down, strangles the existing tree, and eventually replaces it completely).

Pitch the ROI

When you are at the planning table, propose your ideas to address tech debt. Get specific about what you'll do, and why it's important. Quantify the ROI you expect. If you can frame the work from a customer or revenue perspective, you'll increase your odds of getting stakeholder agreement.

For example, don't request to "fix API tech debt". It sounds vague and pointless. Rather, pitch that "we can decrease page load time by 500ms by refactoring API code, which studies have shown can increase customer sales by 3%". The work you'll do is the same, but if you know your audience and speak their language (i.e. the language of $$$), you'll get more cooperation.

Pitch the "Negative ROI"? (an Isaac Trademark)

It's not always obvious what the ROI is, and perhaps there isn't an immediate benefit. But as an engineer, you just know the code needs work, otherwise disaster may hit in the future. So highlight the negative ROI - what is the potential cost or business loss if the tech debt isn't addressed. You can also include the probability of such an event happening. Again, try to quantify it and tie it to customer value / revenue.

For example, don't say "if we don't address tech debt in our API, it won't scale!". Rather, say "in the past 4 months, we had 8 incidents where the server crashed due to a high load, which caused 40 unhappy customers emails, 5 dropped customers, and a loss of $6,000 in revenue. If we invest 5 days into refactoring this code, we'll eliminate these types of issues. If we don't address it, and extrapolate these numbers, this can continue happening 24 times per year with a potential loss of $18,000" (or more, because as we grow our business, this will happen more frequently).

Document it

Keep a log of tech debt items, framed in a business context. Whenever you're up for a discussion on planning work, or prioritization, those items will be front and center to the decision makers. Make sure to point out which future features in your company's roadmap will not be doable until that piece of tech debt is addressed. It's also helpful to see the amount of tech debt items overall, rather than randomly point out individual items as they come up. Seeing the totality of it can help raise awareness.

Squeeze it in

Pad your estimates, and as you address ongoing work, try to tackle some tech debt along the way. Document the undocumented code. Refactor the duplicate code. Automate manual steps. It's kind of like cleaning your house - you can do a lot of little things that over time amount to a lot. And you can at least leave the code you worked on in a slightly better state than when you found it.

Summary

If there are 1-2 takeaways you should leave here with, they should be (1) to treat tech debt like financial debt, with financial consequences, and (2) to stop caving to the pressure of implementing poor code to meet deadlines. If you were considering taking on financial debt (e.g. mortgage, student loan), you'd probably consider the following:

• Trade-offs - what will be the cost vs. benefits of taking on this debt
• When and how will the debt be paid off
• Before it's fully paid off, how will it affect you? For example, will it limit your spending in other areas, like buying that new car?

Treat tech debt with a similar mindset, and you'll see better results.

Originally published at https://www.isaacpalka.com.