Managing Technical Debt with Imran MacMillan, NTT DATA Services

The following was transcribed from a recent interview on The Agile Brand with Greg Kihlstr?m podcast.?

Today we're going to talk about technical debt and how companies can manage this as they continue to feel the effects of pressure to continually modernize and implement more features and services to serve customers effectively.. To help me discuss this topic, I’d like to welcome Imran MacMillan, Director of Application Modernization at NTT DATA Services.

[Greg Kihlstr?m] We're going to be talking about technical debt, what it is and how companies can manage it. So for those less familiar with the term, can you just explain what technical debt is?

[Imran MacMillan, NTT Data Services] When people talk about technical debt, I feel like they're often speaking from the perspective of software inadequacies related to design or development. And sometimes even the term “technology debt” is used to broaden the scope of assessment. But for our purposes here, I'll use those terms interchangeably. And I can just quickly describe two large buckets of debt that arise in IT shops. One definitely stems from poorly written code. This has the ability to bloat your applications and prohibit your ability to develop new functionality. And this is often referred to as “cruft.” And effectively, that's accumulated low-quality code due to either bad practices, rushed implementation or just error-prone software efforts.

Now, the flip side of debt is poorly designed architecture or systems that can be slow, insecure, reaching end of service life or overall just limiting business innovation. And there's lots of individual examples about how technical debt manifests, ranging from antiquated databases that crash all the time to legacy mainframe operations that slow down user interfaces or even reduce developer productivity due to coding in a monolithic framework. So those are just two broader perspectives on what exactly is technical debt.

I appreciate you breaking it down like that because I agree. It's – sometimes there's a few things that get conflated there. You touched on this a little bit, but how is technical debt typically accumulated? And is there a way to avoid it? Everyone, to some degree, is going through some kind of transformation or ongoing initiative or something like that. Are there ways to manage technical debt before it becomes overwhelming?

Those are two great questions. How does it come about and how can you manage it??

Well, first of all, to your point, businesses are sentient beings. They evolve; they grow; they adapt. And as your customer segments change, as new business lines emerge, and as overall strategies for engagements and services are updated, the technology behind them doesn't always keep up. You might put off your database framework update because your customers are only seeing the UI/UX. Or you might not want to re-platform your mainframe because, if it's a massively expensive and risky undertaking and the way it's orchestrated right now is keeping your websites afloat. So that's definitely one way that the debt accumulates, is just through the evolution of business.?

And another is really closely related to the perilous software development triangle. And by that I'm referring to speed, cost and quality. If you're trying to minimize your time to market, you’re trying to minimize your cost of development, a lot of products are rushed to users with inadequate testing, lack of consideration for edge cases, or simply poorly structured code that fits the purpose but limits extensibility and future development. So that's another major factor for accumulating debt. And, finally, you can't ignore cost. You know, when you're talking about re-architecture, some major undertaking and even a smaller-scope refactoring of individual applications can definitely be daunting. It takes time, and in most – in most cases, it takes a lot of manual effort and very skilled resources to unpack that cruft, the stale frameworks, the bad design choices, and rewrite to a modern security-minded design pattern.

So over time, these three factors can definitely relate to the increase of technical debt. So you might be wondering, how do you manage that? Because it almost seems inevitable that you will accumulate some technical debt along the evolution of your business.?

There are definitely strategies to mitigate that. And, first of all, you definitely have to do a good job aligning your business forecast with your IT roadmaps, as much as possible. And of course that poses challenges. But one thing you can do is really enable and empower cross-functional product development. You need to bring in stakeholders from enterprise IT management, from software design and development, product owners, application owners, and business stakeholders, so that they can all engage in conversations together about what the future looks like and what their needs are.

Another major thing that you can do is establish architecture review boards or other means of IT governance that are critical for successful technology-enabled businesses. You can't just throw out any software to user communities. You have to ensure that there's code review, architecture review, security posture review, and adequate testing ahead of release. Having software templates or enforcing software standards for development and testing also definitely helps with that effort. And, finally, something that I think should be a part of just about any IT enterprise annual deliverable is a technology replacement plan. Understanding the service lifecycle of your major applications and platforms that enable your business is key. You have to plan around customer support windows that might be ending, frameworks that are being updated. Some software libraries might be getting sunsetted. And eventually that leads to unsupported applications or systems. So planning that long-term IT investment roadmap around capacity planning and aging technologies is absolutely vital.

Moving from the more technical focus to the rest of the business and strategy, what kind of business issues does technical debt amplify? And how can you jointly find a solution for business optimization and expansion and user experience or customer experience with technical solutions?

That is a pretty weighty question. So I'll try and unpack it a little bit. Quite frankly, a lot of people, you know, look for service partnerships to answer this exact question because it is something that takes time to unwind a little bit. So the way I see technical debt is almost like a ripple effect. You know, it starts as a limited-scope issue and eventually it has the ability to impact operations, staffing, development velocity, and even user experiences.?

So, when we're specifically talking about some kinds of topical business issues in the current enterprise climate, there's definitely a few that come to mind. One of them is enabling business innovation and agility. You know, one poorly written function or model has the ability to systematically hamper new development. If you're trying to figure out how to extend a particular service or achieve a greenfield functionality or even integrate with a new application but you can't understand how it was initially implemented due to bad documentation or confusing code, it takes a lot of analysis, potential rewriting, and that's just to get something functional. So developer cadence is much slower, and ultimately the business issue there is the limited agility to respond to new customer segments and opportunities.

There's also another business issue of how do you bring customers a modern UI/UX, or even CX. You know, if your middleware layer or your data layer is already topping out its performance or capacity limits and you're trying to implement new user features like graphing dashboards or sales charting or service reports, your capacity limits might just provide an unusably slow UI. So that's definitely a critical business issue, especially with the focus being placed on customer-centric design principles and human factor design.

And one other common business effort, especially these days, is how do you build a 360 degree customer profile. Well, you know, this often requires pulling and integrating data from different tables and services. And depending on how your architecture is set up, it might be hard to redraw those boundary layers or interface requirements. So that's just another business use case that technical debt hampers.

When it comes to actually finding solutions for business optimization and aligning that with a technical solution, there's definitely some modern paradigms that help. Well, first of all, there's a reason that application modernization is largely geared towards developing microservices for app platforms. They provide extensibility, portability, and they allow for development of enhancements or new functionality with limited impact to other apps and services. And these microservices go hand-in-hand with container frameworks. While you can deploy a microservice-based application to a VM or any kind of virtualized environment, you definitely gain greater efficiencies by deploying to a containerized runtime environment. It presents a consistent software framework for developers. They don't have to worry about configuration changes in production that might impact the usability of their apps. And these containers are also much lighter than traditional virtual infrastructure since they don't have to support multiple copies of an operating system and have the associated execution environments. So you're pretty much dedicating the maximum amount of server processing capacity to running the applications.?

So those are just a couple of the modern paradigms that, sort of, align that business optimization with technical solutions to tackle technical debt.

When a company wants to modernize their applications, what are some of the challenges that companies can expect in achieving transformation due to technical debt?

There's a plethora of challenges that companies can run into. Rather than get into all of them, what I'll probably do here is just talk about some of the less obvious ones. But, you know, transformation can take the form of migrating to a new software framework, replacing an old technology stack, redesigning or re-architecting business-critical applications, retiring a mainframe, moving to a new Cloud provider, or even integrating a new customer-facing application. So obviously there's so many challenges that can happen in even just one of those efforts.?

So I'll just try to talk to some of these less obvious challenges. And one of them is organizational change management. OCM is something that we're seeing come to greater and greater focus, especially for people like NTT and service integrators. One of the keys to success is adequately planning handover upscaling and right-sizing your workforce. If you fail to address the fact that your current employees might not be the best positioned to manage a new technology platform, that can definitely lead to a lot of operational issues. And the new products and architectures that are delivered by a third party, without adequate training, you're probably not going to be able to manage and operationalize them in an effective way. You just don't want new systems thrown over the fence without enough context to go with it.

Another challenge that might be a little bit discounted these days is recruiting and staffing is not easy when you have a very distributed IT environment that spans across legacy and modern technologies. Obviously, we are in, quote/unquote, “the great labor shortage,” but even beyond that, it's not necessarily as easy to find a COBOL developer or an Oracle database expert as it is to find a Python developer. You know, there's a reputational risk for organizations that are working with old technology platforms. And many new or junior developers aren't going to want to take on and potentially limit their skill development when they could be training in more in-demand areas.

And another issue that people don't really take into as great consideration is the fact that you're probably going to have to instantiate some level of parallel services as you move to a new environment. You know, whether it's a dev ops platform using containers in a public Cloud or just a new database that's being managed in a snowflake environment, you're going to probably need multiple production environments running in parallel during cutover and transition to ensure continuity of operations and evaluate failover. So this presents a high-cost, high-risk and general fear, just in the IT shop, of what operational issues might occur when there finally is that hard cutover. So Hopefully that gets to some of the challenges that companies can expect while they're going through their transformation efforts.

To take a step back from that, you've touched on several things here, but what are some of the risks and common pain points of not assessing technical debt or modernizing applications, you know, before even getting into it and actually undertaking the effort?

So the old adage, “Let sleeping dogs lay,” definitely does not apply to technical debt. So I'm just going to go back to my ripple effect analogy and, sort of, talk through some of the outcomes of looming technical debt. First of all, like most things in life, the longer you spent ignoring an issue or working around an issue, the more expensive it will become to eventually remediate or fix it. You know, if you're talking about a system that's been in use for multiple decades or has been passed through multiple dev and product management teams, the accumulated debt can be almost insurmountable, to the point where the only realistic option is to completely rebuild that service or application. So when you're comparing the cost of rebuilding a platform versus incrementally implementing standard software patterns, code quality control, then you're talking about a huge liability that is much larger than incrementally modernizing along the way.?

Another really common pain point is it can be very, very difficult to implement modern development and delivery methodologies like agile or scaled agile, like SAFe, to a program that is primarily working on legacy technologies. Development takes longer; release cadence is slower; and incremental delivery is very difficult to achieve. So that means there's going to be much longer deltas between product conception and user acceptance. And that just spells a pain point for any organization trying to adapt or evolve in today's market.

So one of the topics that I also touched on earlier is just enabling business innovation and agility. I mean, that's the key goal of just about any IT shop these days. So the more technical debt and aged applications that you have, the less likely you are to be able to quickly respond to new customer needs, develop new business lines, or even tackle emerging markets.?

One final point that I think is probably not as reported is, so in today's IT climate, there's cutting-edge solutions for just about every part of business integration, whether you're looking at your customer relationship management, you know, your transaction processing, your decision support, or any other critical system to your business functions. You might be looking for new partners who can drive smarter business intelligence, maybe bring AI to your software development, or even automate your deploy and integration process with a dev ops pipeline. Integrating legacy frameworks and applications with some of these new solution providers can

definitely be an uphill battle. And it would be significantly more time- and cost-consuming than if you already had a containerized public Cloud deployment of your apps that had key functions available via an API.?

So what happens, this also trickles into the development of competencies and certifications. If you wanted to be accredited or auditable from a software perspective, and let's just say you're searching for that AWS certified competency or partner validation, then, if you're not using modern build paradigms, it becomes much, much harder to attain those actual service validations. So what happens is there's actually a deconstructive flywheel where less people are attracted to work for your organization because they know that your legacy services are hard to build and maintain to, and it leads to even less certifications being handed down to the organization because you're not able to retain talent. And, pretty much, this flywheel works in reverse to make your life a lot harder to adapt to new business challenges.

So let's briefly look at this from the end customer perspective. We've spent most of our time here talking about, within the business, whether it's technology or business owners, or things like that. Although technical debt certainly is an internal issue to an organization, it does still affect the end users of products. So how can companies identify their application delivery standards and determine business value of improving some of these services related to the customer experience?

You know, that's a great point. I mean, customer experience is often the beginning of any new development or design effort that goes on today. And so it's absolutely key that, in today's business climate, ease of use is a priority for greenfield development. One way that you can really get to how you're determining business value from that CX is customer feedback loops. They're a great source of validation for your application's ability to drive value. And, you know, you can conduct this through interviews, surveys, built-in questionnaires, or other creative, useful tools that ultimately are evaluating customer success. You want to build up the kinds of data intelligence and metrics that can tell you, is there repeated use around a particular service or application? How are you defining your customer retention? How are you calculating the duration of interaction with your apps and services? All of that is useful information.

And if you see that, for example, a customer is very unlikely to use a particular service after their first attempt, or if there's a sense of frustration by, let's say that 80 percent of users stop interacting with a webpage after trying to access a particular report that doesn't populate within five seconds, then that's very telling information on what needs to be improved first to drive customer engagement, which ultimately drives business value.?

So we're always trying to reduce customer friction, you know, enable more responsive interface designs to drive engagement. And it can be difficult to, sort of, translate that feature availability into actual perceptible business value without adequate business intelligence and data gathering. So it's really critical that you bake in schemes for understanding your user interaction with applications early into the development process so that you can have those positive feedback loops that really tie together the outcomes into IT services.

Well, one last question before we wrap up here. You recently wrote a blog post on calculating technical debt. Can you talk a little bit about that and how an organization can measure and manage the amount of technical debt they have and determine what to prioritize in order to invest in that?

This is something that a lot of organizations are interested in getting a better handle on. They come to, you know, providers like NTT to really scope and assess what their technical debt is. And, quite frankly, there's no automated or easy-bake solution for getting this out. But, really, the process begins with a true inventory of your application and systems portfolio. If you don't have a good understanding of your IT footprint, especially in today's world, where that can span on-prem, multi-Cloud, diverse SaaS products. You know, it can be very hard to relate your business to your IT operations. And that's really step one, understanding, like you said in the previous question, where the business value comes from, as it pertains to your IT systems and applications.?

You know, determining or defining which services and applications are directly utilized by customers can be a good starting point. But to do that, you need to have solid command over the business processes and rules that enable that commerce and user interaction. Failure to recognize that, underlying services, possibly at the data or network tier, can really prohibit you from prioritizing your investments correctly.?

What I’m really trying to get at is there needs to be a holistic approach to assessment and ultimately transformation. So measuring technical debt can cover anything from deconstructing your user interaction from front-facing websites and mobile apps to understanding your limitations in your current database architecture, or even recognizing problems with your security posture that can lead to unwanted attacks or malicious intent to damage or extract system information. It can take a really pretty in-depth analysis to get a good handle on calculating your technical debt.?

And how do you do that? Well, you have to interview app owners. You have to interview business stakeholders. You have to analyze code application in systems. And you have to do a holistic review of your architecture. And, honestly, the things that I would do just to create and define a roadmap are bucket things into target categories. And so one way you can do that is have your hot list, the things that need to be updated or modernized in the next – in the next six to 12 months, not only to ensure business continuity but to ensure adaptability to future customer demand. Then you can have a medium-term forecast. You know, put your systems apps and services in a bucket that need to be updated or modernized in the next 12 to 24 months. And then you can go through that same process for a bucket for the next two to five years.

So really building that roadmap of priorities and what impacts the customers in the business the most is really highly important. And it's also important to revisit that plan often and recalibrate. Because, as we both know, technologies are released faster and faster, new frameworks, new softwares, new SaaS products. And they might be better fits for your business. So there's definitely a revolving nature to this effort. It's definitely not something that's a one and done deal.?

And so companies that approach technology refreshes or tech debt assessments as a one-time deal are probably not going to have the most success. That's definitely why we like to emphasize the continuous nature of digital modernization in our offering at NTT. And we bring a lot of that to our customers and clients.

Listen to the Episode

Prefer to listen? Click play on the video below to listen to the episode.

About the Guest

Imran MacMillan brings a passion for digital transformation and building customer-centric products to our team, as well as experience in cloud enablement, application modernization, infrastructure delivery and product management. In his own words: "I’m excited to be part of a team that is finding better ways to deliver technology improvement platforms for organizations small and large. In a sector that’s constantly undergoing change, building resilient strategies that can enable our people to transform a client’s digital footprint is what I look forward to most.”

Before joining NTT DATA, Imran had the opportunity to work on major enterprise cloud migrations deploying hundreds of applications to millions of global users. From developing applications and infrastructure, to building migration frameworks for active datacenters, he has a wealth of experience across the technology services domain.

In his free time, Imran enjoys mountain biking, snowboarding, and reading a good book on the beach. He is based in Monterey, California.

About the Host, Greg Kihlstr?m

Greg Kihlstrom is a best selling author, speaker, and entrepreneur and host of The Agile Brand podcast. He has worked with some of the world’s leading organizations on customer experience, employee experience, and digital transformation initiatives, both before and after selling his award-winning digital experience agency, Carousel30, in 2017.? Currently, he is Principal and Chief Strategist at GK5A. He has worked with some of the world’s top brands, including AOL, Choice Hotels, Coca-Cola, Dell, FedEx, GEICO, Marriott, MTV, Starbucks, Toyota and VMware. He currently serves on the University of Richmond’s Customer Experience Advisory Board, was the founding Chair of the American Advertising Federation’s National Innovation Committee, and served on the Virginia Tech Pamplin College of Business Marketing Mentorship Advisory Board.? Greg is Lean Six Sigma Black Belt certified, and holds a certification in Business Agility from ICP-BAF.?