What are the best practices for dealing with legacy and poorly designed software systems?

Only change what needs to be changed, and then replace things strategically and incrementally. A good pattern to do this is the "strangler pattern" which uses selective re-implementation of services. This can be done most easily when there are clearly defined interfaces from the outside, and understanding which APIs are there and how they are used therefore is a important aspect of choosing this strategy.

I have worked in the embedded automotive software systems for about 2 decades now. I have followed three methods to handle this situation 1) Document the detailed design and design decisions- this helps the new comers in the team 2) Understanding of system is often not easy for new engineers. Trainings become necessary at every step. Each handover costs knowledge gap. solution is to video record the expert sessions 3) Every change must be understood thoroughly with a structured analysis like DRBFM - Design review based on failure mode. Refactoring often introduces new problems that cost money. Never touch the running system until unless absolutely necessary.

While legacy systems often carry a lot of technical debt, often they have evolved over many years with new features being introduced. An audit of features actually used is a good starting point. We’ve worked on projects in the past where 50% of the system can be moved out of scope as it’s simply not used anymore.

Understand, understand and understand. Make sure that there are either no behavior changes in a change or that the only behavior changes are the ones intended.

First point is to get the functional part: - What is this system for? - How good or bad is it working for its users? - What parts are the most used/critical The second is to start with empathy: - Assume the people that built it did their best with the information they had at the time - Remind yourself that software is built under less than optimal circumstances - that's true for that one too - and that while companies evolve, the software tend to stay Analysis: try to combine talks with developers working on it (to get the whys) and code analysis. If it's big, start with running some tools on it. It won't give you answers, but it will give you area to watch.

Refactoring the code might be very expensive. Moreover you need to rely on robust testing to ensure intended functions work correctly. You would go to this when the code is non maintainable at all and you are required to do a major change. Always remember, if it works; don't fix it.

Refactoring, indeed, adapt to your company conventions. And this sometimes may unveil slipped mistakes as to what variables are really used and for what reason. There are static code checking tools available, which help a lot, but less to never useful when the logic is checked.

When dealing with legacy systems it is most important rule I have - "First, do no harm" (similar to what medical doctors have). If there is a slight risk that your refactoring might break something - don't do it.

Refactoring can cost a lot. Try to refactor to a new system rather than the existing one. Try to create plugin code that replaces the existing legacy code if possible. (microservices)

Divide and concur! When reviewing the working system try to find the high frequency interacting clusters of code and isolate them from each other and the rest of the system, breaking the system into small manageable testable pieces. Essentially, you are applying a conforming transformation keeping all the existing functionality intact. Then you can refactor each isolated fragment independently, thus enabling a parallel work for multiple teams, including QA! Those blocks might become micro-services, or reusable widgets, or whatever practices your office of Architecture recommends.

Isolation and segregation of concepts and components is to be considered anyway, in many cases, not only for this one. Keep in mind, though, the safety dimension, and where is the case, the performance impact, especially for critical actions and communication.

Mitigation of risks is a crucial factor that has to be considered and legacy systems can be fragile. Before making changes, creating a robust set of automated tests to ensure that existing functionality remains intact is essential. This will act as a safety net during refactoring and prevent regressions. Sometimes, a complete rewrite is not feasible due to time and resource constraints. Considering an incremental approach where you modernize or refactor smaller parts of the system with each release is a gradual approach and can reduce the impact on users and stakeholders.

Its always better to consider and apply separation of concern design principle whenever its possible. To be able to do this, you must understand the system and break it into functional units. And then determine the dependencies between those units. Then those units are integrated by implementing their dependencies and the result should be something like lego architecture. Where the final product could represent a building built by integrating those lego units together.

Isolate the system, but most importantly, break the system into parts, and isolate those parts one from another! This is the best way to concur the complexity of that legacy black box your are challenged with.

Change management and organizational maturity. Isolating legacy systems can be seen as a form of software "quarantine," preventing the spread of technical debt and design flaws to newer, cleaner systems. This isolation isn't merely a technical barrier but can also represent a psychological and organizational boundary. It delineates the old from the new, the problematic from the well-designed, and in doing so, it can shift the organizational mindset. This demarcation encourages a culture of continuous improvement and adherence to modern best practices in the newly developed systems, while also acknowledging the existence and containment of past mistakes within the legacy system.

Quite often, legacy systems support only outdated integration and/or authentication protocols, causing pain and additional overhead when integrating with them. This is especially costly when many teams or services have to integrate with such a system. In this case, it makes sense to reduce the complexity of integration with the legacy system by placing it behind an intermediary facade that is convenient to integrate with, implementing the Strangler Pattern. This will also allow for a gradual decommissioning of the system while keeping the integration contract intact.

If the legacy system is extensively used by other parts of the larger system and requires ongoing integrations, it is advisable to modernize it with a current and trending architecture instead of simply applying bandages and guards.

Bad performance is a usual Achilles' heel of those legacy systems. What is the easiest and fastest way to address? Try to wrap it with an adapter with a load balancer, and run multiple instances of the system. If you are lucky (that system allows concurrency) you might fix the performance in no time! It might be pricy, though...

Continuous end to end testing. Do you have it? If you don't then you won't know when the sky has dropped. Automate schedule integration tests.

Consider middleware or an adapter layer when the legacy process is synchronous. Often legacy systems can be less responsive and create a poor user experience. An adapter layer can be used to abstract long processing times from the end user and create a more transparent interaction.

Replacing a legacy system is often necessary. One approach is to replace or upgrade the system gradually, one part at a time. This is important for several reasons, but the most important reason is to avoid disruption. Replacement should provide benefits, not problems.

To ensure a successful system replacement, which often requires a migration, the following factors can improve those chances of success: 1. Ensure the priority is set across the business for the replacement, and have it stack-ranked versus other published priorities. 2. Develop a realistic "time to adopt" and lower the bar required to adopt the service such that less senior individuals can do the work on behalf of teams. 3. Ensure that you disable the legacy onboarding path as soon as possible when the adoption of the new solution starts. Many migrations create additional concurrent (and unnecessary) technical debt as they move through the migration.

Versoning is very important if you want to have smooth system replacement. Try to reduce the risks of downtimes or weird behaviors is pretty important if you want that your company accept the idea of replace an existent system that already does what the enterprise is expecting should does and versoning may help to gradually replace the system and also could be used as AB testing and decide the right time for decommisioning gradually the system.

All the above "divide/concur/isolate/refactor" activities properly done would allow to replace that legacy piece by piece with manageable risks.

Rewriting the the system can be tempting but since you keep the original code legacy, it means you likely don't know what it does, so how do you actually write better system? And if you finally figure that part out from legacy system, is it actually legacy anymore?

Perception is often that an entire solution is a failure when reality is that a system usually has a handful of poor performing components that can be modified to restore the performance and improve scalability. Start with observability, identify the primary offenders, evaluate for restoration or replacement of those components, and execute. Very often, a few key improvements can right the ship rather than a complete refactor or replacement.

The legacy system is twofold - its code contains both the set of problems and their solutions. It should always be remembered that legacy systems are full of experiences from different engineers who have worked on them. Before starting to change the system, make sure you understand all three parts.

One thing to pay close attention to is the way that requirements changed over time for the existing system. Are they minor changes and the core purpose of the project is the same, or do the current requirements appear completely different from the initial ones? This can help inform your decision on how flexible you want to design your new architecture. Designing a perfect solution for the current requirements might not be as effective as designing a solution that is more adaptable to changes in requirements.

Understand if the legacy software really is a problem and why. Is the legacy code actually causing user errors or significantly holding the rest of the team back? Could you actually get more impact by maintaining the legacy system and applying all the refactoring effort to new features?

Balance the carrot and the stick. If a replacement system is identified (built or purchased), ensure significant value added or operational burden is reduced for customers. This will motivate clients to onboard and adopt.

Starting off tactically by eliminating the biggest sources of technical risk or operational overhead can buy time to come up with the a suitable long-term approach. Try to avoid overly criticizing the existing system, as it may have been the best possible solution at the time it was implemented.

Legacy systems often have limited budgets for maintenance, which leads to a “patch” mentality. This makes the system harder to maintain over time. Since re-architecture the system is very costly, it is crucial to be careful in selecting tasks for maintenance. Doing in depth research and providing statistics that show the “future cost” of the current change would help. I have observed a collective lack of long term interest and focus on solving today’s problem, result in cutting thorough investigation to save time (and cost). The overall quality then depends on a few “senior members” of the group.

Ok, this is a good articulation for what to do if you encounter a legacy system. The important point this article does not cover is the parallel efforts management when transforming a legacy system into a modern tech effective system. In a happy path, stake holders will agree to the improvments proposed, but if it is a critical business piece, unless the new system is thoroughly tested and approved by Stake holders, cannot replace the legacy one. So, it is very important to plan before hand the effort estimation of supporting the existing legacy system while it is undergoing the transformation. If it does not need parallel efforts, then decommissioning is the best solution.