Plan Your Modernization Project In Reverse

In a modernization project utilizing a rewrite methodology, if you want safety and to avoid unnecessary cost overruns, here's how to do it: start planning your project at the back end, and work your way in reverse to the front end. The result may be a very different project design, but I guarantee that you will sleep comfortably at night.

Eliminate The "Big Bang!"

The last thing that you do in a project is to put your new system into production, so that's where we start. A "big bang" deployment is to be avoided if at all possible, since the "bang!" may be the project blowing up in your face. Standard SDLC processes say to test and test and test, and then put it into production (and cross your fingers and hope - no, that's actually not part of the SDLC, but should be).

However, there is a different way that completely eliminates the big bang: if the new application uses the same physical database or an equivalent data store that is kept in synchronization (more on how in a minute), then you can deploy your new software incrementally, as each sprint's work is ready. Modernization becomes more like maintenance.

Your project cannot fail because you always have a working system!

Query processes can use production data for testing without danger. Update processes can use the same testing process as current maintenance. Best of all, you can deploy the modules into production in parallel with the old application. Users can use either one, and drop back in an instant if any significant problems develop in the new one. The risk mitigation from this instant fallback capability is worth the price of admission by itself.

Solution 1 - Use The Legacy Database

Great idea, Don! But how are you going to pull this one off - connect the new application to the old database - really? Well, if the legacy system uses DB2 on an IBM mainframe or uses a relational database on another platform, then the new applications can directly connect to the database and use it as is, so this is actually the simplest way to do it and it works. (Data architects - please keep your hat on - I'll get to you in a minute - I promise!)

You can go to Oracle, SQL Server or your favorite database once you have decommissioned the old system.  As a bonus, if you have z/OS DB2, the cycle cost on the mainframe can be significantly reduced using a zIIP processor for its residual operational life.

Solution 2 - Convert The Legacy To Use A Relational Database

If the legacy application uses indexed files or a pre-relational database, it is likely to be possible to interpose an emulation layer between the program logic and a relational database to make this possible. Then you can use solution 1 as above.

If no such emulation exists off the shelf, it can be generated into any program as a preliminary step in the project using well known (and relatively inexpensive) code translation systems.  It may cost some more on the front end, but I guarantee you will get a hefty ROI on the back end, and be in production sooner to boot.

Solution 3 - Convert The Legacy To Use The New Database

This is the opposite of solution 1.  Here we stop using the legacy database entirely and move to using the target database from the legacy system. This is easier than it sounds, particularly if the application is on an IBM mainframe - the database can be moved to DB2 LUW, Oracle or your database of choice on the same physical hardware. Mainframe Linux has a very high speed, memory to memory transfer capability between z/OS applications and z/Linux applications. The application programs can be linked to the data via Hipersocket calls. There are variations on this strategy if the application is not on an IBM mainframe, and there are (rather large) upper limits on throughput for very I/O intense applications, but it's been done and it works for most applications.

Solution 4 - Re-Host The Legacy To The New Platform

This is not a very popular option, though it's relatively cheap and offers a sound ROI for many projects.  Key to using this solution to eliminate the big bang is that the re-hosted code must be translated to use the relational database upon which the rewritten application will be based.

Solution 5 - Data Synchronization

The most flexible solution is to maintain the old and the new data stores in real time synchronization until you are ready to decommission the legacy application. This is appealing if the legacy application does not use a relational database for its primary data store, and none of the other solutions above are applicable for one reason or another.

Uni-Directional Data Replication for Synchronization

Most people are familiar with uni-directional data replication from one database to another, often for use with reporting and/or data analytics. Every change is picked up by a utility or the database itself and propagated to the target. The target database is sync'd to the legacy database, with a latency typically around 1 second.

Bi-Directional Functional Synchronization

Uni-directional data replication stops working when the new, modernized application starts updating the new database.  Those updates have to be replicated back to the legacy. In effect, the old and new applications morph into a distributed database design which must allow for detection and resolution of incompatible simultaneous updates to both data stores. Sometimes, the rate at which such incompatible data collisions occur is so low that they can be handled manually - just so long as they are detected.

Data Synchronization Example

The most ambitious implementation of this concept was deployed for the project I worked on at the US Patent and Trademark Office, where we kept a legacy CODASYL database synchronized bi-directionally with Oracle. This approach used a dual strategy of uni-directional data replication at the database level and functional synchronization at the transactional level (sending equivalent transactions to both platforms). This dual strategy co-existed nicely with the deployment strategy, depending on whether or not an equivalent transaction had been deployed to determine which strategy to use.

Extension to Automated Testing

Functional synchronization is particularly valuable because it allows the results of the two transactions to be compared in real time and the source of any discrepancy identified simultaneously. Even after extensive testing, how do you find the one in a 1,000 or one in 10,000 discrepancies that indicates a subtle bug that can drive you crazy for years?  This is the only practical way to run an extended production parallel test - the USPTO plans to run it for 5 years! The reduction in testing costs can more than pay for the investment in data synchronization.

Data Modeling Issues

OK, now it's time to address the data architects' objections, which will center around their plans to completely revamp the data model and provide new data tables and elements to support planned new functionality. That's OK - you can still do that, in fact you must to be able to do it if you are going to achieve your business optimization objectives.

Key to all of these alternative solutions is having a relational database that presents a view of the data that is compatible with the legacy system while simultaneously offering the flexibility that building out the modernized application requires. This may require some minor technical compromises in development of the new data model, not necessarily but possibly.  If so, it becomes a business question whether to abandon the benefits of incremental deployment by insisting on technical purity, or accepting a few temporary compromises.

Production Parallel Testing

As discussed at greater length in my recent post on production parallel testing, this once universal process seems to have fallen by the wayside. Production parallel is the only way to test for defective or incomplete business rules associated with each requirement, since requirements based testing, by definition, cannot find defects in the requirements upon which the tests are based.

However, having a compatible - not an identical - data model is the only way that production parallel testing will work.  Otherwise, there is no way to test for equivalent results.  Production parallel is data driven testing, not requirements based testing, and is therefore dependent on being able to map from one data model to the other in order to compare intersecting data elements.

Other Constraints on the Project

Because we want to compare results of programmed business rules between two executing systems when running production parallel, obviously, the business rules that control the results of a transaction must be the same - after all, this is what we are testing!

At this point, people say, "hey, wait a minute - I want to improve the business rules in the new system, not keep them the same."  No problem - and here's why: optimizing a business process and the software supporting it requires changes to the query processing, not changes to the business rules controlling the updates.

Think about that - if you have an order entry system, even with an optimized process you still want to capture the same information to feed the order fulfillment processing.  If you have a debit transaction, you still want the account balance to be reduced by the amount of the debit.  Etc.  Maybe you will now have one message transaction whereas the old system used 2, 3 or more to get to the same result, but the new system must get to the equivalent (not identical) result.  You may have additional information being captured in the new system - that's OK too, since we only compare the information that is equivalent between the two systems.

This concept takes some getting used to, I'll admit, but the bottom line is that the constraint on the system regarding business rules only applies to the update transactions. You can do absolutely anything you want with queries, including setting up the user to execute an update transaction, because the queries do not change the state of the database.

It also requires a strict definition of what a "business rule" is.  We use the technical definition of a business rule from page 5 of the GUIDE Business Rules Project,

… a business rule expresses specific constraints on the creation, updating, and removal of persistent data in an information system.

There may be new conceptual business rules associated with new query transactions, along with new data elements, but the update business rules must transition smoothly from the old to the new system.  Otherwise, business operations will get different results from the new system which few sites can tolerate.

Summary

Key to this whole discussion is the realization that it can be very, very expensive to get your application to the point where you think it is almost ready to deploy, only to have it sit there for months (or years, in some cases) while new bugs surface and are fixed so you can deploy. Worse is to get it to that point and have it rejected by the users as unreliable, and the project cancelled.  Worst of all is to deploy it into production and discover that there are major problems that were not caught in testing - and you have no way to go back. (Google "Maine Medicaid Disaster" for a classic example.)

People who have been through a project which has failed are far more receptive to these ideas than those who have never had the dubious honor of navigating a major disaster. Of course, maybe you will be lucky.  Many projects do make it through after only a few months of pre-deployment schedule delays, and a few do sail right through into a successful deployment.  However, the bigger the project, the less likely this becomes. Above a certain size, serious problems are almost certain, and should be planned for.

The most important take-away message here is that if you design your project without a big bang deployment, so that you always have a running system, and can always fall back to the legacy if needed, then you cannot fail. Your project can still be late and overbudget, because you have underestimated the complexity of the task, but once you eliminate the big bang then even the possibility of failure has been eliminated.