Comments on Mainstreaming the Mainframe

This is an extensive comment on Mainstreaming the Mainframe from the LinkedIn Mainframe Experts Network group.

Technical debt is a direct result of underinvestment in maintaining software applications, so that short-term fixes accumulate over time resulting in fragile applications.  Fix one thing and three things break is a common experience with aging legacy applications (a problem not restricted to mainframe applications).  However, this is a double edged sword: given free reign, there is a legitimate concern that the result would be gold plated applications that have been over maintained. Striking the right balance is both difficult and rarely done.

Because of the high cost of the mainframe, there is a natural tendency to make applications as efficient as possible.  However, there is an inverse relationship between efficiency and flexibility.  The most efficient applications are the most difficult to change, and vice versa.

There is a natural dichotomy between what are now being called "systems of engagement" and "systems of record". The front-end systems of engagement can adopt the "minimum viable product" concept for the rapid evolution of an application.  Conversely, doing the same thing for systems of record - our back-end, often mainframe based applications - could lead to organizational chaos as business rules are imperfectly implemented and erroneously used to update data.

Unfortunately, the front-end groups and the back-end groups seem to speak different languages, and, worse, seem unaware of the reality of their respectively different world-views. To an extent, this is a result of the relationship between efficiency and flexibility - the hyper-efficient back-end applications, particularly those on mainframes and burdened with high technical debt, are very inflexible. Conversely, the front-end applications, freed from the burden of having to have the results of processing be as correct as humanly possible, can be as flexible as their programmers can devise.

None of this has anything to do with the mainframe as a platform, other than the historical burden of applications with a long lifetime of accumulating technical debt and the use of COBOL, PL/1 and (in more cases than most people would ever believe) mainframe assembler for core applications. However, millennial attitudes to the contrary, Java is not that much more efficient than COBOL.  Yes, it is more efficient, but we are talking about roughly a factor of 2-3, not a factor of 10-20 or more. I safely predict that when core applications that have been rewritten in Java are as old as the COBOL applications the replaced, they will have accumulated similar burdens of technical debt. "Java is the next COBOL" is how I describe this prediction.

Technically, COBOL and Java are both "imperative" languages, meaning the order of the statements matter. This is opposed to "declarative" languages, in which the order of statements does not matter. This seemingly academic differentiation has substantial business implications. As a corollary, declarative languages do not have the problem of side effects, the "fix one thing and three things break" phenomenon. This leads directly to a dramatic reduction in the sheer quantity of code, in some cases I have seen by as much as three orders of magnitude. This can result in a similarly dramatic rise in flexibility, for both front-end and back-end systems. In other words, focusing on Java vs. COBOL misses the essence of the problem.

Unfortunately, declarative languages are almost unknown in IT applications. The only one commonly known is SQL, and it has a very restricted domain - accessing and updating a relational database. One can loosely describe the use of decision tables in certain business rule management systems as meeting the definition, but while familiar to a larger number of technicians the use of BRMS' is far from universal.

The other major distinction between front-end and back-end systems is complexity resulting from the sheer number of business rules that control the updates to the persistent data store (using the definition from the Business Rules Group).  Front-end applications, while legitimately complex by some measures, tend to be very light on business rules by our narrow definition. Conversely, the back-end applications do the heavy lifting in the business rules arena. Managing this complexity is a daunting task indeed, though it is rarely conceptualized as managing complexity. 

Managing such complexity using an imperative language is a Sisyphean task. I recently reviewed an application with 3,500 business rules - which required 3 million lines of COBOL to implement and 12,000 hours a year to maintain. Worse, maintenance changes took up to three months because the technicians had to test rigorously to ensure against unexpected side effects.  Front-end systems almost never have a burden like this.

With this as background, I think that this article from Compuware, despite some very good points, also misses on a few points. Especially on the "need for speed" - the agility of the applications must be tempered with the reality of the differences in the nature of front-end and back-end applications. Although the complexity and degree of technical debt varies hugely between different applications and different industries, attempting to apply standards from front-end development to back-end applications will not work unless one understands the different nature of the respective business requirements. I listened to an argument from a proponent of the "minimum viable product" point of view expressing his concept that we can adopt the same approach for replacing a mainframe applications because "if it breaks we'll just fix it". This is a very bright person whom I otherwise respected very highly, but he seemed incapable of appreciating the cost of an error in a back-end system, particularly an error that may not come to light for a very long period of time, years in the case in question.

I very much agree with the comments from Daniel Raisch, that what we need is BusDev not DevOps. This is not to denigrate the value of efficient development and deployment systems, but the DevOps concept by itself is not sufficient to meet the goals of the business. It is too IT focused, too concerned with doing things right rather than doing the right things.

By taking a BusDev approach, it is possible, though hardly trivial, to move IT in the direction that the business requires, but doing so, in turn, has its own requirements:

1. Primum non nocere, or "first, do no harm." Don't break the applications that the organization requires to conduct its business operations. The degree of paranoia required for this can be far, far greater than ever imagined by those who grew up on front-end, minimum viable product application development. We have developed a (patent pending) process to ensure this, but using it is a lot of work that people don't want to do at the start of a project (if at all). The business usually has to force IT to adopt the degree of paranoia appropriate to their organization, because technicians resist doing the software archeology required to ensure safety due to boredom with the process. I'm sympathetic - it is boring - but it is also necessary.
2. Recognize the difference in front-end versus back-end requirements for your particular organization and industry.  How agile do you need to be in the front-end versus the back-end? What is the potential cost of an error in front-end processing versus the cost in back-end processing? How much risk with the application functionality driving daily operations are you prepared to take? (The last should be a board level question, IMHO.) These are the questions that should drive any attempt to fix the problems discussed in this article.
3. Recognize the difference between efficiency and flexibility. Don't expect a flexible application to be anywhere near as efficient as an inflexible application.  Be prepared for the cost increases that accompany accomplishing true flexibility in your back-end operations. You don't have to abandon the mainframe to do this, and there are good arguments why you should keep your mainframe, but as a consequence you have to expect to see many newer technologies in your mainframe environment. Don't reflexively say "we can't do that, it's too expensive" - figure out what it will cost, contrast that with the value to the organization, and present your arguments to management. After all, it is management's decision at the end of the day. A true BusDev digitalization initiative is going to cost money, a lot of money, but the ROI can be there. I'm aware of a $700 million project right now that is expected to save $650 million a year - but most savings will be outside of IT.  If your focus is entirely or mostly inside of IT you'll miss such opportunities.
4. Start counting the business rules that you have to manage in your front-end and back-end applications, and be prepared to invest in an appropriate technology to manage them, whether that be a business rule management system, a declarative language, a NOSQL database, or whatever.

I take issue with the assertion that "re-platforming of that application logic has proven to be highly impractical." I have many counter-examples to prove this wrong where it was in fact highly practical, as well as examples where sites failed to gain the anticipated ROI or where the projects failed outright. It can definitely be done, but I submit not only that this point is wrong, but that this is the wrong point: re-platforming is an IT focused solution to a business problem. The platform is not the issue nor will re-platforming solve the business issues in most cases - the business requirements should be the issue, and the only issue. I use re-platforming as a tactic, not as a goal, and sometimes that "re-platforming" project means going from z/OS to z/OS, or to a combination of z/OS and z/Linux.

Achieving the DevOps goal in this article requires addressing technical debt, and in a big way.  The implementational complexity of an aging application is likely to be much greater than the essential complexity of the problem the applications was built to solve in the first place. You can gain a great deal of flexibility by retiring that technical debt, but you can never reduce the implementational complexity to less than the essential complexity of the problem, by definition.

This goes back to the problem of the different levels of complexity and risk between front-end and back-end systems. Agile development processes alone will not ameliorate an inflexible mainframe application that is inflexible because of high levels of tuning. It can reduce implementational complexity, provided the agile process is focused specifically on reducing technical debt and not on changing functionality.  It costs real money to retire technical debt, and you don't gain any new functionality by doing so - but you do begin to gain flexibility. Expectations should be set correctly.