How do you handle mainframe troubleshooting?

Mainframes do a great job at providing refcodes when problems are encountered by the hardware or firmware. These refcodes generally end up in the hardware message log, called IQYY. These refcodes are a great source for understanding the problem.

One thing I found helpful was to understand the application which was build in Mainframe. This help us in understanding where the issue in the application flow and how easily we can fix it. Another helpful insight is to use different Mainframe tools like Omegamon , BMC DB2 tools.

Some of the things I would be considering when analyzing a problem that occurred in a Mainframe component/process is that to determine if its the Infrastructure, Code, Data or other. Any recent changes in the components that form a part of the process will take my priority to check if that has caused the abend. If its data, a hard stop like an abend is an easier way to at least find the root cause; if its the problem is due to bad data created by the process, then its going to be a challenge in identifying and resolving the root cause that caused the bad data.

The way I would approach analyzing a problem would be to segregate what caused the issue, whether it is caused by code , data or by system. If an application is support or maintenance type then most probably the issue could be caused by data. If an application is development based then it could either be data or due to erroneous code. Trying to find what exactly caused the issue can help fix the problem sooner.

A few control systems helped organizations I worked at in the past significantly reduce abnormal endings. 1. A robust and multi tiered, change control system can help reduce abnormal endings. Many years ago, a couple of organizations I worked for implemented a product called Endevor. It was a change control system that managed source code, compilation of source, and elevation of those objects through different phases of change: unit testing -> integrated systems testing -> Production. 2. Another product by CA was implemented that aided programmers test every branch of their code.

In the absence of refcode messages, data collection is key. Depending on where you are experiencing symptoms, a number of methods can be used to collect data. The data collect logs, usually reffered to as dumps, hold lots of valuable information of the problem. In certain cases the the data in the dumps may contain customer critical data, and visibility of whats inside these dumps are restricted.

One key input in analyzing a problem is to check the history. When a program/job had run fine yesterday but had failed today, then we need to check what changed between yesterday and today. Was there any new code change? Or are there any differences in the incoming data . Analyzing the history of job/program/data will help in pinpointing the root cause.

An example I have seen for batch are job abends. Few steps we generally use for analyzing batch abends are identifying the abend , checking job and system logs , review program logic, inspect input data.

Isolating the problem to a single component or to a smallest granular data will get you closer to the resolution. Exception handling or abend error codes in the logs provide insight into identifying the component. If its batch problem, job log or IBM Fault Analyzer can be great tools to use. Debugging the code helps in isolating the line of code that caused the failure.

If the problem is with your hardware or firmware, then the maintainer of your systems support need to be engaged to apply the fixes required. These fixes generally will be in the form of microcode fixes (MCFs). A collection of MCFs is called an microcode load (MCL), which is generally targets for a single firmware stream. If multiple MCLs are released, across different firmware streams, they are packaged and tested as a unit, called a Bundle.

One important step is to do testing the fix or change in the lower environments before moving it to production, that helps a lot in identifying any issues in the changes.

If simple JCL, correct it. See who made the last change and if you can not figure it out, give them a call. Yada Yada Yada Yadaàassssssssa

Before implementing the solution, its important to have it tested in a prod-like environment. The emphasis here is "prod-like", the data data and the code should be a clone of the production setup to effectively test the solution. When refreshing the DB2 tables, its imperative to consider the Referential constraints and working with a set of tables even though only one table refresh from prod is necessary. This will involve a lot of communicating with the team to make sure we are not interrupting any on-going testing activities.

From a test perspective, it is important to document what scenario in which the initial problem was found. If the scenario was reproducible, this increases the value of that procedure. From these procesdures we can build more and more complex test cases and test scenarios to ensure that we do not encounter similar issues we deploying new fixes or new code into production environments.

One thing I found helpful was to create a repository of all the issues occurred for an application and its remediation measures which fixed the problem.

Costumo dizer que nada é fácil no ambiente mainframe. Ent?o, aprender com a experiência é algo extremamente importante. Algumas vezes, encontra-se uma solu??o que n?o é facilmente visível nas documenta??es do sistema, e com isso, cabe ao administrador aprender com a experiência, para que, no futuro, possa adotar sua solu??o novamente.

One of the key steps that should be done after fixing the problem is to make sure if its an isolated incident or if there is a process that needs to be fixed, or its a training issue. Another approach that has helped me in the past, is to see for patters and see if a similar fix may be needed for other components that may have not failed yet as they would have run or encountered a similar situation. Identifying all those components and having them fixed should be the secondary task after the current problem has been resolved.

Mainframe environments are complex, and debugging issues to find the root cause may be challenging. Take this as an opportunity to learn the environment, learn how the pieces and parts interact, and most importantly begin to uncover the dependencies between these parts. Drawing an architectural diagram of your environment may help speed up the process of debugging future issues.

One thing I found helpful was don't lose the hope in the analysis of finding the issue ,it may takes time but being frustrated won't help ,but as mentioned a calm mind will help to figure it out easily.

Honestly I would say not different as on other platforms. Know what you have, what you expect and what you changed. Combine it with data you collect from the system itself and components interact with it.

Keep calm? I'd say this must be item #1 on this list, not #5! But, you know, try to move around a bit faster than usual while you're pretending to stay calm :). Mainframe problems can be expensive.

If a production system is down you have to do whatever is necessary to get it running again. Stripping the bad data and saving it somewhere is usually the easiest, but sometimes you will have to patch the system. Hopefully you will have a 2nd set of eyes to review any patches before you implement them. Once things are executing properly then you can go after the root cause (and having a good audit trail built into the system is priceless).

I think that there is also the 'culture'. As a very young sysprog I had some excellent colleagues and mentors who instilled in me - find the problem, fix it yourself if possible - if not call support (in my case IBM) - then more importantly - bend over backwards to make absolutely sure that this same problem will never happen again......when joining IBM I saw that the same applied in any mainframe error scenario - and it is still the case - at the first error assemble everything you can and work it from there. Culture !

Ideally, you would have fail safes in place. Workload management, ghost LPARs and maybe even failover CECS to first get away from the problem area and resolve the outage. Then, you can take the time to investigate and come up with a fix. Otherwise, like Russell said, band-aid that thing to get you running and start testing fixes to apply during a dark window.