Tips for Troubleshooting Pilot Plant and Laboratory Equipment

Troubleshooting is a tedious and time consuming operation. Often one hears that X is ‘a good troubleshooter” or has “good troubleshooting skills” almost as if these are innate qualities one must be born with. Nothing could be less true. Troubleshooting is a skill anyone can learn and improve at as long as they understand some simple basic principles. While troubleshooting skills are similar across many fields and activities, I am going to focus on those most useful in a research environment (laboratory to pilot plant to demonstration unit scales) since that is where my interests and experience reside.

·        Haste always makes a troubleshooter less effective. Taking some time to assess the situation before doing anything while everyone is agitated over the issue is far from easy. Operators want you to start doing something right now! A more effective approach is to take a moment before you touch anything to look carefully at the system. Note the position of all the key valves. Look at the pressures and temperatures. Note what is, and is not, operating. Often taking the time to do this will reveal the problem very quickly and obviously. The author once solved a problem defying the combined efforts of a much more senior group because he could not get close enough to the area to participate and, while waiting, noted that the feed valve had been inadvertently closed.

·        It may often be necessary to ask the operator to give you a moment to think. Alternately you may need to patiently wait for the operator to run through their explanation while only partially listening so you can have some time to think. One way I have found to address their concern and still function effectively is to think out load. This shows that you are working the issue while keeping them involved. This does not always work, as some operators insist on arguing or commenting on each point or trying to get you to stop and move in the direction they believe best. A polite, but pointed comment, that you are not as familiar with the problem as the operator and need a few moments to come up to speed often helps. (An occasional “Could you please stop and let me think for a minute” may, hopefully less often, be necessary!)

·        Write down what the exiting conditions and equipment status before you do anything including the positions of all the key valves. Update these as you make changes being sure they are clear enough and legible enough to understand what you did and when you did it. This may sound like wasteful added work, but after doing 5 or 6 things it is very easy to be uncertain if the last test was with the valve on manual or automatic or if the nitrogen was or was not flowing. Are we sure the pressure on the tank started at 10 psig? Was the system temperature constant during both tests? It is so easy to get confused and not know what exactly you have done without some attention to documenting each step. It is even easier to read 10 psig as 100 psig or fail to note the temperature is in F while the operator is giving you C. Jot down each step as you make changes. Make sure that you can (1) read what you wrote, (2) understand what that your abbreviation meant, and know that the top number is the pressure and the bottom the temperature. (In other words, take a moment to be neat, semi organized and careful to label things.) I can’t tell you how many times I have had to use this information to unravel a confused operator (or myself).

·        Treat everything the operator or person discovering the problem says with a very large grain of salt. People will leave out key items because they think they are unrelated, fail to tell you something they did out of embarrassment, be reluctant to admit they have no idea what the pressure gauge was reading or if the feed valve was on, or simply get confused and swear they put it on manual when, in fact, they didn’t. And occasionally, even in the best organizations, sometimes an operator senses they may have created the problem and do not want to give you all the background for fear of making their culpability apparent. After wasting an hour finding out the operator simply forgot to set something properly or did something in the wron sequence it is very hard not to let some annoyance show. But being open enough to ask without criticism often helps. “It’s like the cooling pump wasn’t on when you started. Is there any chance that that might have happened?” A reply of something vague like “I guess that is possible” often allows saving face and solving the problem.

·        Be suspicious of the person who has “solved” the problem and just wants you to fix something. The author has wasted much too much time getting a pump changed out, a controller pulled for repair, and a regulator replaced only to find the pump was not working because of an unsatisfied interlock, the controller because of an incorrectly set pressure balance, and the regulator because the gas cylinder was empty. I often try to address these “solutions” by a discussion about how the solver is positive that this is the problem. While often not received well, good operators rarely mind explaining their thought processes; weaker operators can often be helped to understand that perhaps they are not 100% certain and allows a more grounded discussion to commence. In either case, the time spent is usually well worth the time saved on an incorrect solution. Even experienced operators can be badly misled by the similarity with a past problem. The current erratic flow (±50%) that last time was due to a pump seal leak is often not recognized as being a lot different than the previous ±10%.

·        Train yourself to start from the beginning of a system or the end of a system and work your way to the other end. This is often felt to be counter intuitive as most people want to start in the middle and feel they can easily identify if the issue is behind or in front of their starting point. Sadly, they are wrong. Starting at one end or the other of the system avoids false moves and saves time in the end. Is the breaker on or does the equipment have power? Either is an unambiguous starting point for further checking. That JB3 is not energized may be much less so. And complex systems often have numerous paths to hold out a system making the ability to figure out which direction to go suspect at best.

·        Do not assume that the most complex or troublesome component is the culprit. We all tend to think that the flow has stopped because that (expletive deleted) valve has hung up again or that difficult to access filter clogged yet again. That may be the case but it is just as likely to be something entirely different like the feed valve being closed. Troublesome components exert a magnetic pull on troubleshooting efforts that often hinder a more logical, and ultimately faster approach. Learn to avoid the pull. (And yes, the author freely admits he often wasted 1 minute on a fast check of the pressure drop across the filter or a fast confirmation the valve is moving before doing much else. He also must admit it has often led him astray.)

·        Recognize that everyone has a “hunch” where the problem may reside. While these are sometimes correct based on past experience or simply a good intuitive sense, I have seen hours wasted checking an issue that a few moments analysis would show could not explain what you are seeing. Sometimes this becomes even more of a problem as if you don’t trust a component you tend to keep circling back to it, often needlessly. This is where the next step helps.

·        Develop a plan before you do anything. The author has seen too many things done to troubleshoot a problem which, when asked what this test will prove, is met with dead silence as people realize the proposed test will not prove anything. Taking the time to draw a simple flow chart and show where the results of each step will lead to is invaluable and well worth the time to develop. Often it will show that a certain path will show very little or that you need to do an additional step first to make sure that you can reach a valid conclusion later. And don’t be surprised when you get to a point and suddenly have to say, something like “and if the pressure rises …. then I have no idea what that means!” Sometimes you get lost in the process and forget what the step can or cannot show. Talking through the steps with a knowledgeable individual often helps you avoid faulty logic or skip over another, more viable, path.

·        For particularly troublesome problems, review what you have done and seen with someone. If no one is available, writing down a summary can sometimes work. Many times, the effort to organize your thoughts and explain the problem will help you solve the issue or at least identify the next steps. The person can often ask you a question or challenge an unfounded assumption.

·        When you have done everything you can think of and nothing has fixed the problem, resist the urge to go do it all over again. Instead take a step back and try and identify what you have not checked. While it is possible you checked something wrong, it is more likely you just are not looking in the right area. I often have found the problem by asking simply “what have ee not looked at so far?” The key you lost is always in the last place you look.

·        Doubt everything you are told no matter how much you trust and respect the operator. All too often the tank is empty, the cylinder not connected, the power not on, or some other incredibly obvious issue that someone failed to note. I have stood near too many valves I swear I opened and too many switches I swear I closed to not recognize how easy it is to get confused or fail to look closely enough at something.

·        Look carefully at the system and pay attention to any nagging points. Why do you keep looking at that pump? Why is your attention always drawn to that feed system? Often your subconscious is noting something is not right and if you look at it a bit longer you will realize the issue.

·        Never ever, do more than one thing at a time. Keep a close eye on everyone involved so that you are not starting something while some else has stopped it. One step and one step only at a time. Having someone double check that all 5 valves are all in the right position or double checking that you traced that feed line to its source correctly often saves incredible amounts of time and effort.

·        Make sure you understand the status of everything involved before the problem started to develop. The author once pushed an operator to remember if anything was different right before a heater stopped working. It took some effort for him to remember to mention the electrician who borrowed a ladder a few minutes before the problem; a fast trip confirmed that the electrician had locked out the wrong circuit by mistake and cut all the power to our panel.

·        Conversely, be willing to let go of something that you just cannot find any logical linkage to the problem. The lights may well have flickered in the control room just before the problem but if the DCS system is working properly it may well have nothing to do with the problem.

·        Always remember that the proposed issue cannot violate basic physics and chemistry. I do not care what logic path is involved, closing the valve does not increase the flow. Of course, you have to be very diligent in making sure you have all the right information. If the valve was reverse acting then perhaps you are actually opening and not closing the valve.

·        Never trust any drawings without verifying they are correct. Yes, I might not go down this path as the first thing I do but I always try to confirm how the equipment is piped and wired. It is too easy for a drawing to be wrong. And good MOC systems have reduced, but far from eliminated this problem. Operators who resist this step loudly arguing they know their units often have to be cajoled into letting you check anyway. My favorite tactic was to agree but note that unless I verified it I would always have a lingering doubt.

·        Always gather all available documentation on intermittent problems. Make sure all the observable facts are written down when the problem was discovered not afterwards. Doing it later makes it is too easy to make a mistake. Make sure the operator knows that if they forgot to note the pressure it is better to admit that then put in a “normal” value. Make sure the operator knows to stick to verifiable facts and not opinions. “The seal started leaking sometime just before the problem” is valid if the drip pan is always dry and the operator saw it a few hours before. If is not so valid if the drip pan is soaked and it just reflects the operator’s opinion that there is more than the usual liquid in it. The author once spent a week trying to find why a 30 psig rupture disk kept bursting when the system never had more than 5 psig on it. Only after reviewing the documentation that we finally forced ourselves to keep was it discovered that the disk burst whenever someone turned on the GC. However, ridiculous that sounded, it allowed us to find the incorrect line that that fed 60 psig of helium into our system.

·        Always ask the people involved about anything odd they notice, particularly the recurrence of anything when the problem appears. I finally was able to find why a gas monitor intermittently shut down a unit despite there being no releases when an operator casually noted that it always seemed to happen when another operator was cleaning his reactor. (The monitor was much more sensitive to the cleaning solution.)

·        When you appear to have fixed the problem, try and find a way to prove you are right. If you can go back and prove that when the regulator is set too low the problem appears and disappears when it is set properly you can be fairly sure you found the true culprit. But often ancillary things you do during troubleshooting fix the problem and you don’t realize it. The author once changed out a metering pump twice before he realized the issue was vapor build up and not the pump. Each time we changed out the pump we carefully bled the lines fixing the problem until it built up again. Obviously, this is not always practical but if the replacement component, for example, looks fine, a fast off line test may raise real suspicions that you have not found the problem and so may not have fixed it.

·        More is not always better. Involving numerous people in the troubleshooting process often simply slows it down (everyone needs to have their say), creates the potential for confusion (too many people doing different things at once), and rarely solves the problem faster. A better approach is usually to step aside and let someone else have a try when you appear to have hit a dead end. A fresh set of eyes or a new approach, without you breathing down their neck, often can solve the problem.

·        Stepping away from the problem for a time can sometimes also help. I am always amazed at how many problems are solved faster the next morning than the previous evening.

·        Problems sometime fix themselves. (They don’t but sometimes you fix them without realizing it.) Sometimes you don’t understand why doing X solves the issue. Both cases are rare and worthy of further investigation. Yes, I have walked away from some and still, to this day, share the stories in the hope someone smarter than me will tell me how I fixed it but these are rare. So be careful in assuming the problem has truly been fixed of gone away in these cases. Often, they are hiding a very subtle or intermittent problem.

And, even if you follow all these guidelines religiously, troubleshooting will take time and effort and often be incredibly challenging. However, I think you will find it will, on average, go a bit faster and easier.