Intermittent Faults & Alarms Are Important Warnings

Introduction:?I thought I had written my last article in response to the recent rash of unnecessary DP incidents, but Mat Bateman flagged that the MTS DP Committee has released a draft guideline for comment to deal with the same issue.?They start from a different position and take a different approach.?Most of what they say is good, but they want to make it better.?It is a draft document, so it is worth collecting your own thoughts and providing your own feedback to MTS to improve the document.?Let’s have a look.

Personal Thoughts:?First, I need to summarize the difference in position.?I looked at the DP incidents and identified three main causes.?The first article dealt with the failure to make safe after a fault or to perform maintenance in the safest mode.?It reminded operators that Murphy’s Law and worst case failure (WCF) were friendly reminders and useful safety tools, rather than obstacles created by enemies.?As many of the DP failures were the result of troubleshooting and maintenance on DP3 vessels in closed bus mode, the second article reminded people that there was no such thing as redundant closed bus tie operation in DP3 and it should not be used.?The belief that they had bulletproof protections caused people to operate unsafely, so that needed corrected.?Finally, the third article looked at the underlying cause of these problems – the myth that redundant closed bus tie operation is more efficient.?It isn’t, but the belief that it is causes lots of problems and wastes lots of money.

MTS Position:?Some of the prime movers in the MTS DP Committee have been championing achieving equivalently safe DP3 closed bus tie operation.?They are aware of the problems, champion research to close the gaps, and removed the primary barrier to it when IMO645 was updated to IMO1580.?They know redundant closed bus DP3 operation is a future goal rather than current reality, but hope to achieve it.?They have already released good, free guidelines that address redundancy and open or closed bus ties (available at dynamic-positioning.com).?While my response to the DP incidents was aimed at making safe after a fault, split bus DP3, and not being mesmerized by closed bus, they aimed at safe handling of intermittent faults.

Importance:?At first, I thought safe handling of intermittent faults was common sense that was easily resolved by applying Murphy’s Law and WCF.?It is simple for veteran captains, engineers, and managers, who have seen or heard the worst, to apply these, but it is less simple for newer people under pressure to immediately solve a problem and improve performance.?I assumed experience and judgement, but neither are a given and intermittent faults can be less black and white.?MTS are right that this is an important sub-issue with a number of related DP incidents.?It is worth looking at and improving understanding and practice.

Intermittent:?The primary characteristic of intermittent faults is their variation in time.?They can be mild or severe, common or uncommon, and regular or unpredictable.?To use concrete examples, a car with a fault that only resets its clock or radio is not safety critical, but one that locks the brakes or prevents their use while driving is very critical.?A regular fault is expected at certain times or conditions, like needing to blow up a tire with a slow leak or avoiding wet weather due to an unlocated insulation fault.?An unpredictable fault is triggered by a rare and unknown failure, like suddenly losing a tire or burning out a headlight.?Safely driving the car partially depends on the severity, frequency, and predictability of the possible faults.

Warning:?Some things are so well understood and automatic that they require little attention, such as regularly filling up with gas, but other problems can be more insidious.?Maybe that harmless clock or radio fault is caused by a growing electrical fault that will eventually affect major systems??If the clock goes, then the clock & stereo, and then the clock, stereo, & dome light, then there is clearly a theme, but the real world of complex systems isn’t always that clear.?Intermittent faults are warnings that we need to pay attention too.?They tell us that there is a risk.?We need to pay attention and understand that risk if we are to control it.?They need recorded, tracked, and investigated, rather than ignored.?Soft brakes, weird noises, and electrical faults are a warning in a car.?Increased frequency and/or severity of faults usually means increased risk, but sometimes their disappearance means that the problem has gotten worse rather than better.?Understanding is key, but no one has time to catch and understand everything, so caution is advisable.?I’ve used cars as an accessible example, but the same principles apply to DP vessels and their critical operations.

Dangers:?The main dangers of intermittent faults are that they are harder to investigate and take seriously once they have disappeared, and that they can be far worse than a permanent fault in an active control environment.?It’s much easier to find a solid ground than a partial, intermittent, DC (but not AC) ground in an active control system based on AC power, but both can cause critical malfunctions.?Some intermittent faults are essentially hidden faults that occasionally manifest.?This provides an opportunity to define an otherwise hidden risk, but troubleshooting resources are limited and troubleshooting presents its own risks.?Once a fault has occurred, the vessel can no longer pretend to be healthy and must be recognized to be in a degraded state, until the problem is definitively cleared.?Alarms can’t just be cleared and equipment reset, the problem needs to be considered.?Based on limited information, the risks associated need to be realistically evaluated (Is WCF + intermittent fault + realistic hidden faults acceptable?) and appropriate protections put in place, such as opening bus ties, running other equipment instead, or limiting operating conditions.?This should include means of reducing failure effect and spread, whether the problem is hunted down or just monitored.?Finally, a permanent problem only provides one change that the system has to adapt to, while an intermittent problem can provide two or many.?An actively changing fault in an active control system can provide much worse results than a single fault due to the interaction between the two.?Some FMEA producers are not aware of this and it invalidates a lot of approved systems.?E.g. an intermittent fuel blockage can be far worse than a solid blockage.?An intermittent fault that occurs at a critical control frequency usually invalidates control.

Examples:?I’ve been trying not to give too many examples but the MTS document provides several.?When we look at them, we see that they don’t always involve intermittent faults:

1. Poor network design left all forward thrusters vulnerable to a common network fault.?When the network degraded, the thrusters were repeatedly lost and needed repeatedly reselected by the DPOs over 90s.?IMCAM259 and basic principles tell us that this is a design problem manifesting itself in a relatively benign failure mode.
2. A vessel, with a known thruster control network problem and a weak tripping thruster, lost main propulsion when the network fault caused a false runaway indication, the crew stopped one main thruster, and the heavily loaded remaining main thruster tripped on overcurrent.?Two known intermittent problems combined.?Failure to manage the obvious risk that the remaining weak thruster wasn’t reliable after a failure of the other redundancy group.
3. Closed bus DP3 had intermittent unexplained ground fault alarms for weeks, but didn’t open their bus ties or otherwise make safe.?Led to a blackout.?Failure to manage risk.
4. The perennial classic: poorly maintained cooling systems discovered as inadequate in hot conditions.?This led to multiple equipment alarms and then failures.?The real problem was inadequate maintenance and failure to make safe before the system failed.
5. Thrusters and power management system dependent on dual common network.?Network degrades causing intermittent alarms, but vessel does not make safe.?Network fails.?A common design problem and failure to make safe.
6. Until the controller was replaced, a control fault caused an azimuth thruster to intermittently turn to a wrong direction until reset.?Choosing to operate with a known problem.
7. Open bus tie DP3 with unknown AVR faults that cause occasional trips and hidden incorrect network shield grounding.?Eventually, the two combine and blackout the vessel.?Open bus tie but the hidden grounding cross-connection created a fault path.

Some of these were design or maintenance problems, but others are examples of failing to take faults seriously, think through their implications, and operate more safely.?One example showed a vessel operating in a supposed safe mode and it being defeated by a hidden fault.?There are some other examples here.

SPF??The MTS guidance rightly points out that any ship will have multiple known problems at any time, but it is vital to maintain critical redundancy functions.?It notes the difficulty in catching critical alarms in the typical bulk alarm system, and recommends DP FMEAs identify critical alarms and behaviors to aid operators in decision making.?They recommend paying attention and evaluating intermittent faults as they appear.?For example, good data helps, but while hard resets sometimes solve problems, they also sometimes destroy data [not to mention making smart devices dumb].?I’m not sure that their ideas of detailed data collection, in depth testing, and responsive OEM interface can be used by most vessels, but they can be performed in the high value, high risk portion of the DP market.?Even in the lower end of the market, operators need to pay attention to intermittent failures to avoid expensive incidents.

Conclusion:?Although I initially dismissed the document as complicating common sense on my first skim through, after writing my own simple thoughts on the subject and comparing it with theirs, we generally agree.?My quibbles are minor.?Their technical guidance document won’t solve the psychological and market problems created by the issues addressed in my second and third articles, but it is a good elaboration on a problematic subset of failures to make safe after a fault.?We all need more common sense, and the occasional reminder of it.?Especially, when we are tempted to believe that a problem just went away.

Recommended Homework:?Most of this was my own summary of the problem and solutions.?It only made direct reference to the new MTS guidance in a couple paragraphs.?The MTS document provides more information and more detailed advice.?Read through preliminary TechOp D-12 and provide MTS feedback before Apr 23/23.?I’m uncertain of where that feedback should be sent.?Perhaps someone could clarify in the comments.