DP Incidents Jan/24

Introduction: It’s Wednesday and you are due a DP article. I was going to write about something else, but as usual, I was mesmerized by the useful lessons that come from DP incidents. The first interesting one comes from an oil company alert, and IMCA has put out both a DP Event Bulletin & the 2023 DP Incident Report.

Keep It Simple, Stupid: If you have been reading my articles for a while, then you know that I am suspicious of the use of “error proof” equipment to bypass making safe configurations. That’s what happens when you know how the equipment has to work, have seen lots of similar “high reliability” designs go wrong over the years, and wish people could keep it simple and get it right. In DP, we prefer not to put all our eggs in one basket, because active elements fail and we can’t have single point failures. Anything complex that works can stop working or work wrongly. Thus, we would rather split our power supplies, the DP has an IJS backup, and we object to any black box system that can kill DP. It’s against the rules, we’ve been there, and we’ve read the incident reports. Sometimes, the makers of such systems pretend their system is error free and have powerful forces behind them. One example was the main discussion during the monthly incident review at the School for ROCK.

Large Oil Company Alert: I’ve already discussed emergency shutdown systems a couple of times, and this alert was about how failure of a single “bulletproof” ESD input module caused a blackout. It was a well-known safety system from a well-known, trusted, and popular manufacturer, whose documentation might well have expressed the chance of failure of that module in terms of whether two could occur before the heat death of the universe (maybe we should start worrying about this global warming after all). The bad design put all the ESD activation lines into a single module, instead of splitting them or having two module verification. When the failure occurred, one and then multiple safety monitored inputs fluctuated on, as they had been activated. If you are from the old days, when we used to hunt down electronic faults rather than replace black box modules, then you have some ideas of how this this could happen, and know why DP people dislike these designs. The top level ESD shutdown is supposed to reliably cause blackout when required, and that is their major safety concern. Not causing a blackout is a secondary concern that must not interfere with the primary concern.

DP vs. ESD: The ESD need for assured blackout and the DP requirement for no active fault causing a blackout leads to conflict. The ESD designers prefer to solve the problem by claiming they have high reliability SIL 2 or 3 systems. That reduces the chance but doesn’t eliminate it. The use of SIL gets the designer free, but doesn’t solve the problem if the SIL equipment isn’t properly used or maintained. The MODU Code was updated to eliminate this design shortcut and force a split. The DP guidelines recommend using multiple modules to avoid a single point failure (e.g. MTS DP Design Guidelines 20.5). That’s how it should go and is the best practice, but if you are not building a MODU or are on an older vessel, then it probably comes down to probabilities.

What’s Enforceable? In DP2, the vital ESD triggers and trips should be split according to the redundancy concept. E.g. two protected ESD0 pushbuttons (system A & B) in a manned space rather than one, or one or two out in the rain. DP engineers can lose this fight in DP2 non-MODUs, because the ESD manufacturer has “documented” reliability and DP2 has probability limits (less likely than loss of the space to fire or flood), but should still strive for good design. In DP3, there is no excuse, as the DP3 reliability requirements are higher than the chance of the manufacturer doing the risk calculation right. Old vessel’s might be grandfathered, but should fix the system when they can. New DP3 vessels and DP2 MODUs have to split the system.

Hurray! DPE! IMCA put out DP Event Bulletin 04/23 and it can be found here. These are always worth reading and digesting. I’ve gone long on the first subject, so I will provide a brief introduction to each:

1. PRS Recalibration - It is important to be able to identify PRS weaknesses. Deselecting and reselecting can cause more problems than it clears. I think we had a recent previous event bulletin on this subject, so watch out.
2. Follow The ASOG - An example of a vessel starting to make safe when they should, but being tempted to trouble-shoot the problem before they were outside the 500m zone. Don’t get rushed and make a failure worse.
3. Repeated Drive Faults - Intermittent faults are a warning of coming trouble. In this case, repeated VSD trips and reset led to a partial blackout. Note how the drive fault took out the DG. Drive protections aren’t bulletproof, breaker coordination is normally based on simple faults like shorts or grounds, and coordination is moot without regular calibration, maintenance, and testing. They were operating open bus, so they only lost one side.
4. Split Isn’t Always Best - A vessel was running with minimum DGs in closed bus mode when it was accidently switched to split mode, but the uneven load distribution and time to start standby DGs led to a position deviation. DPOs need to beware of button-itis or menu-it is. Don’t do things automatically.
5. Sea Water Cooling Drill – A recommended drill. It is important to practice and ensure faults can be reliably detected and corrected in time to support redundancy. Worth investing the time to think about.

2023: IMCA has also put out their 2023 DP Incident Report. It is also for free and can be obtained by ignoring the G-Pay button and hitting the document button and providing an email address. Unlike the previous years, there is no separate document with more information on each incident.

Conclusion: Making mistakes is an effective way to learn. I hope some of this has be useful. Go read the DP Event Bulletin if you haven’t.