You're facing a sudden power outage in a critical facility. How do you effectively troubleshoot the issue?

Hope for the best, but prepare for the worst! I personally have seen several cases where we thought we had everything reserved, two power inputs from different transformers, diesel generator, on particularly dangerous mechanisms (eg. lift drive an oxygen lance for a oxygen-converter steelmaking) gas engines (not electric). But there is always a chance that something will go wrong, and you are lucky to see three mistakes in one place, mouse on the switchgear makes a short circuit, diesel generator does not start (and the backup input from the GRID/network is also not available at this moment), Of course you have batteries at the substation DC110/220V, but what's the point if you don't have power supply?

Start with an initial assessment to identify the extent and cause of the outage. Check the main power source and circuit breakers to ensure they are functioning correctly. Inspect any visible signs of damage or malfunction in the electrical system. Communicate with facility staff to gather information on when the outage occurred and any recent changes that might have impacted the system.

When facing a sudden power outage in a critical facility, the initial investment in backup systems like Uninterruptible Power Supplies (UPS) and generators can make all the difference. These systems provide immediate power to essential operations, allowing for a smooth transition while the root cause of the outage is identified. Ensuring these backups are in place and regularly maintained is crucial, as they safeguard against downtime and potential damage to sensitive equipment, ultimately preserving both operational continuity and safety.

I will fire on the spot the one who was responsible for proposing alternative sources of power (i.e. fuel generator, solar with batteries, etc.), to mitigate the risk of a critical facility being without power. Then I will call a company to provide me technical support, and if possible also to generate power for me.

First of all, and before you touch anything, be sure that the problem is within the facility and not a general blackout of the entire city. City lights are not always the best, indications, as it is possible to operate with separate cables/source/phase/other. Then find out if the entire facility is out of power. If yes, move ASAP to the main fusebox. If not, go to where you have power and move to the area without power. First thing check for people locked in elevators. Then check if there are coolers/refrigerated rooms, etc., and inform all to not allow cold air to escape, so as to avoid opening the doors, etc. If you find that the reason is a fire. Remember: PROHIBITED water.

Backup power is crucial in managing sudden outages, especially in critical facilities. Installing uninterruptible power supplies (UPS) and generators can provide immediate electricity when the main power fails. Regularly testing these systems ensures they function correctly in emergencies. Additionally, integrating automatic transfer switches (ATS) can seamlessly transition from primary to backup power without manual intervention, minimizing downtime. Properly maintained backup systems are a vital safety net, ensuring that critical operations continue uninterrupted during power outages.

Every critical facility has always alternative solutions in place (risk continuity plan). This includes alternative sources, fuses, cables, plugs, and more. Lights and computers are all in a UPS (to a battery system that will initiate automatically, keeping power for some time. Also, the internet can be via mobile telephony, turnstiles, electromagnetic locks, active barriers, bollards, and road blockers, which should operate via generator and UPS. Not that easy to make a scenario without details. Without knowing why is critical the infrastructure? For example, if the power failure is in the metro, during a terror attack? What can supply fresh air to the victims? If we have a blackout at a nuclear power plant?

Es fundamental no solo verificar que los sistemas de energía de respaldo, como los SAI o generadores, se activen correctamente, sino también realizar pruebas regulares para prevenir fallos inesperados. Identificar y resolver problemas como baterías descargadas o fallos en interruptores de transferencia antes de que se necesiten es crucial para evitar interrupciones en instalaciones críticas. Un mantenimiento proactivo asegura que estos sistemas estén listos para proporcionar energía temporal y mitigar el impacto mientras se resuelve el problema principal.

A detailed inspection is crucial for thoroughly understanding the root cause of a power outage in a critical facility. Begin by systematically examining key components such as transformers, circuit breakers, and switchgear. Check for any signs of wear, damage, or overheating that could have led to the failure. Use diagnostic tools like thermal imaging and multimeters to detect hidden issues, such as faulty wiring or insulation breakdown. A meticulous inspection not only identifies the immediate problem but also uncovers potential vulnerabilities, allowing you to address them before they escalate into future disruptions.

We can always use special instruments to locate the problem. A Megger, for instance, can help us find insulation problems that can lead to cuts in the power cables. Also, we should calculate the ampers/consumption of every line/fuse and check if the cables, harnesses, junction boxes, etc., can carry such load, or sooner or later we will have a fire. A portable thermal camera (Flir) can also be a critical instrument, especially before a power failure.

Repair and testing are essential steps in resolving and preventing future power outages in critical facilities. After switching to backup power, immediately assess the affected systems to identify and repair the fault, whether it’s a faulty circuit breaker, a damaged cable, or a malfunctioning transformer. Thoroughly test the repairs to confirm the issue is resolved and that all systems are operating normally. Regularly scheduled maintenance tests, including load testing on backup generators and UPS systems, can also preemptively catch potential issues before they result in outages, ensuring reliability and continuous operation.

In most of the cases in electric damages, we need a replacement. Even a cable or a switch that was overheated, even if it's not melted, and it didn't take a fire yet, can do so sooner or later. Use always all your senses to find damage. Especially your nose can be a critical instrument of fire detection. If there is a very bad smell in the area, do not relax until you find out why.

Preventive measures are essential in minimizing the risk of sudden power outages in critical facilities. Regular maintenance schedules, thorough inspections, and testing of electrical systems help identify potential issues before they lead to failures. Implementing surge protection, proper grounding, and redundancy in power supplies are additional steps that bolster system reliability. By proactively addressing vulnerabilities, you can significantly reduce the likelihood of unexpected disruptions, ensuring that the facility remains operational even during unforeseen events.