Reliability be damned

Nature does not deal in certain, in fact she usually deals in chaos.

Everyone knew what was coming. It was one of those stifling, unsettled spring days.

A tornado would touch down somewhere today, no question.

 Near quitting time, the Wisconsin sky grew ugly. Swirling, dark purple clouds pulsed with ominous energy.

 The funnel touch down on the hydro plant roof, ripping the old slate off its steel supports. The ensuing drenching downpour, through the hole in the roof, was so heavy that the busbars in the 15 kV switchgear below exploded, literally blowing apart.

Fortunately, no person was exposed to either calamity.

The resulting electrical surge was so intense throughout the mill's electrical distribution system that none of the circuit protection could adequately respond. 

The stress cones on the step-down transformer to the local utility tie line failed, instantaneously. 

No lights, no radios, no power, no air, no water, no anything. 

Emergency conditions prevailed throughout the mill. 

• Paper machines with crumpled wires. 
• Chlorine fumes unvented. 
• Smelt beds popping. 
• Overflowing vats and tanks. 
• Sewers backing up. 

The mill was out of control and nightfall was approaching.

Damage control/emergency action headquarters was established in the maintenance office where the radio base station and outside telephone tie was located.  

Emergency generators were set up to power the radio and phone system. A frantic search for additional generators throughout the immediate area was undertaken. 

Heads were counted.  

Buildings searched, top to bottom, until everyone was accounted for. 

No outside power was available, nor would be until the tie connection could be remade. 

The internal supply was not available until the buss' were replaced. 

Work teams were assigned.  

Some were sent home, to rest, in anticipation of an enduring and tiring shutdown.  

Work priorities were established. 

A schedule defined.

While the operating crews had their hands full, not much could be done without light, air and water. 

No clean up.  

Most were sent home. Some kept for fire watch. Some to help maintenance set up auxiliary lighting. 

The first two tasks of maintenance were to re-establish the tie line connection by replacing the stress cones on the tie transformer. 62.5 kV to 15 kV. The cones on the low end blew completely apart. 

One experienced electrician, who had made many cones in the past was sent to take on this assignment, with help from younger assistants, who were expected to observe and learn.

The second task involved a group of electricians beginning the job of tearing out the scorched and damaged switchgear to get at the buss' running in the rear of the switchgear cabinets. 

Several switchgear were severely damaged, but the mill and a sister mill upstream had some spares. 

Some equipment would have to wait for start-up until rebuilt switchgear could be acquired elsewhere. 

The busbars looked like thick metal plate that had been burned through by a blowtorch. Piles of now cooled, once molten slag hung from the ends and dripped into piles on the concrete floor.  

Splattered pieces of once molten metal could be found everywhere in the hydro plant. 

All through the night and the next day the crews worked, replaced by rested workers, who were, in turn, replaced again.  

Well into the first night, the buss was cleared and the mill ready for the 'juice to be turned on'.  

Work on the stress cones and clean-up of the line-up of switchgear was still in progress, the stress cones being the least critical of a critical situation.

In the meanwhile, it was expected that the hydros could be brought online as soon as the bank of switchgear was cleared for power. The mill had several hydro-generators and three steam turbine-generators. The mill was a net producer of excess electricity when under normal operation, thus the reason for the utility tie connection.  

As soon as the hydros would start-up they would provide enough electrical power to build steam and then the rest of the mill could start, regardless of the utility tie-line.  

Poor planned maintenance exposed itself when it was discovered that the station batteries, used for backup supply for excitation on the hydro generators, were no good. 

Furious, frustrated, finger-pointing eventually gave way to reason and clear thoughts. 

Forget the batteries, and why they don't work, what can be used to provide excitation?  

In a mill with nothing operating. No power. No supply. The worst possible scenario. 

"Wait, #4 Hydro is self-excited," the corporate electrical engineer remembered, "we'll just start that up." 

Man plans, nature laughs

The Number 4 Hydro, a self-excited unit, had lost its residual magnetism and if that unit was expected to bring up the rest of the units is was going to need an outside source to be 'flashed.' 

Chaos had met its match...a desire, an overwhelming, focused emphasis on finding a means to best the set circumstances that had created this will-sapping incident. In this case good luck and opportunistic thought created the way.

After explaining the need to the group of assembled mill and corporate personnel, the Chief Electrical Engineer began to think of all the traditional possibilities.

One electrician, having listened to his explanation asked if a portable power source would do the trick, he knew the mill had just purchased a trailer-mounted, gasoline powered arc-welder.  

It was, indeed, capable of providing the necessary power to flash the generator's field the engineer pointed out.

#4 hydro-generator came online, and then another, and another.   

Slowly, but progressively, as is the nature of a start-up, the mill returned to normal.  

Finally, even, the connections on the utility tie transformer were remade and everyone sighed a big relief.

A lot was learned from this failure.  

In some ways the mill was well prepared, in others, poorly. 

 If you had suggested that such circumstances could happen, prior to this incident, the general response around that mill would have been..."could never happen here."   

Nature never acknowledges that places exist where 's--t don't happen'. 

That mill’s responsiveness was, in a large part, due to its unique circumstances.  

It was in the heart of town, much like many older, northern mills. This simply means that any 'unusual' mill occurrence is felt by everyone in the immediate surroundings. 

The town's emergency services are basically made up of mill workers, thus immediate response of any emergency equipment (radios, lighting, fire protection, first-aid, and so forth) is available. 

The power company, to which the mill sold excess power, is a local cooperative and was especially responsive. 

The company's corporate headquarters is located just minutes away, providing additional resources, such as engineering expertise in project planning and management decisions, plus 'extra' human resources for job site supervision and to act as 'gofers'.  

Additionally, the 'sister' mill is also within a few minutes’ drive and that mill's resources were put on call for whatever might be necessary.

The sense of urgency that prevailed, shared by all concerned throughout the emergency, was a huge factor in the success of getting the mill up and running. 

Luck played a large part in no one being injured. 

At some point a preventive maintenance action is put in place (plan) for a good reason, over time more actions are added to the plan, but few are removed.

With time, demand for resources exceeds resource availability. Adjustment to any plan is required but too often overlooked.

The downside was the irreplaceable loss of production, equipment damage and the use of limited valuable resources correcting an unexpected act of nature. 

How is your emergency preparedness? Will a near catastrophic failure evaluate it or, does your plan and appropriate action cover the unexpected, the overlooked and shortcomings of your plans?  

Nature does not choose, nature does.