Reliability be damned

Reliability be damned

Nature does not deal in certain, in fact she usually deals in chaos.

Everyone knew what was coming. It was one of those stifling, unsettled spring days.

A tornado would touch down somewhere today, no question.

 Near quitting time, the Wisconsin sky grew ugly. Swirling, dark purple clouds pulsed with ominous energy.

 The funnel touch down on the hydro plant roof, ripping the old slate off its steel supports. The ensuing drenching downpour, through the hole in the roof, was so heavy that the busbars in the 15 kV switchgear below exploded, literally blowing apart.

No alt text provided for this image

Fortunately, no person was exposed to either calamity.

The resulting electrical surge was so intense throughout the mill's electrical distribution system that none of the circuit protection could adequately respond. 

The stress cones on the step-down transformer to the local utility tie line failed, instantaneously. 

No lights, no radios, no power, no air, no water, no anything. 

Emergency conditions prevailed throughout the mill. 

  • Paper machines with crumpled wires. 
  • Chlorine fumes unvented. 
  • Smelt beds popping. 
  • Overflowing vats and tanks. 
  • Sewers backing up. 

The mill was out of control and nightfall was approaching.

Damage control/emergency action headquarters was established in the maintenance office where the radio base station and outside telephone tie was located.  

Emergency generators were set up to power the radio and phone system. A frantic search for additional generators throughout the immediate area was undertaken. 

Heads were counted.  

Buildings searched, top to bottom, until everyone was accounted for. 

No outside power was available, nor would be until the tie connection could be remade. 

The internal supply was not available until the buss' were replaced. 

Work teams were assigned.  

Some were sent home, to rest, in anticipation of an enduring and tiring shutdown.  

Work priorities were established. 

A schedule defined.

While the operating crews had their hands full, not much could be done without light, air and water. 

No clean up.  

Most were sent home. Some kept for fire watch. Some to help maintenance set up auxiliary lighting. 

The first two tasks of maintenance were to re-establish the tie line connection by replacing the stress cones on the tie transformer. 62.5 kV to 15 kV. The cones on the low end blew completely apart. 

One experienced electrician, who had made many cones in the past was sent to take on this assignment, with help from younger assistants, who were expected to observe and learn.

The second task involved a group of electricians beginning the job of tearing out the scorched and damaged switchgear to get at the buss' running in the rear of the switchgear cabinets. 

Several switchgear were severely damaged, but the mill and a sister mill upstream had some spares. 

Some equipment would have to wait for start-up until rebuilt switchgear could be acquired elsewhere. 

The busbars looked like thick metal plate that had been burned through by a blowtorch. Piles of now cooled, once molten slag hung from the ends and dripped into piles on the concrete floor.  

Splattered pieces of once molten metal could be found everywhere in the hydro plant. 

All through the night and the next day the crews worked, replaced by rested workers, who were, in turn, replaced again.  

Well into the first night, the buss was cleared and the mill ready for the 'juice to be turned on'.  

Work on the stress cones and clean-up of the line-up of switchgear was still in progress, the stress cones being the least critical of a critical situation.

In the meanwhile, it was expected that the hydros could be brought online as soon as the bank of switchgear was cleared for power. The mill had several hydro-generators and three steam turbine-generators. The mill was a net producer of excess electricity when under normal operation, thus the reason for the utility tie connection.  

As soon as the hydros would start-up they would provide enough electrical power to build steam and then the rest of the mill could start, regardless of the utility tie-line.  

Poor planned maintenance exposed itself when it was discovered that the station batteries, used for backup supply for excitation on the hydro generators, were no good. 

No alt text provided for this image

Furious, frustrated, finger-pointing eventually gave way to reason and clear thoughts. 

Forget the batteries, and why they don't work, what can be used to provide excitation?  

In a mill with nothing operating. No power. No supply. The worst possible scenario. 

"Wait, #4 Hydro is self-excited," the corporate electrical engineer remembered, "we'll just start that up." 

Man plans, nature laughs

The Number 4 Hydro, a self-excited unit, had lost its residual magnetism and if that unit was expected to bring up the rest of the units is was going to need an outside source to be 'flashed.' 

Chaos had met its match...a desire, an overwhelming, focused emphasis on finding a means to best the set circumstances that had created this will-sapping incident. In this case good luck and opportunistic thought created the way.

After explaining the need to the group of assembled mill and corporate personnel, the Chief Electrical Engineer began to think of all the traditional possibilities.

One electrician, having listened to his explanation asked if a portable power source would do the trick, he knew the mill had just purchased a trailer-mounted, gasoline powered arc-welder.  

It was, indeed, capable of providing the necessary power to flash the generator's field the engineer pointed out.

#4 hydro-generator came online, and then another, and another.   

Slowly, but progressively, as is the nature of a start-up, the mill returned to normal.  

Finally, even, the connections on the utility tie transformer were remade and everyone sighed a big relief.

A lot was learned from this failure.  

In some ways the mill was well prepared, in others, poorly. 

 If you had suggested that such circumstances could happen, prior to this incident, the general response around that mill would have been..."could never happen here."   

Nature never acknowledges that places exist where 's--t don't happen'. 

That mill’s responsiveness was, in a large part, due to its unique circumstances.  

It was in the heart of town, much like many older, northern mills. This simply means that any 'unusual' mill occurrence is felt by everyone in the immediate surroundings. 

The town's emergency services are basically made up of mill workers, thus immediate response of any emergency equipment (radios, lighting, fire protection, first-aid, and so forth) is available. 

The power company, to which the mill sold excess power, is a local cooperative and was especially responsive. 

The company's corporate headquarters is located just minutes away, providing additional resources, such as engineering expertise in project planning and management decisions, plus 'extra' human resources for job site supervision and to act as 'gofers'.  

Additionally, the 'sister' mill is also within a few minutes’ drive and that mill's resources were put on call for whatever might be necessary.

The sense of urgency that prevailed, shared by all concerned throughout the emergency, was a huge factor in the success of getting the mill up and running. 

Luck played a large part in no one being injured. 

At some point a preventive maintenance action is put in place (plan) for a good reason, over time more actions are added to the plan, but few are removed.

No alt text provided for this image

With time, demand for resources exceeds resource availability. Adjustment to any plan is required but too often overlooked.

The downside was the irreplaceable loss of production, equipment damage and the use of limited valuable resources correcting an unexpected act of nature. 

How is your emergency preparedness? Will a near catastrophic failure evaluate it or, does your plan and appropriate action cover the unexpected, the overlooked and shortcomings of your plans?  

Nature does not choose, nature does.


Joe Conyers

Territory Sales Manager at GRAPHALLOY?

5 年

When I was sailing, I used to say "Nature doesn't care how competent you are...."

Great story John!

回复

要查看或添加评论,请登录

John Yolton的更多文章

  • Preparing the Journey to Asset Management Improvement

    Preparing the Journey to Asset Management Improvement

    Most plants have several areas where opportunity exists to create better reliability at less cost Supply and demand out…

  • Wading for nightcrawlers

    Wading for nightcrawlers

    Often wondering what it would have been like to have grown up in a city rather than the small village, less than 900…

  • The Best just keep getting Better

    The Best just keep getting Better

    A while ago I wrote an article about the ‘Top 3%’ in the global Pulp & Paper (P&P) industry in terms of reliability…

  • Defining backlog control improvement

    Defining backlog control improvement

    Sam is thinking about his mill’s backlog and how to schedule the resources accordingly. ‘My planning and scheduling…

    2 条评论
  • The gofer

    The gofer

    Sam was basically rudderless when he arrived at the mill, straight from college, for his brand new assignment. He…

  • Contract negotiations and unintended consequences

    Contract negotiations and unintended consequences

    Sam was reviewing the contract negotiations newsletter (memo) sent to all supervisors within the company every evening…

    1 条评论
  • Reusing reconditioned bearings

    Reusing reconditioned bearings

    Sam was visiting a pulp mill during an annual outage as part of his responsibility to provide advice for improving…

    5 条评论
  • A mistake forgiven; a career path altered

    A mistake forgiven; a career path altered

    Sam was recently promoted to maintenance manager from assistant maintenance manager now with a long time maintenance…

    11 条评论
  • Preservation of Facilities

    Preservation of Facilities

    There is a lot of discussion, these days, about the condition of the ‘infrastructure’ within the USA. I can not help…

  • The case of the fiber choked oil reservoir

    The case of the fiber choked oil reservoir

    Sam was touring a tissue mill in Latin America, one that was in the process of installing a new tissue machine. He…

    4 条评论

社区洞察

其他会员也浏览了