Never Be Sure The Computer Is Right

The preliminary report from the investigation of the crash of a brand-new Boeing 737-8 in Indonesia last year is out, and it seems like a computer error killed 189 people.

The trigger for the accident looks to be a faulty sensor, which told the computer the nose of the aircraft was too high. Pulling the nose of an aircraft up will eventually result in a stall, where the wings stop generating lift. To prevent that, the computer was programmed to push the nose down if it was too high.

Chilling data from the flight data recorder shows the fight between human and computer.

The orange line is the automatic trim, ordered by the computer. Each period below the line shows the computer pushing the nose of the aircraft down. The blue line shows the manual trim, ordered by the pilot. It is clear that after each time the computer trims the nose down, the pilot frantically trims it back up. Until the computer wins. And the plane crashes.

When you build systems – both administrative procedures and IT systems – you need to make sure there is an override. The world is complex and there will always be situations where your system will result in a stupid, costly or even deadly outcome.

On the small planes I fly, we deal with pitch trim runaway by pulling the circuit breaker that provides power to the trim engine. With no power, the trim motor can’t move the control surfaces and the pilot is back in control.

Do you have circuit breakers in place in your systems?

This post originally appeared in the Technology That Fits newsletter. Don’t miss the next one, sign up.