CRASHING THE SYSTEM

Too Long; Didn't Read

On January 15, 1990, AT&T's long-distance telephone switching system crashed. This was a strange, dire, huge event. Sixty thousand people lost their telephone service completely. During the nine long hours of frantic effort that it took to restore service, some seventy million telephone calls went uncompleted. Losses of service, known as "outages" in the telco trade, are a known and accepted hazard of the telephone business. Hurricanes hit, and phone cables get snapped by the thousands. Earthquakes wrench through buried fiber-optic lines. Switching stations catch fire and burn to the ground. These things do happen. There are contingency plans for them, and decades of experience in dealing with them. But the Crash of January 15 was unprecedented. It was unbelievably huge, and it occurred for no apparent physical reason. The crash started on a Monday afternoon in a single switching-station in Manhattan. But, unlike any merely physical damage, it spread and spread. Station after station across America collapsed in a chain reaction, until fully half of AT&T's network had gone haywire and the remaining half was hard-put to handle the overflow. Within nine hours, AT&T software engineers more or less understood what had caused the crash. Replicating the problem exactly, poring over software line by line, took them a couple of weeks. But because it was hard to understand technically, the full truth of the matter and its implications were not widely and thoroughly aired and explained. The root cause of the crash remained obscure, surrounded by rumor and fear. The crash was a grave corporate embarrassment. The "culprit" was a bug in AT&T's own software—not the sort of admission the telecommunications giant wanted to make, especially in the face of increasing competition. Still, the truth WAS told, in the baffling technical terms necessary to explain it.