When you switch to your on-site generators, make sure that you really have switched to your generators. The physical internet in New York, from Art Deco to Brutalist. "This 550 foot slab-faced building is the AT&T Long Lines Building. … designed to be entirely self-sufficient and resistant to nuclear fallout for up to two weeks after a nearby nuclear blast. … Despite all the design and planning, it isn’t perfect. It completely failed on September 17, 1991.
It was a combination of management planning failure, equipment failure, and human error. Because of the self-sufficiency plan, AT&T had a load-shedding agreement with Consolidated Edison, the electrical power utility, where the facility would switch from utility power to on-site generators on request from ConEd.
This switchover was a routine operation which usually worked with no problem. But on this occasion, scheduled training got in the way of the standard procedure to check on all the equipment power supplies, known as the DC plants. One DC plant had not successfully switched to generator power, and it went on battery backup. That was not detected until it was too late to maintain uninterrupted power.
The result was that over 5 million calls failed. The Federal Aviation Administration’s private lines were disconnected, disrupting air traffic control to 398 airports serving most of the northeastern U.S.
When you switch to your on-site generators, make sure that you really have switched to your generators.
"