Postmortems

Fire alarm destroys hard drives Not really a post-mortem. I know.

Fire alarm destroys hard drives

Not really a post-mortem. I know. But an interesting combination of unrelated things resulting in real problems.

“A loud sound emitted by a fire alarm system has destroyed the hard drives of a Swedish data center, downing Nasdaq operations across Northern Europe.”

[…]

"Because of the incident, Nasdaq Nordic could not start trading operations at 08:00, local time, when stock markets across Northern Europe were set to open. Affected markets include the ones in Sweden, Finland, Denmark, Iceland, and the three Baltic states. Norway’s Nasdaq operations were somehow not affected.

Nasdaq Nordic fixed the issue by 14:00, local times, when it brought online a backup system to facilitate stock and securities trading in the area."

A historical detail in another article on the same hazard:
“From their advent in the mid-late 1950’s, data centers have faced a universal threat: fire. One data center fire in particular sent a clear message to both the computer industry and the fire prevention community. On July 2, 1959, a fire started in the computer room operated by the United States Air Force.1 It was located inside the Pentagon building complex. The fire burned for more than five hours and caused an estimated $7 million in damages. Fire investigators attributed the fire’s ignition to an incandescent light bulb in a magnetic-tape storage area. The National Fire Protection Association (NFPA) responded by drafting the first edition of NFPA Standard 75, Fire Protection for Electronic Data Processing Equipment.”

That’s very interesting! I remember an old video from Sun showing a person yelling in front of hard drives and their monitoring dashboards showing an increase in errors/decrease in throughput, but I never knew that loud noises can actually destroy the drives.

“Don’t shout at your JBODs, they don’t like it!” https://www.youtube.com/watch?v=tDacjrSCeq4

Local news claimed what destroyed the drives was shock from halon discharge. Those nozzles are really loud: https://www.youtube.com/watch?v=Jx1s8u2Mpmo

@Juhani_Simola yes, the article has been updated to reflect that. Integration testing beats unit testing :wink:

Same thing happened at InterXion in Copenhagen about a decade ago, causing the loss of up to 50% of drives for a few companies I worked with at the time. They switched to less loud nozzles after that.

2 Likes