Postmortems

"Final Root Cause Analysis and Improvement Areas:

“Final Root Cause Analysis and Improvement Areas: Nov 18 Azure Storage Service Interruption”

“The engineer fixing the Azure Table storage performance issue believed that because the change had already been flighted on a portion of the production infrastructure for several weeks, enabling this across the infrastructure was low risk. Unfortunately, the configuration tooling did not have adequate enforcement of this policy of incrementally deploying the change across the infrastructure.”

Service Dashboard showing incorrect status because of faulty failover - failing over because of this very outage.