Postmortems

An update on Sunday’s service disruption | Google Cloud Blog

In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighboring regions, and it caused those regions to stop using more than half of their available network capacity. The network traffic to/from those regions then tried to fit into the remaining network capacity, but it did not. The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows, much as urgent packages may be couriered by bicycle through even the worst traffic jam.

3 Likes

Here’s a more detailed writeup:

https://status.cloud.google.com/incident/cloud-networking/19009

1 Like

Thanks! There’s a discussion on Hacker News here:
https://news.ycombinator.com/item?id=20120059