Stripe, a payments processor startup, had an outage. An index was removed, the DB slows down, load builds up. From the discussion at HN: “an application developer didn’t add a new index description or remove an index description, but rather modified an existing index description. Our automated tooling erroneously handled this particular change and interpreted it not as a single intention but instead encoded it as two separate operations (an addition and a removal).”
Also from HN: “This is one of our few remaining unsharded databases (legacy problems…), so we can’t easily canary a fraction of serving capacity.”
And “There’s something that a lot of retailers don’t get - never put an obstacle in front of the customer giving you their money. Don’t let lines form at checkouts…”
And “This incident report reminded me of the ‘Game Day Exercise’ post from 2014: https://stripe.com/blog/game-day-exercises-at-stripe … So I am completely unsurprised that they are having this kind of problem. It kind of shocks me that anyone lets Stripe have anything to do with money.”