So, at every point in this story, there is gross negligence and incompetence, be forewarned.
1 Like
And another call to test your restore procedures.
We all make mistakes, and we all know that, but we don’t seem to anticipate our own mistakes. We seem to proceed as if being careful means it won’t happen to us.
Not everyone who runs a web service is competent. Not everyone who is not competent to run a web service knows it. Not everyone who uses a web service for valuable data realises what they are doing.
GitLab only lost six hours, but presumably with a large active userbase. Losing six months, even if you’ve only got tens of active users, that’s tens of very unhappy people. Why don’t we see this coming?
Reading through the thread, it’s notable that this community of role-players already has a term: Total Party Kill. They should, in a sense, be emotionally prepared to lose characters, which is what has apparently happened. Most commentators are supportive, a few are angry.
Ouch. What a Profanity Removed. 
I wonder how many postmortems feature some variation of this: “The logs for each day show the backup starting and ending, with no exceptions or errors, all the way up to today.” The general idea of “backup seemed to work, restore didn’t” seems awfully familiar.