I deleted production on a Friday. Full confession, with timestamps

I am writing this on Sunday because on Friday and Saturday I could not type.

What Tavolo is: table bookings for 38 restaurants at the time.

What happened, Friday 14 June:

• 17:52. I want to clean old test bookings from the staging database before the weekend.
• 17:53. I have two terminal tabs open. They look the same. They are not the same.
• 17:54. I run the clean-up script. It finishes very fast. Staging has never been that fast.
• 17:55. A restaurant in Bologna calls my mobile. Their Friday night bookings are gone.
• 17:56 to 18:40. I do not remember.
• 18:41. I look for the backup. The backup job is a cron I wrote in February. It has been failing since March with a permissions error that it emails to an address I never set up.

What I lost: every booking made after 11 March, for 38 restaurants, on a Friday evening in June.

What saved me: the confirmation emails. Every booking sent one, and my email provider keeps 30 days of logs. I rebuilt four weeks of bookings from email logs, by hand and by script, until 4 in the morning. For the older ones I phoned every restaurant on Saturday and apologised in person. Thirty-eight calls.

What I lost in the end: two customers. Thirty-six stayed. Several said the phone call was why.

What I changed:

1. Production terminal has a red background now. Bright red.
2. Backups are tested by restoring them, every Monday, automatically, and it posts the result where I see it.
3. No clean-up scripts on Fridays. No anything on Fridays.

Please check your backups. Not that they exist. That they restore.

oh mate

i ran an uptime company for four years and the number of people who have a backup and have never restored one is, roughly, all of them. this post will do more good than my entire product did

You called all thirty-eight. That's why you kept thirty-six. Most people would have sent an email and lost half.

I lost a customer database in 2003 to a failed RAID controller. Tape backup was blank. I've tested restores monthly for twenty-one years since.

checked mine just now because of this. the backup existed. the restore failed on a missing extension. so. thanks

I am going to say the thing nobody else will: you ran a destructive script against a database at 17:53 on a Friday without confirming the hostname. The backup is the second failure. The first is process.

That said, this is the most honest post I have read on this forum and I have just checked my own backups.

You are right. Both failures were mine. The red terminal is for the first one.

Losing 2 of 38 after a total data loss is a 94.7% retention rate through the worst event your product can have. I would like to study you.

Dario I am SO sorry and also thank you!! I just found out my backups were going to a bucket I deleted in April!!!

Pinning a link to this in the welcome email. Thank you for writing it while it still hurt.

One week later and I count nine people in this thread and the chat who found a broken backup because of this post. That is nine companies. Dario, you have done more for small software this month by failing than most of us do by succeeding.

I would like that on a mug. Apparently mugs take a while here.

I am new and I have just read this. I understand the mug reference now and I am concerned.

Walt sent me here. I understand everything now. "Did you check the backups" is not a joke, it is a liturgy.

It is both. Welcome. Check your backups.