This ladies and gentlemen is why one has a robust backup routine!

For the techies out there. I just LOVE this commit from Tai, after our momentary server catastrophe on Monday night! ~Ben

Yeap, that’s probably good advice :smile:

1 Like

Still lolling at Tai’s subject line =Hugely underplaying the reality of team staying up till 3.30am restoring 63GB of sample data and website images!

The best part was we got it done while 90% of our customers were asleep.

When you want to kill your server 1am on Monday is a good time! :smiley:

PS The bug that caused the mass deletion had been there from 2013 when we first built the server.

PPS The actual bug. “Ooo, I want to delete something. DELETE… what? Nothing? OK let’s delete something random… OOo. I still feel like deleting something DELETE…” and repeat and repeat and repeat… till everything was GONE!

1 Like

Yeah, that’s a PITA - I learned the hard way when putting servers and networks together to check, double-check, and check again for those nasty little bugs, and after you put a few of them together you get to the point where you know you’ve got one still in there, so you keep on checking - and that’s why they pay me the big bucks!

And why I’ve gotten use to 36-48 hour days without sleep; only coffee, coffee and more coffee :smile:

1 Like