An oldie, but goodie – I haven’t trotted this story out in a few years, and it’s worth considering on a regular basis, so here we go again!
It was a sickening moment.
“We should have listened to you, Mary Elizabeth,” the client said. He didn’t look well.
I had been pestering this client for quite a while to remedy their inadequate database disk mirroring and backup/recovery practices.
A few hours ago, his production database had been obliterated in a hardware failure. He tried but failed to recover any recent backup. Finally, he called to see if we could find any way to salvage recent transactions. Unfortunately, we couldn’t. The damage to the database was too extensive, and, additionally, there were no usable recent backups. The client hadn’t been recover-testing his backups, so he didn’t realize he had a problem until he tried to use them in an emergency. He had no other option than to recover an older backup.
The client was in the embarrassing position of having to ask partner organizations for re-sends of transactions made since the last good backup. Word eventually seeped out onto the street that a grave technical failure had occurred at his company.
It was no salve to my ego to have been proven right – yes, the client should have fixed their inadequate practices. But it was painful to me to even watch a situation like this unfold.
Please don’t break my heart by being the next one to call me with an unsalvageable loss. When was the last time you tested your backups?