An oldie, but goodie, from the McNeely Tech archives:
It was a sickening moment.
“We should have listened to you, Mary Elizabeth,” the client said. He didn’t look well.
I had been pestering this client for quite a while to remedy their inadequate database disk mirroring and backup/recovery practices.
A few hours ago, his production database had been obliterated in a hardware failure. He tried but failed to recover any recent backup. Finally, he called to see if we could find any way to salvage recent transactions. Unfortunately, we couldn’t. The damage to the database was too extensive, and, additionally, there were no usable recent backups. The client hadn’t been recover-testing his backups, so he didn’t realize he had a problem until he tried to use them in an emergency. He had no other option than to recover an older backup.
The client was in the embarrassing position of having to ask partner organizations for re-sends of transactions made since the last good backup. Word eventually seeped out onto the street that a grave technical failure had occurred at his company.
It was no salve to my ego to have been proven right – yes, the client should have fixed their inadequate practices. But it was painful to me to even watch a situation like this unfold.
Please don’t break my heart by being the next one to call me with an unsalvageable loss. When was the last time you tested your backups?
It’s too bad that some people have to learn lessons the hard way. It’s so much better to be proactive and prevent problems.
This reminds me of my ex-boss telling me about a call he received when his client deleted the SYSTEM.DBF file because it was taking up too much space on the disk and then realizing that all the backup tapes are empty.