A year ago, Alaska’s Department of Revenue was preparing their 2006 effort to deliver the annual checks to Alaska’s residents of their share of the State’s oil revenues when a technician accidentally erased a disk drive containing applicant/member information on an account worth $38 billion.
This technician was at work reformatting a disk drive when he accidentally deleted and reformatted not only the wrong drive, but also the backup of the wrong drive. It’s one of those oops moments that we all have had, but one that had a major impact on the Department of Revenue.
Hope still remained for this still unnamed tech because the Department had backup tapes of all of its data and the now destroyed and irretrievable data was held on these backup tapes. One problem was the backup process and tapes had never been tested by anyone. The process was just assumed to work. So, when they turned to the backup tapes to recover the information, they were found to be unreadable and unusable. Not only this data, but also all of the backup data was found to be unusable. Despite the efforts of Microsoft, Dell and several other companies, it was impossible to retrieve usable data from either the reformatted hard drives or the backup tapes.
This is the definition of a disaster. At risk was the Department’s ability to abide by their legal obligations to manage the distribution of more than $6,000 in checks of which is one of the great perks of Alaskan living. With no electronic recourse, the Department had to return to the original applicant paperwork, stored in over 300 boxes. Each piece of paper had to be scanned, reviewed and manually linked to the electronic record that remained. It took almost six weeks of extra personnel working extra shifts and weekends to get the information re-entered. The cost was more than $200,000, which now the Department has to ask State Lawmakers to cover in a supplemental budget.
The Alaska Department of Revenue now has a viable backup solution for their systems. It is regularly tested and reviewed to ensure that this disaster is not repeated. They have taken steps, belatedly, to also make sure that they have processes in place to prevent new oops moments. They did this after they had the problem.
Are you going to wait until you get your own oops moment or are you going to invest money today to review your business’s ability to survive a disaster of any kind, natural or accidental?
To learn more about disaster recovery and business continuity, contact RJM Technology at 661-254-2017.