Introduction
What is the most likely cause of losing a file? The first thing that springs to your mind is probably having a hard disk die. You've heard the stories: someone lost all the photos of his wedding; someone lost the book he was working on; someone lost all his school work or all their letters (i.e, email). Despite many hearing disk failure horror stories, hard drive failure is quite rare. I have never lost data due to the hardware on a disk failing, and I have approx 6TB of unique data on 20 disks (most are 1TB, i.e., 1000GB, drives). Despite having a perfect hardware record, I most certainly have lost data.
Accidentally deleting files is the most common cause of data loss. In my experience, hitting the delete key on files in drives with no recycler (e.g., mapped network drives) or on files "too big" to fit in the recycler, or using shift+delete is the most common way to loose data. "Edit Undo!" I scream in horror as an ISO image disappears, "Edit Undo!". But there is no "edit undo" action to be done: the undo button is coldly greyed out, leaving a feeling like the "insufficient funds" message that we all dread does.
Most backup schemes assume that all actions taken by the user are intended and that the only thing to defend against is hard disk failure, resulting in a lot of useless backup schemes including RAID and many online backup solutions.
Also despite my perfect hardware record, I am certain that I will lose data due to hardware failure some day. I am certain of this because I know that I will continue to use my hardware until it fails, and my guess is that you have the same intention. My main boot drive (i.e., the one with Windows XP on it) is about five years old and I have no plans to replace it: its doom is certain. Therefore, it is vital that I store no valuable data on any one drive; doing so will doom that data with certainty. Most people I know store all of their data on one hard drive, the one hard drive that came with their laptop (or, more infrequently now, their desktop). If you are one of those people, if you have never purchased a second hard drive for backup, know this: YOU WILL INEVITABLY LOSE ALL YOUR DATA (that is, all your pictures, all your documents, your financial databases, your code repository, all your music, all your videos, all your email (unless you use web-based email like Gmail), all your bookmarks, and everything else on your computer). Why? Because you are going to use your hard drive until it catastrophically, irrecoverably dies. We all do that. You don't get a warning indication when your drive is about to die. You don't change your hard drive once a year. You just use it until it dies. So do I. Why don't I lose data? Because I backup.
Unsafe places
In general, your data is not safe if it is in only one of the following locations:
- On a local hard drive. This is especially true if you are using a shared computer (e.g., a computer that is shared among different people in your family, your work place, or your school).
- On a RAID. RAIDs often fail above the disk level. When this happens, data redundancy at the disk level is less than useless.
- On a USB flash drive. USB drives "go bad" often if you unplug them without safely removing them, and sometimes they refuse to safely remove, so you have to pull them out or reboot.
- On a network drive with no backup. When you click "delete" on something on a network drive, the file is gone. It is NOT in a recycle bin. It's just gone.
- In a location that is managed by a program that you haven't tested. Programs have bugs and often decide to delete all your data. I realize that at some point, all data is managed by a program, but I have had bad experiences with, for example, media managers reorganizing my music and then deleting songs it thought I didn't want. I did want. My backup was safe though.
In general, you should force yourself to believe that your data does not exist if it exists in only one of these locations. If you give in to the temptation to think that a file that is saved somewhere (especially in one of the above locations) is safe, you forfeit the right to complain someday that your files are missing. Of course they are missing; they never existed. If you resist the temptation to believe that files that exist somewhere are safe, you are on the road to never losing data again.
Online Backup Services
Several services backup your data to cloud stores (i.e., data stores hosted somewhere on the internet). I dislike these in general because I feel a loss of control (a bad feeling for something that I wish to be my last line of defence. I also dislike them because most internet connections are not fast enough to backup significant amounts (100GB+) of data quickly. Even after the initial backup (when your files have to be transferred for the first time), the bandwidth required is quite extensive. Download an MP3 to listen to? Now you have to upload it to your backup service too. What would it be like to restore an operating system from such a service? Not nice. You would have to have a partition image to do such a feat, and I am not aware of any service that offers a partition level online backup.
Does your backup solution permit that much data to be stored? Is there a premium to having more than, say, 100GB stored? Imagine restoring 500GB of files over the internet. That would easily take a month of bandwidth at DSL speed. Does your ISP allow you 500GB/month? Would you want a clogged connection for a month? Do you need your files back faster than that? Who else has access to your files when they are in the cloud data store? These are important questions to answer.
That said, Cryptonite is probably your best option if you choose an online backup solution ($5/month, unlimited storage).
If you want something more flexible, check out Jungle Disk which provides a mapped network drive for $4/month + $0.15/GB stored (and free upload/download).
Use a "no-delete" policy
Not following this step is what makes most peoples backup schemes useless. Here is the scenario: Joe is merrily replicating his files every night to a USB disk. Once a month he replicates to one of two disks: at least one disk is always at home so he won't lose everything if he does a backup when the building decides to burn down. Brilliant, Joe! Joe then accidentally deletes his photo collection. No problem: he has a backup. Problem: he didn't notice that he deleted it. Two months of replications later (after both his monthly backups have "rolled over"), Joe has lost his own photos without drive failure; he lost his files because of a poor backup policy.
The correct policy when backing up is not a "replication." Replication is not backup. During the backup process, if a file is absent on the source drive but present on the destination drive, the file on the destination drive should not be deleted (this is called a "no-delete policy"). If your backup software does not have a "no-delete" option, stop using it (you are just wasting your time). Beware of online backup services that perform replication and do not guard against the most common form of data loss: accidental file deletion. Understand that unless you can prove that your chosen software or service is using a "no-delete" policy, you do not have a proven backup system.
Untested backups are NOT backups
I once was hired as an IT consultant for a company which claimed to do a complete backup to an internet data store every night. Sounds great, right? The idea that one's data is stored safely in some secure data center is sure to bring warm fuzzies. The problem with that companies backup was that, as I discovered from reading the logs, it hadn't run for over a year. Not only that but IT staff had changed so often that the login password to the data store was lost. And not only that, the size of the data that needed to be backed up had exceeded the purchased storage size on the remote server. And not only all of the above, the amount of time that would be required to do a complete backup was longer than 24 hours due to the increase in storage. Their untested backup was no backup at all.
A proper test of a backup scheme is to mock a disaster scenario: attempt to restore to a working state from your backups. Remove your primary hard drives and see how much data you can restore and how long it takes you.
Store your backups on another computer or an unplugged drive
A common joke in the IT world is the "backup on the same drive"; a hapless computer user attempts to get around data loss by copying his files to multiple places on one hard disk (when the drive goes, so do all of the files). In my opinion, another nearly useless method is to backup files to more than one drive when the source and destination drives are always connected to the same computer. Operating system goes nuts one day while your little brother was playing a "free download" game? There goes your backup. Store your backups on another computer to avoid this.
My recommendation is to use a server to store your document files and map the network drive to access files locally. This makes backups easy: get the server to copy your files from the disk (to which you have mapped a connection) to another disk that can only be accessed by the Administrator group. Using this method prevents even someone with physical access to your desktop from being able to erase everything. I have my files replicate in the middle of the night, every night.
Never put all your data in one place at the same time
Don't put all your backups in the same room that you keep your source files. If you are backing up a computer at work, do not bring all your backup drives to work on the same day; the building could burn down that day and then what good will your backups be? Also, do not travel with all your data at the same time. Doing this would force you to have all your data in the same place at the same time.
What qualifies as a backup?
A backup is something that prevents data loss (i.e., reduces the probability of it to an acceptable level (L) over a given time period, usually assumed to be L < 0.5 one's life span or ~100 years (T)). Data loss can only be prevented by reducing the likelihood that all assessable copies of a piece data are simultaneously lost. The probability that all copies of assessable a piece of data will be lost (P) is equal to:
P = 1 - ((1-A) * (1-D))^T (Oh, yikes! Subtraction, multiplication, even exponentiation! Calm down and think it through. This equation matters to everyone with important data.)
Where
A = the probability that access will be simultaneously lost,
D = the probability that the data will be simultaneously lost
T = time periods (assume 100 yearly time periods)
For example, if you thought you had a 5% chance of losing your access (A = 0.05) (e.g., your proprietary RAID card) in a year and a 0.01% chance of losing all your data (D=0.0001) (e.g., the disks in your RAID) in a year, your chance of losing all your data in a 100 year period would be:
P = 1 - ((1-0.05) * (1-0.0001))^100 = 0.994 or roughly 100%. Say bye-bye to your data! We need to change these numbers so that P < 0.5.
The probability of losing access and losing data is governed by the same formula: the chance of failure to the power of the number of redundancies of that point of failure. For example, if you have two drives in a RAID 1 with a 1% per year failure rate per disk per year, the chance of the RAID failing is 0.01^2 = 0.0001.
As these examples show, many RAID configurations are NOT backups because the raid controller is a single point of failure. Standard RAID 1 (pure disk mirror), however, is a part of a backup because the RAID controller is not required to access the drive. However, a backup solution must enable you to have P < 0.5 and RAID doesn't do this alone. You will need to have multiple RAID controllers or use a backup system that does not rely on a single point of failure. For example, copying your data to non-RAIDed disks accomplishes this: you can plug a non-RAIDed disk into any computer; you don't need a special, proprietary RAID card. In addition, a corollary to "use a no-delete policy" is that RAID is not a backup even if you have spare RAID controllers. RAID is redundancy and won't protect you the slightest from accidental file deletion (the most common source of data loss) or file corruption (if it happened during a normal file access). RAID will redundantly delete and corrupt all copies of your files on the RAIDed disks.
Keep in mind that A above should be calculated to include the chance that your location will suffer a disaster (think volcano/meteor impact). It doesn't matter how many copies of your data you have in your house if your house burns down. In order to get P < 0.5, you will need multiple locations.
With all of these considerations in mind, I offer the following system as an example of how to get P < 0.5 by placing a copy of your data in the following places:
- Your local disk in your home
- An external, offline disk at work
- A backed-up server such as JungleDisk
At least four disks (probably more) or three locations would have to be simultaneously destroyed for this to fail. P ~= 0.01. If you do this right, use a no-delete policy, and test it, you won't lose your data for 100+ years 99 times out of 100.
Backing up is a trade-off
The act of backing up one's work on the computer should be viewed as the same kind of trade-off that Schneier talks about in Beyond Fear (and in various blog entries on his site): a trade-off between convenience and security. It takes some effort to make a backup: there is the planning, purchasing of additional equipment, and executing of the plan. If the total cost of losing everything, e.g., catastrophic hard drive failure, was $10,000 and the chance per year of such a failure occurring was 10%, then the maximum one should spend per year on mitigation and prevention should not exceed $10,000*10% = $1,000 (assuming that such action would make the chance of loss zero). If $1,000 were spent to this effect, there would be absolutely no net gain or loss: over 10 years, for example, $10,000 would be lost whether such a backup plan was implemented or not. Therefore, spending more than $1000 per year would be foolhardy; one would be better off simply letting his hard drive die than spending that amount of money. Consider the case when the value of stored data exceeded $10,000. If it were, say, $20,000, spending $1,000 per year for reducing the chance of loss to zero would effectively save $10,000 over the same 10 year period.
With both of these cases in mind then, one can see that it is vital to consider the total cost of catastrophic drive failure. Thankfully, backup plans are far cheaper than $1000 per year. When amortized over a 5 year period, the equipment should cost a family less than $100 per year (possibly as low as $25 per year). Even with such a low cost of backing up, it is still important to consider the cost of losing your data. If your data is not worth anything to anyone then don't back it up.
A simple backup plan for a casual user
I recommend the purchase of the following equipment (this is Q4, 2009; double storage capacity for each six quarters elapsed):
(If 1TB is not enough, simply buy multiple pairs of hard drives and enclosures)
Then follow these steps:
- Take the hard drive, install it in the enclosure (see instructions from the enclosure), and plug it into your computer.
- Format the drive (use NTFS).
- Create a new account on your computer called "backup" (with a password) which can read all of the needed backup files but never needs to hold any of them open (for example, backing up your Outlook's .pst file will fail if you have Outlook open. A separate account is best to mitigate this risk).
- Logon to the backup account.
- Download and install Karen's replicator on the backup account.
- Create a new backup job in Karen's replicator (some of the following instructions are in HSL):
- Karen's Replicator[open program]{PROGRAM}>Edit Settings{BUTTON}>New Job...{BUTTON}
- Enter the appropriate source and destination folders (likely Source: C:\Documents and Settings Destination: F:\docs_and_settings_backup)
- Disable the schedule (uncheck the "Enabled" box).
- Leave the other default checkboxes the way they are:
- "Include Sub-Folders" is checked
- "Copy Files Only if Changed or Added" is checked
- "Copy Only if Source is Newer" is checked
- "Compare File Sizes" is checked
- "Replicate Folder and File Deletions" is unchecked
- "Delete Old Copy Before Creating New" is unchecked
- Save Job{BUTTON}>Save Changes and Close{BUTTON}
- Run the job by selecting the job in the main window of Karen's Replicator. The first run will take the longest; subsequent runs will not copy unchanged files.
- After the backup is complete, remove your drive and store it in a watertight container (like a Pelican case) in a different building than where your computer is (to prevent fire from destroying all copies simultaneously).
- Repeat the backup procedure every month.
The above procedure does not solve all issues:
- What if your house burns down as you are backing up? You lose all your data.
- What if your hard drive dies the day before you perform your next backup? You loose a month's worth of data).
For a more comprehensive backup strategy, see the next section, "A simple backup plan for a small company."
A simple backup plan for a small company
Please first read the above section: "A simple backup plan for a casual user" before reading this section as many of the instructions are detailed in that section and will not be repeated in this section for brevity's sake.
I recommend the purchase of the following equipment to have two copies of backup data (this mitigates loss of all data if a catastrophe ensues during the backup procedure, when a backup copy and source data is in the same location at the same time) (this is Q4, 2009; double storage capacity for each six quarters elapsed):
- Purchase at least two 1TB hard drives
- Purchase at least two USB enclosures (I recommend getting a USB+eSATA drive if you have eSATA available)
(If 1TB is not enough, simply buy multiple pairs of hard drives and enclosures. For example, if you have 2.5TB of data to backup, buy six 1TB hard drives: 3TB for each copy the two copies of data.)
In the end, you will have three copies of your data: one copy will be what you compute with (your C:\ drive for example) and the others will alternate between your nightly and monthly backups:
- Plug both of your USB drives into your computer and run the same backup twice: once to each of them (see the above section: "A simple backup plan for a casual user" for details).
- Unplug one of these drives and mark it "Odd" or "Even" depending on the month you are in ("Odd" for January, March, etc. and "Even" for February, April, etc.) and store it in a separate building from where your source data is.
- Leave the other drive plugged in and enable a nightly schedule with Karen's replicator (set the backup to start at, say, 0200 hours).
- On the first of every month, switch drives (being careful never to put all three drives in the same location at the same time).
The ultimate backup: exponential decay
I plan on writing a program that backs up on what I call an exponential decay schedule. This program would create an initial backup and then a backup every night (or however long the user decides) and delete old backups to match the exponential decay curve: there would be one order of magnitude of more backups for every magnitude of time one moves toward the present from the initial backup. For example, if I were using base two and my first backup was in January of this year, I would have the following backups if it were now December:
- January
- June
- September
- November
- December
This makes the backups look like an exponential decay (notice how the backups get continually closer together as time progresses toward the present):
- January (backup)
- February
- March
- April
- May
- June (backup)
- July
- August
- September (backup)
- October
- November (backup)
- December (backup)
I am hoping to have an addon to my directory management software (not written yet either) finished by Q1 2012.
Further Reading
Home | Up | Copyleft | Contact
the probability that access will be simultaneously lost (A) multiplied by the probability that the data will be simultaneously lost (D):
Comments (0)
You don't have permission to comment on this page.