Storage Migration

Get out before it's too late

All media - or at least all the electronic media we have now - will fail eventually. The objective is to get your data onto new media before the old system fails with your images no longer accessible. How do you know when that's about to happen? Well, you don't exactly. But you can get some good clues by practicing the periodic data validation described on this website.  If you keep good backups, and if you monitor the general health of your storage, you should catch problems as they start, rather than after you've been burned.
Read more about data validation in the validation section

Moving on to something better

For several decades, Moore's law has continued to predict the development of new storage technologies. This rule of thumb says that capacity and speed will double roughly every 18 months. In many cases, this will make upgrading an attractive proposition long before the current storage technology stops working.

If you have four 500 GB external drives sitting on your desk, taking up space and generating heat, consolidating them into a single faster 2 TB drive will be reward enough to justify the time and expense of an upgrade. Remember that each individual drive should have its own data validation overhead. Migrating to larger drives can save you time since you have to validate fewer of them.

Figure 1 An archive is easier to manage if you have fewer drives of larger capacity. You'll need to migrate periodically as larger drives become available.

Migration steps

Any storage migration should follow some basic steps in order to achieve a safe and orderly transition to new media. Here is a basic outline.

Gather primary data

When migrating a primary archive, you'll want to be sure you have accounted for everything before you start. Spend some time pulling everything together to make sure you don't leave something out. Remember, if it's confusing at the time of migration, it's only going to get worse after the migration has happened. Now you'll have even more copies of the data to sort out.

Validate

Data migration is a great time to do some basic data validation. You don't want to migrate corrupted data without knowing it. If you are going to be replacing backup storage as well as primary storage, it's even more important. It would be a shame to migrate the primary copy of the data, and replace backup copies, only to find out that the primary version was corrupted. We suggest you review validation processes as you prepare for the transfer.
Read more in the data validation section

Figure 2 Migration transfer folders are named with the source, the destination and the date of transfer. This helps you keep track of what happened.

Validated transfer

Once you have the material gathered, and validated to the best degree possible, you'll want to make a validated transfer from the old to new drive. Make sure to set it up to transfer with bit-for-bit compare. Instead of using the Outgoing and Incoming folders as outlined on the Validated Transfer page, we suggest you make Migration Transfer folders. These folders are named with the source, the destination, and the date of transfer. This way, you can tell what files were migrated, and when.

Temporary drive storage

After you've made the transfer, we suggest you pull the old drive from service for a while. Label the drive, and put it somewhere safe. You won't want to do anything with the old drive until you are satisfied that the new drive is working just fine.

We also suggest you label the drive, so you can tell what's on it without having to plug it back in. The label shows the old name, the contents, where it was sent to, the date of migration, and some date in the future when the drive can be repurposed.

Figure 4 After you have migrated data off a drive, put it away for a while before you repurpose it or get rid of it. We suggest labeling it and storing it in a good container, like this one from Wiebetech.