Works in Progress

Introduction

Works-in-progress or Working is the lifecycle phase between Ingestion and Archive. It’s the phase of life where the work cannot be easily automated. Because the files are undergoing change, it’s much more challenging to protect them thoroughly. One important objective of your workflow should be to pass files steadily through the working phase and off to the archive.

Separating these three phases - Ingestion, Working and Archive - is one of the most effective tools we have in creating workflow that adds speed, adds security and reduces cost.

How this page is structured

This page starts with a discussion of what a work-in-progress is, and how to understand this from a file-based perspective. This includes an outline of different workflows and how that informs the concept of a work-in-progress. This is followed by an outline of typical Working tasks, and finally a discussion of backup scenarios for Working files.

Between Ingestion...

The Ingestion lifecycle phase is marked by the download of the images but, more importantly, by the set of automated and batch-applied tasks that can be accomplished. Once you get to the point where individual evaluation must be done to the images, you have entered the Working phase. These tasks include tagging subsets of images, rating for quality, and adjusting the rendering settings. This might also include deleting outtakes, final renaming, adding individual keywords, and preparing the images for some kind of Publication.

....And Archive

The Archive phase, as we will see on a later page, is a “permanent” home for the images. Ideally, the files can be put away into the archive in a folder structure that is stable and scalable, with permanent file names attached. Files really only need to be in the Working phase until they are ready to put away.

Note that, as we said on the Lifecycle Overview page, often the original file can be put away before the finished image is created. This is because the work we do to raw images never changes the original image data. This means that you might be able to put entire shoots into the Archive, even though you will continue to rework the images with Lightroom or Aperture.

Preparing files for archive

Ultimately, the Working phase is all about preparing the files for archive. Your image files need to be treated as works in progress - on a file level - only as long as it takes to get them ready to be put away.

The shortest path to the archive

For some users, the files can go straight into the Archive during Ingestion. This would be easily possible for someone who:

• uses Lightroom or Aperture to adjust their files
• always works with their computer connected to their photo archive drive(s)
 
• does a simple sync for their backup processes

For a user like this, there’s not much reason to create a Working phase to their file handling. The faster the images can get into the archive and get the full archive backup, the better. Note that the Lightroom or Aperture catalog itself, however, would be a continual work-in-progress and should get the backup treatment of a Working file.

A slightly longer road

Many users will not be able to send the files directly to the Archive during Ingestion. In this case, let’s look at someone who keeps current Working files on a fast internal drive, and stores the Archive on some kind of external device. It makes sense for this photographer to do some work to the files before Archiving. The most important changes to make to your files before Archiving are:

• any work to rename files
• final folder arrangements
• any deletions you want to make

These are the most common file-based tasks that are necessary before Archiving the images. Any deleting, renaming or refoldering that is done after the files are Archived makes your backup and restoration process more complicated and less secure.

While you are doing the tasks outlined above, it can also make sense to do the following tasks before Archiving, but these are less necessary for someone using Lightroom or Aperture to manage a photo collection. These tasks are broken out separately because they don’t require the alteration of the files themselves, if you are working with catalog-based PIEware.

• Additional grouping of the files
• Rating for quality
• Image adjustment
• Assigning custom keywords and other custom metadata
• Proofing multi-capture works like time-lapse, HDR and Panorama sequences

Full raw image prep before Archiving 

There are also some users who will want to take images even farther before Archiving. This includes people who use browser-based images editors like Bridge and Camera Raw, Capture One, or Nikon Capture NX. Since the work you do to organize and adjust your images lives side-by-side with the file, you’ll generally want to complete this work before Archiving. 

Even some users of Lightroom or Aperture may want to “finish” their files before Archiving. Sometimes this is because it’s faster and easier to work on files that are stored on their working drive(s), providing a workflow advantage. And sometimes it’s just the way they like to do things. These people will probably need to do all of the individual tasks outlined in the section above. It might also include creation of master derivative files. 

Note that workflows that require full image prep prior to Archiving can create very large storage requirements for Working files. Given the added expense and difficulty of fully protecting Working files, this could be a significant workflow impediment.

File-based “hard” changes

Before you put your images away in their permanent home, it’s important to make certain “hard” changes to the files. If you ever need to restore your archive from backup, the process will be much easier if the folder and file name structure is an exact match. If the filenames and folder structure don’t match, it can be impossible for the catalog or non-linear editing software to properly relink to the files.  

Therefore, these hard changes must be a part of the Working file preparation.

Can’t backup software keep primary and backup in sync?

It is possible to set up backup software to keep the primary copy of the files in sync with any backup copies. This seems to remove the need to do the hard changes prior to archiving. But there’s a big problem here.

If you set your backup software to always copy changes in the primary copy to all backup copies, you’re removing a lot of the protection that a backup is supposed to provide. If you accidentally delete a file or make some unwanted change to a file, this backup arrangement will copy those unwanted changes to the backup. It’s much safer to set up your software so that backup versions are updated very rarely.

File naming

In general, we prefer the assignment of permanent unique names in the Ingestion process, if possible. Early file naming reduces the possibility of confusion between your primary and your backup, your editing software and your files, and between you and your clients. In general, the only people who can’t rename during or immediately after Ingestion are those who rename to remove numbering gaps caused by deleted files, and people who add subject matter tags to their file name strings.

Deletions

Before Archiving a shoot, it’s advisable to delete any files you don’t want to keep. The safest archive arrangement is one where files are never (or only rarely) deleted. So if you know you’ll want to get rid of files, it’s best to make this a part of the process for Archive preparation.

Note that we suggest that, when deleting, you make a firm decision to delete, rather than a halfhearted one. Anytime you keep a backup “just in case” you might need the file, you’re asking for trouble. If you might need the file, you should probably keep it. Of course, it’s possible to periodically prune your archive. But that should be a relatively rare and deliberate process.

Folder arrangement

Creating a static folder arrangement is essential for people who use Catalog PIEware like Lightroom or Aperture or for video NLEs. If you move files outside of the software, it may mean that you’ll have to relocate every original one-at-a-time in the event of a problem. So, as you work with images, you’ll want to maintain a stable folder structure.

And, of course, any adjustments you want to make to your folder arrangement should be done before the files are Archived.

DNG Conversion (optional)

If you archive DNG files rather than proprietary raw files, you’ll want to do the conversions prior to archiving the files. If you save this step until after the soft changes from the next section are done, you’ll get an added bonus - all metadata, adjustments and embedded previews can be embedded in the DNG, providing a disaster-recovery copy of your organizational and imaging work.

Multi-capture proofing (optional)

If you shoot panorama, time lapse or HDR sequences, you may also want to output some kind of proof of the stitched result alongside the raw images. While this is not essential, it is very helpful. For all of these multi-capture creations, it’s difficult to know how well the process worked until you have done a proof stitch.

Metadata-based “soft” changes


The file-based work to your image should almost always be done prior to Archiving, at least for professional workflows. There are also a number of steps that are typically part of the imaging workflow which are accomplished in a non-destructive manner. Since these don’t require a change to the file itself, it’s not essential to do them prior to archiving the files, but many people choose to do so.

Grouping

If the photographs or footage you have is part of a multi-part project or assignment, it is often helpful to group the images into different situations at the very start of the Working phase. Images may be divided by subject matter or other relevant characteristics. Multi-capture creations should be tagged immediately so that you don’t inadvertently delete a pano, HDR or time-lapse element. Grouping helps to make your editing process more effective, and we suggest some grouping is helpful at the very start of the Working phase.

Note that for Grouping to be “soft”, it must be done with metadata, rather than with folders.

Rating for quality

We suggest that images should be rated for quality as soon as possible in the Working phase. The ratings you assign to your images can help with every subsequent task in the workflow. It’s easy to use ratings to help you determine how much time to invest in different sets of images, whether you are annotating the photos, or working with rendering optimization.

We suggest a broad broad use of the ratings stars that lifts the best mages up above their more ordinary peers. Think of it as floating the best images to the top.

Image adjustment

After you’ve decided which images you like, you can spend time making them look their best. You can filter to a particular rating, such as “2 star images and better”, and get those photos looking their absolute best.

Derivative files


We strongly suggest that you do as much of the work as possible to your images using non-destructive PIEware. Sometimes, however, you need to do some work that requires pixel editing, or image compositing, or some other transformation of the file that must be done in Photoshop. At that point, you need to make a derivative file from the raw, which creates a more complex workflow.

The treatment of derivative files is different to that of raw files for an important reason: the work that is done to a raw file is always a non-destructive reinterpretation of the image. When you do work to a derivative file (like a TIFF or PSD), you are working on the file itself. To save this work, you need to alter the file. This means that for derivative files, the file is a Working file - on a file-based level - until you are truly done optimizing it, and ready to Archive it.

When you make a derivative file, we suggest that it gets saved to your Working file storage until you are certain that you are really “done” making any changes. For long-term projects and important files, this may mean that some derivative files are Working files for months or years. In many cases, it’s possible to archive the raw files long before the derivatives have been completed. This helps you cut down on the storage needs for your Working files, using the cheaper and more secure Archive storage for the bulk of your raw image data.

Storage and backup considerations


The hardest kind of data to protect is data that is changing. You can’t just burn a copy of the data to optical disc, since that won’t protect the work you do to the files on an ongoing basis. Likewise, it can be pretty difficult to maintain an offsite copy of your Working files that is up-to-date. And finally, it can be difficult or impossible to distinguish between a desired modification to a file and an unwanted change.

This means you’ll need to devote more time and energy to creating both your disaster-recovery backups and your rolling backups for Working files. The one consolation here is that we can reduce overall expense (in both time and money) by separating the Working files from the Archive files.

Primary storage for Working files

In most cases, you’ll want to use your fastest storage for your Working files. You’ll also want to keep the files close at hand. If you are a laptop user, this probably means storing the Working files on your internal hard drive. Desktop users will generally also want to use internal drives to store Working files. (If you need a lot of storage space and a lot of speed, you may opt for an external RAID device as well).

Backup configurations


While the primary storage for Working files can be pretty straightforward, setting up the backups can be more complex. Working files are in flux, which makes them harder to preserve. And, as undelivered files, they are also high-value files, which makes them very important to preserve. You need rolling backups, as well as disaster-recovery copies. And you need off-site copies. Let’s outline some systems for backup, starting with the cheapest/easiest, up through the most secure.

Single rolling backup

Okay, so let’s say that you are poor and that you don’t mind taking risks. At the very minimum, you need to have one backup of the files on another drive. You could just buy an external hard drive and use backup software to sync the primary copy to the backup, as shown in figure X. This does provide decent protection in the case of hard drive failure, but little or no protection against fire, theft, corruption, virus, accidental deletion or other unwanted change to the files. 

Figure x In this movie, we examine how to create a single rolling backup of your primary data.

Single rolling backup with off-site disaster-recovery backup

We can get a lot more protection by adding a second drive to the mix, as shown in figure X. This one can also be updated by means of backup software that simply keeps the files synchronized. Ideally it would live off-site (or at least off-line). This adds a lot of protection against fire, theft, lightning strike or some other catastrophe. However it still leaves you vulnerable for periods of time between updates. Any work done or files downloaded between off-site updates still has all the vulnerability of the single rolling backup system. 

Figure x In this movie, we add a second backup drive to provide disaster recovery protection.

Single rolling backup with swapper drives for off-site

In this system, we’ve changed the configuration of the off-site drive. Instead of a single drive, we have a rotating pair of off-site/off-line drives. This is a really solid system, and protects against the vast majority of potential hazards. The “Rolling Backup” protects work done every day (or every hour, depending on configuration). The swapper drive that is off-site gives maximum disaster-recovery protection for any file it contains, and the swapper that is on-site adds a good level of protection for any files waiting to be copied to the off-site drive.  


Figure x In this movie, we use swapper drives for the off-line/off-site backups. This provides a really comprehensive level of protection.

Additional option - RAID device for onsite backups

While the systems above provide a very high-level of protection, some people will want even more. Use of RAID 1, 5, 6, or 10 or other drive-spanning solutions like Drobo can add instantaneous protection for drive failure, since the data is written to more than one drive at once. This can provide no-down-time recovery from drive failure. RAID devices are appropriate for both primary storage of Working files, as well as backup storage.

Additional option - internet backups

For many photographers, it’s simply not practical to backup over the internet, since the internet connections take to long to upload a typical shoot, and the cost of reliable storage is prohibitive. There are a couple options here that could help, however.

JPEG image upload

While uploading a large raw shoot might be impractical, uploading JPEG files may be workable. You could use Lightroom’s publish tool, for instance, to create JPEGs of some or all of your images and upload to a Flickr or Photoshelter account.

Dedicated peer-to-peer upload

It’s also possible to create a dedicated network for upload that gets around the speed problems with conventional internet uploads. This can be expensive and difficult to configure, but some people manage to implement it.


Additional option - Virgin Backup

All of the backup scenarios listed above carry one particular kind of risk. Eventually, each of the backup copies gets updated with any changes made to the primary copy of the files. That’s ideal in most cases, but it does come with some risk. If something bad were to happen to the primary copy of the files and went undetected, it is possible that all backup copies might be damaged before you discovered the problem.

For those who are extra cautious, a “Virgin Backup” provides additional protection. A Virgin Backup is created very early in the workflow - perhaps at Ingestion - and it is never updated. It adds protection against virus, corruption, accidental deletion and other human errors. For maximum protection, the Virgin Backup should be checksummed so that it’s possible to verify file integrity automatically.

Ideally, a Virgin Backup will have the same file names as the wWorking files. It’s not as important to have an identical folder structure, depending on the software you are using. And Virgin Backups may be used as either long-term or short-term backups, depending on workflow, image value, budget and personal preference.