Ingestion

Introduction

Once you have captured your images, you’ll need to get them onto a computer. You might pull the media card out of the camera and attach it to the computer with a card reader. You might transfer the files wirelessly to the computer. Or perhaps you shoot with the camera tethered to the computer, and the files come in each time you release the shutter.

No matter how your images get to the computer, the Ingestion process is a time to do some basic housekeeping to the files.

• You’ll almost certainly want to rename them
• It’s vital to make at least one backup copy right away.
• It’s also a great time to add some metadata tags that identify the photographer, the usage rights and the subject matter of the photos.
• And you may want to add some custom rendering settings, such as black-and-white conversion.
• If you use DNG as an image archive format, you may want to do the DNG conversion at this point as well.
• You will want to send the photos to some kind of image viewing application, in order to see what you’ve shot.
• Once you’ve finished the inspection process, you can reformat and reuse the media card.

If you approach these common tasks as an integrated part of the download process, you will save yourself time and increase the security of your images. In fact, creating an automated ingestion process is one of the most important things photographers can do to improve their workflow.

The ideal Ingestion


Let’s start by describing the steps of an ideal Ingestion, leaving aside, for the moment, whether your software supports all of these tasks. We’ll look at each step, and describe what it entails and why it’s important. Once that’s finished, it will be easier to understand which of these steps is most important for you.

Create download folder(s)

In order to keep track of where your downloads are placed on your hard drive, we suggest that you create some kind of a landing zone folder as a standard target for the download. Within that parent folder, you can create a folder for each project. For large projects, it may make sense to create subfolders for different situations in a project. It may also be ideal to create a new subfolder for each download.

Figure 1 shows several different Ingestion folder arrangements.


Figure 1 This movie outlines some good arrangements you can use for download folder structure.

Naming folders for each download

One technique for helping keep track of large projects is to name your download folder for the starting and ending number in the download. (This only works if you tell your cameras not to reset numbering with each media card). That way, a simple visual scan of the folder names can let you know that every card has been downloaded, if no gaps appear in the folder numbers. Figure 1 shows this arrangement.

Download

Once we have a target folder, the next step in Ingestion is the transfer of the file from the camera or the media card to the computer. The most important thing here is that everything transfers properly, never leaving anything behind unknowingly. Many people simply copy the files over in the Finder or Windows Explorer. While that works fine most of the time, there are problems that can sneak in. It’s possible that you may not select all images properly. And it’s possible that the computer may silently fail to copy everything.

For this reason, we suggest using some kind of download program to manage the process. If it’s well-designed, the program will alert you to any errors in the download process, as shown in Figure 2. 

Figure 2 ImageIngester Pro is dedicated Ingestion software. It is designed to alert you whenever the files don’t download properly.

Use a card reader

We suggest using a card reader, rather than the camera itself, for the connection of the card to the computer, unless you are shooting tethered. This prevents problems that may arise if the camera battery drains during the process, and helps to keep the camera’s USB or Firewire connection from wearing out. Of course, if you are shooting tethered, then you must use the camera’s connection for the download (unless you can do it wirelessly).

There are a variety of ways that media card readers can connect to your computer, such as USB, Firewire, ExpressCard port and Thunderbolt. The speed of the connection is dependent on several factors. The speed of the port, the speed of the card, the compatibility of the card and the reader, and the number of devices sharing the connection bus will all affect the ultimate download speed. UDMA cards are the fastest class of cards right now, but in order to get that speed in download, the reader must be a UDMA reader.

In most cases, (with the exception of USB1 connections), the speed will probably be limited by the speed of the card, or the compatibility of the card and the reader. Another common limiting factor is the number of devices sharing a bus. If you have a card reader and two external drives in the same firewire chain, then they will all fight each other for the firewire bus bandwidth. Some sports and event photographers choose multi-slot readers to streamline multi-card downloads. These are most effective when used in conjunction with software that automates multiple downloads.

If speed of download is an important factor for your workflow, we have two suggestions. The first is to do some testing, and see what different configurations yield. The second is to study the media card speed ratings published by Rob Galbraith.

Read more about media card speed and selection at www.robgalbraith.com

Clone download

In the video world, some people like to use cloning software to manage the ingestion process, particularly for formats like AVCHD that use a complicated directory structure to store all the bits of information. 
Read more about cloning in the backup section.

Rename

Each of your image files needs a unique file name in order to reduce confusion and the possibility of accidental overwriting of files. If possible, it’s best to do this as part of the iIngestion process. Many programs offer the capability of renaming during Ingestion, and offer flexible naming templates. Using a date and a sequence number can help to create unique filenames automatically. Figure 3 shows some examples of this.


Figure 3 This chart shows some camera-generated filenames, as well as some examples of automatic renaming.

Be careful if you shoot raw + JPEG and rename on Ingest. We found that some software gives the raw files and the JPEG files sequential numbers instead of the same number with different extensions. This behavior can vary according to the model of the camera and whether Ingest is done from cards or done auto-imported when the camera is tethered. (PK - Need to check with Richard and see what program did this)

Two-step renaming

Sometimes it’s impossible to do final renaming in a single step. Listed below are some common reasons for that.

• If you are shooting with multiple cameras, and you want to create sequentially-numbered images, you’ll need to download everything first, then line it up in chronological sequence, and then you can rename everything.  
• Some people prefer to remove all gaps in file numbering for delivered images. Final renaming can only happen after all images have been selected for deletion or delivery.
• Some people like to put subject matter terms in the file name, and for these people a two-step rename is also frequently necessary. While we generally don’t recommend this type of file naming for stills, it is reasonably common.

In the above cases, it’s a good idea to do a temporary rename on Ingestion. It’s best to use a temporary naming convention that is different than your final name style, so that you can easily tell if a file has been given its final name.

For video workflow, subject-matter naming is generally more important. Unlike stills, video software does not offer the same level of support for keyword-based organization. It’s also harder to know what’s in a video by browsing thumbnails, unlike photos.

READ MORE ABOUT FILE NAMING IN THE FILE NAMING SECTION

Add bulk metadata

The Ingestion process should include some basic metadata tagging. Any characteristics that the entire download has in common should be written to the metadata tags right away. In most cases, this will include the photographer’s contact information and some basic usage rights tagging. Additionally, we suggest creating metadata tags that describe the image, including the location where the pictures were made, a few keywords, and/or usage rights specific to this shoot. The movie in figure 4 shows how this can be done, and how this helps you to to find the images later.

Here is a list of the basic contact and rights information that we suggest you add to all images during Ingestion:

• Name of the creator/author
• Address
• Phone, E-mail, website
• Copyright status
• Copyright notice
• Generic rights usage

Here is a list of the subject-matter tags we suggest for Ingestion:

Caption (if all images can share the same caption, such as a portrait shoot)

Specific rights usage for this shoot

Location (as specific as you can be for the whole shoot)

Keywords (subject matter, client and/or project name)

Special instructions

What if you have several shoots on the same card?

Sometimes you might have several shoots on a card. There are two approaches you can use in these situations:

• If there is a small number of shoots, you might want to do multiple ingestions. Select all the images from each shoot, and run the entire process separately for each. Once you get more than two or three shoots, however, this approach can lead to accidental omission. It can also become too time-consuming to run multiple ingestions.
• You can also ingest the entire card and divide the shoots up after download. In these cases, perhaps the only tags you can add during download are the ones that describe the photographer. Or perhaps all the images were shot in the same state or the same country. 

Figure 4 Adding bulk metadata is an easy way to tag images for easy retrieval later. This movie outlines the basics of that process.

Add rendering settings

When raw image files are processed, the software needs to make some assumptions about how the rendering should be done. If you don’t tell the software otherwise, it will use the default settings, which may be a perfectly acceptable starting point. Sometimes, however, you may know that you’ll want a certain look for the images.

If you can add rendering instructions at the time of ingestion, the images will show up with these settings already applied. For instance, if you know you want the images to be converted to black-and-white, that can often be specified in the ingestion. Creating custom rendering settings is particularly valuable when shooting tethered. The movie in figure 5 shows how that can help speed up your workflow. 

Figure 5 This movie outlines the benefits of applying processing settings during ingestion.

Backup


Backing up the files is one of the most important parts of Ingestion. It should be an integral part of everyone’s download process. The files should be backed up at least once (to make two total copies), and preferably twice for professional workflows. Ideally, this backup should take place after the work above has been done, so that the backed up files will have the same names, metadata and rendering settings as the primary copy of the files.

There is a more thorough discussion of backup process at the bottom of this page.

Convert to DNG (optional)

) Some software supports converting files to DNG format on ingestion. We feel that this promises to be the most secure way to handle images because of the data validation checksum built into the DNG file. This checksum file will serve as a means of telling if any corruption has occurred to the file in subsequent handling and storage. However, you will need to perform at least one visual inspection of the files since the checksum will only report any data changes that occur after the DNG file is first created.

Some users may be required to convert to DNG on download. If you are an Aperture user shooting with an unsupported camera, conversion to DNG can make the file readable by Aperture. If you fall into this category, you’ll need to do your download with a tool that converts to DNG during ingestion, and then pass the files along to Aperture for inspection and further editing.

Read more about DNG in the File Format Section

Read more about DNG validation in the Data Validation Section

Verifying ingested files

The easiest method of verification is to put the downloaded files into a raw processor, and let it build new thumbnails from the raw data. Once the previews have built, you can do a visual inspection to make sure everything turned out alright. This inspection confirms the integrity of the download, and it also confirms that you shot the images correctly. You’ll want to do this visual inspection before you format the media card, particularly for important photos and videos.

Be aware that browsers set to use the embedded camera previews cannot be used to fully verify raw files. The preview JPEG can appear fine, but the underlying raw data can be corrupt. (This is rare, but we have seen this problem before. For an important shoot, this could be disastrous.) When using parametric image editors (or browsers set to build previews from the raw data) be aware that you need to let them build their cache and create thumbnails from the raw data before you can judge whether the underlying raw data is good. The movie in figure x shows how to do that. 

Figure 6 Let your imaging software build previews from the files in order to examine them for file integrity.

Release media for reformat


Once you are satisfied with the integrity of the images and the Ingestion process, you can reformat your cards and continue shooting with them. Ideally, the cards should not be reformatted before the inspection process is complete, since errors in file transfer might have occurred. If the media has not been reformatted, then it might be possible to re-transfer. Once the card has been reformatted, the possibility of recovering existing files starts to diminish rapidly.

Additional batch tasks


While the fully automated tasks may be completed at this point, you may want to consider some additional processes to be part of your Ingestion protocol. This may include the second round of file renaming (sequencing, adding content terms) as outlined above.

In most cases, however, we’re now at the end of the Ingestion process for still images, and all the work ahead of us falls under the Working phase of the lifecycle. That lifecycle stage is marked by tasks that must be done by a human, and require some level of individual evaluation of the images.

Tethered shooting

Most DSLR cameras can attach directly to a computer as they shoot. This is called tethered shooting, and images are downloaded as soon as they are created. The ingestion, therefore, happens one-at-a-time. Most of the Ingestion tasks outlined above can happen while tethered, depending on the capability of the software you are using. Renaming, adding rendering settings and the creation of bulk metadata are all common capabilities of tethering software. The biggest issue with tethering is probably the backup process.

Backup configurations

The configuration of your Ingestion backups depends on your need for protection, and the number of computers and hard drives your images move through during your workflow. Let’s look at several options for these backups, starting with the simplest system and moving up in complexity.

One-computer users ingesting into the archive

If you have only one imaging computer, and you always download your files straight into the photo archive, then the backup you make at Ingestion can also be your Archive backup. This is the simplest method, and it is shown in the movie in figure 7.



Figure 7 This movie outlines an Ingestion process for someone who can download the files straight into the Archive. One-computer users

Ingesting into a Working folder

Often it makes sense to store your works-in-progress in a place that is separate from the Archive. You might have your archive stored on an external hard drive, but keep your Working files stored on your laptop until you are ready to put them away. In a case like this, the Ingestion backups can be the same as the Working file backups. This is a good workflow for people who want to do a second rename of the files, or who want to cull bad images before archiving, or those who want to convert to DNG at a later time. The movie in figure 8 outlines this. 

Figure 8 This movie outlines an Ingestion process for someone with one computer, but who wants to do some work to the image files before transferring them to the Archive.

Temporary ingestion backups for field work

For photographers who download to a laptop on location, we suggest the creation of dedicated ingestion backups. The simplest way to configure this is to have at least one, and preferably two, external hard drives that can serve as backups while on location. The movie in figure 9 shows how this can work. 

Figure 9 This movie outlines an Ingestion process for someone who downloads in the field and keeps temporary ingestion backups.

Virgin backups

Some people like to make a backup of the original camera files as early as possible in the workflow. This “virgin backup” can offer very good disaster-recovery protection of your image files, since it creates the copy before any processing is done to the files. Each step in the processing chain introduces at least some possibility of corruption or error.

To be most useful, we suggest that virgin backups are created from files that have been given a permanent file name. This makes the files considerably easier to restore in the event they are actually needed for something. It is helpful to create a validation checksum for your virgin backups. This is done automatically as part of DNG creation, and can be done very easily and inexpensively for raw, JPEG or video files with dedicated software.

Backup while tethering

Unfortunately, tethering software does not always allow backing up files as they are shot. This leaves you exposed to potential data loss, if your hard drive were to fail before you ran a backup program. For that reason, it is advisable to use a mirrored RAID device to store the files until you can run a proper backup.

Some cameras, such as Canon DSLRs, can write to the media card in the camera at the same time that they download to the computer, providing at least one backup. Others, like Nikon and Phase One cameras, will write to the card or to a computer, but not to both. If your camera will not write to the card and to the computer at the same time, it is strongly advisable to use a mirrored RAID for storage while shooting tethered as shown in Figure 10.


Figure 10 This movie outlines the benefits of applying processing settings during ingestion.