The Beginning
I used to have a very simple workflow that most amateurs probably use; I simply copied the photos off the camera cards into a series of folders named for the activity I photographed. This reflected how I used to manage my film based photography. Stick the negatives into bags in a folder and organise the prints in boxes. This suited me fine in those days simply because the cost of film photography made you choose your images carefully. Any one activity would probably generate only four or five images.
Around 15 years ago I converted (almost) exclusively to digital photography. My image habits have changed from four or five images per activity to hundreds and sometimes thousands. Most professional photographers will shoot between 500 and 1500 images per activity. I don’t (generally) earn an income from my images, nor do I intend to but l follow professional practices and also generate huge numbers of images.
This was simple while I only had a few thousand images. Once I reached 100,000 (you'll be surprised how quickly this happens!) I had generated a nightmare for myself. It became very hard to find images I wanted (that I remember taking) for some particular purpose and keeping backups in sync became more and more problematic. I made the situation even worse by managing pictures on more than one computer.
It took nearly 10 years of digital photography I have nearly six terabytes of image files. I have three different camera RAW formats and have JPEGs ranging from lowly 320 by 160 from my first digital camera to the 21 megapixel images from my current camera of choice my Canon 5D mk II.
Modern photographic techniques such as High Dynamic Range where you take 3 to 20 (or more) images from a tripod with various bracketing, white balance and other settings to be merged into a single image worsen the problem.
The problem really came to my attention this year when I simply could not find an image that I needed. I knew it was there, but fruitlessly scanning through hundreds of folders left me drained and wasted a great many hours with no outcome at all.
I decided that I needed a management plan for my images so I started to seek what others had done, I read some blogs and personal web sites of photographers, I researched other online sources such as www.dpbestflow.org but they only scratched the surface. I was a member of the Australian Photographic Society. These organisations regularly distribute information and member only offers and include magazine subscriptions. One of the offers was for Peter Krogh’s “The DAM Book”. I had read the information on Peter’s website previously but wasn't all that convinced the book would help me, this time it came with a complimentary license to Phase One’s Expression Media 2 (formerly a Microsoft product). This swayed me because I knew I would need some decent catalogue software. The available free products like Microsoft Pro Photo Tools and Bridge that came with Photoshop really didn't cut the mustard. Not to mention that Microsoft have recently dumped their image meta tagging, geo-tagging and image management software.
The book arrived and I read it from cover to cover – I skipped some parts such as the explanations of how computer storage works and the available options because, well, I do that for a living. I digested Peter’s suggestions and constructed a workflow that is part way between what I had before and the extremely rigorous practices suited to a professional who is earning their bread and butter from their photography. In generating my workflow I took into account that storage is extremely cheap and that external drives can be purchased for around $25 per terabyte at the time of writing. This has lead me to the point of deciding not to backup to digital offline media such as DVD-W or BluRay. With hard disk storage being extremely cheap and reliable and archival quality media still being expensive and an unknown quantity for lifespan I have built my model around multiple online and offline hard disk storage. Update 2011: Ironically I no longer use EM2, I outgrew it very quickly. Today I use Adobe Lightroom. The cataloguing and editing facilities are excellent. It is now rare for me to need to visit Photoshop because I can do it all in LR.
Will my workflow work for you? Only you can answer that question. All I can say is that it works for me and that I’m writing this article to make it available to other people.
My Workflow
I have tried to keep my workflow as simple as possible while still following the principals of good digital asset management. I have balanced being anal retentive about backups and keeping copies and the costs of time and money in doing so.
To construct my workflow I determined what my basic requirements would be:
Need to be able to create images in any format supported by my cameras both today and into the future
- Need to be able to ensure my images are available in the future
- Need to be able to view, edit and update my images in the Microsoft Windows platform
- Need to be able to manage my images both in a master repository and on the move
- Need to keep a backup of my images as they came from the camera before I work on them
- Need to be able to find my images easily and quickly
- Need to be able associate metadata with the images to interpret them for the future
- Need to be able to surface my best work
- Need to support post processing of my images
- Need to support publication of my images
- Need to support delivery of my images to other people
Your requirements may vary so you should start the process of determining your own workflow by writing down your requirements and thinking about how you will measure your success when you have “finished” creating your workflow.
Each of my requirements has dictated elements of my workflow. Some elements satisfy one requirement, some satisfy several at once. Let’s take them one at a time and interpret what they mean.
Need to be able to create images in any format supported by my cameras both today and into the future
Over my digital image taking experience I’ve had several cameras, initially they either had a proprietary version of an open standard (e.g. Casio had their own JPEG) or supported the open standard directly (e.g. Sony FD Mavica supported JPEG) or had both their own RAW format and JPEG support (e.g. Canon 300D, 40D and my current Canon 5D mk ii and my baby Olympus EPM-2).
Those of you who have been around computing for a while will be aware that file formats change over time. It can become challenging to handle and old format in a modern computing environment, particularly if that format is proprietary and requires tools from the manufacturer to work properly.
I don’t intend to enter the RAW vs. JPEG debate, but for my purposes I have changed my position – I used to shoot exclusively in JPEG because it was easy and open. Today I shoot in RAW because my photographic talents and my requirements for post processing have outgrown JPEG – I simply demand higher quality of my own work than I used to. The problem that this causes is RAW formats are proprietary and usually brand unique and often body unique.
Because I need to be able to use my images in the future and I choose to use a RAW format I don’t want to be locked into to any manufacturer or have to keep out-dated software on my modern computer that might impact its ability to work properly I choose to convert my images to the “open” image format known as Digital Negative (DNG) by Adobe. While this is still a proprietary format in that it is owned by a commercial organisation, Adobe has promoted it as a public archiving standard and has published the file content and format to allow other software vendors to utilise it without having to use Adobe’s software.
DNG conversion appears to be faithful, but to mitigate the risks involved in file format conversion I back up the camera RAW files prior to conversion. A secondary reason for retaining the original RAW files is to enable use of new software that appears from time to time that requires the original RAW format to work correctly. Examples would include updates to Canon’s RAW utilities to reduce noise reduction.
Need to be able to ensure my images are available in the future
This requirement both crosses over into the previous requirement and dictates that I need rigorous backup of the images. To ensure my images are available I have to consider:
- Protect from software change
- Not be locked into any one software vendor
- Not be locked into any one operating system
- Not be locked into any one camera platform
- Protect from destruction by fire or other natural event
- Protect from theft
- Protect from hardware failure
- Protect from software failure
- Protect from myself (problem between brain and keyboard)
Use of DNG covers the first four points and my insurance and backup strategy covers the others. My backup strategy is a little anal (but does not go as far as some people do) and would work well in any enthusiast or studio setup where there is only one main photographer.
My desktop computer is my primary or master image store. I have it configured to use several sets of two drives together so that if either of them fails the data is intact on the other one and it is a simple matter of replacing the failed unit and the computer itself will copy the data to the new unit. This configuration is called Raid 1 (mirror) and is managed in hardware.
During ingestion the images are backed up, converted to DNG then copied to a working area.
Post ingestion I backup my images to an internal mirror set and to offline portable drives. Note if you’re not a computer geek you could purchase network based file storage or use an additional portable drive in place of the server. The point is to have your master and THREE backups to cover you from various problems that could occur. I'm not going to go into why three here as there are plenty of articles on the Internet you could read.
The portable drives are kept offsite. I have two that I use in rotation and perform a full backup weekly. At least one of the drives is always offsite.
Need to be able to manage my images both in a master repository and on the move
In its simplest form I want to be able to access my images at home or at work. I also travel around and often want my images to be with me. With nearly 6 terabytes of images this is not really practical – even with the affordable drive storage available today.
However, what is practical is the use of cataloguing software in which thumbnails are stored. This same cataloguing software has other uses that meet other requirements.
I currently use Adobe Lightroom but keep an eye on the improvements in other available software. Don’t get too wed to any one software tool, just make sure your choice has the ability to export and import in common formats and you’ll be ok with any choice.
Need to keep a backup of my images as they came from the camera before I work on them
This requirement is met by the backup strategy of keeping the virgins, both of the camera RAW and of the converted DNG. Keeping both means that I can edit my DNG files to my heart’s content and do not have to be concerned if I use an editor that is not parametric (in other words it changes the actual image file) because I always have the originals to go back to if I need to (and I have needed to from time to time).
Keeping all these backups would be onerous if done by hand, so I use Ingestamatic to manage this process (along with metadata). Ingestamatic is produced by Marc Rochkind http://basepath.com/site/detail-Ingestamatic.php
Need to be able to find my images easily and quickly
There are people working on being able to search for an image by describing the image or by giving the search tool a similar image. These tools are a way off. In the meantime text based searching is all we have. But where is the text in an image? That is where metadata comes in.
Metadata is the secret to finding anything when you have a lot of things. Metadata is defined as being data about data, in my case data about images, when, what and how they were taken. There are four sources of metadata that I care about in my quest to quickly find images and there is other metadata that is useful during image use.
- Metadata recorded by the camera with the image – e.g. the camera records details such as the date and time, the camera settings, the file name, and sometimes the attachments used
- Metadata recorded during the ingestion process by the computer – e.g. File dates and file location
- Metadata recorded in bulk against a set of images by me – e.g. import of a GPS log to geotag, bulk metadata such as the location, the event or activity being recorded
- Metadata recorded both individually and in small groups by me – e.g. metadata particular to the image such as people’s names, a description of the image, a rating (out of 5)
- I use Light Room for these tasks. I keep a single catalogue for all my images. I have to keep my computing hardware up to date to manage this successfully.
Need to be able associate metadata with the images to interpret them for the future
After searching, the next use of metadata for me is to interpret the image for the future. This usually means recording the names of the people and places in the image. It can also mean describing the image so that you know what it is later in life through text and keywords. As an example, if you take a series of abstract macro photos and want to refine or reproduce some element of them again in a few years’ time then you need to know that the source of the yellow abstract was in fact a daffodil.
Need to be able to surface my best work
Another (and final in my workflow) use for metadata is to make it simple to locate your best work on a particular subject when you need to. You might be searching for an image to enter into a competition or you might be fulfilling a family request for an image from an event or providing it to your model. In my case it also supports my web activities so that I never need to purchase stock images from someone else to go with my articles.
The ability to locate your best work from an image stream cuts down on the amount of time you spend on images in post processing. Why waste time on poor images that you may never use. If it comes about that you do use one you have not lost anything but not immediately post processing – just do it later.
To achieve both, I use a system of ratings – most cataloguing software supports ratings, whether it be numeric or by stars. For me:
Stars
|
Meaning
|
Percentage
|
☼
|
Low quality image
|
5
|
☼☼
|
The image is mediocre but useable
|
20
|
☼☼☼
|
The image is good – the composition, lighting and subject are
ok
|
60
|
☼☼☼☼
|
The image is exceptional – the composition, lighting and
subject are all great
|
10
|
☼☼☼☼☼
|
The image is amazing – everything is just right – it could not
be better
|
5
|
Why do I have the bulk of my images falling into two and three stars - am I that poor a photographer? No I'm not but I would be kidding myself if I said most of my shots were exceptional or amazing. Even top notch professionals have plenty of mediocre photographs. I will do detailed post processing putting possibly hours into perfecting a four or five image. I put in a few minutes on the others and nothing on the 1’s. Some of my images don’t even make it to 1’s. If during rating I think an image is so bad that it would never have a use then out it goes. Remember this is not gone – I still have the virgins hiding in the wings if I ever really need it back. It is probably worth nothing that the post processing could promote an image in the stars rating. If I think it is deserving after PP up it goes - it could also go down when I discover it isn't as sharp as I initially thought. There are some things you don't see in thumbnails!
Need to support post processing of my images
I choose to post process all the images that get two stars or better. I won’t spend much time on two’s, each three gets a little more individual attention and the fours and fives get much more.
I keep the working copy and manage all of the post processing on that copy – either through scripts or manually. The better the rating the more manual the post processing gets.
This is where my workflow departs from the really anal retentive ones; they will talk about derivative images and will take great pains to not modify the original working copy making only parametric changes. This means the original copy stays true and that the post production is a set of adjustments to be applied to the image and not an image in its own right. This work method does not suit my practices. It is rare for me to generate a true derivative that I would keep alongside the original – not just as an adjustment. When I do, that new image is manually copied into the virgin backup manually. I also cannot see the point of keeping the working copy true when you have at least two copies of it in the virgin backup anyway.
Need to support publication of my images
I publish a great many of my images on my website www.steamengine.com.au and this blog and on social media such as Google+.
My images are huge. I'm not going to put huge images up on the Internet – even if you want to, most upload image systems resize it on the way – all you do is waste your bandwidth.
To publish my images I resize them, (generally) strip the meta data, apply a watermark (I've had images stolen and used commercially – the watermark IS necessary) and save them as an 80-90 quality JPEG depending on the use. Remember when I said I don’t make derivative works? Ok, so I lied. Generally speaking I do NOT keep these derivatives which are why I don’t consider them to be derivative works. Once they are uploaded to whatever gallery they are going into I consider that they’re no longer managed.
I've been publishing on the Internet and in magazines for many years and while I've had occasion to replace lost images from the website after a server corruption the workflow to create them is automated and quick. Having to do some rework also forces me to consider what I'm publishing – is it still current – do I have better images of the same object/activity/event? Could I do a better job post processing them again from the virgins now that my skills have improved? And of course… is it actually worth my time to republish? Some articles on my site are a given – yes, I’d always replace them and republish but the little news blurbs that are only useful for a couple of months (e.g. locomotive G42 returns to service after an overhaul) are simply not momentous enough to keep!
Need to support delivery of my images to other people
From time to time I deliver images to other people for various uses and these images I do want to keep a copy of. I normally deliver via Cloud services such as Google Drive. My cataloguing software needs to support to copying a selected set of images to the archive\delivery folder then uploading them. On the same folder I will drop a copy of a boiler plate license agreement that I've filled out for their use.
Examples of the situations where I will give out images:
My model wants a copy – if I'm photographing a human I will always give them a copy of the images for their own use (e.g. their portfolio or on social networking sites) if they ask. This could be a professional model or just someone I've casually photographed.
I'm photographing an event run by a not for profit organisation (e.g. a preserved railway) and they ask for a copy of the images.
Someone sees me taking an image or images and wants a copy – e.g. the property owner or someone with a related interest.
To reduce the hate mail I'll get from professional photographers who charge for their images: I'm not reducing the work available to you by handing out images. Puffing Billy Railway for example, has a professional photographer who they engage for their major events but they simply could not afford to engage someone to work on the railway every single day capturing every little moment. They depend on people like me. They may never use my images or they may use them in a book supporting the railway. This is one of the ways I support them.
My workflow does not need to support
When you read about workflows you'll discover that a lot of them talk about archiving the working copy of an image. I don’t do this – I keep all my images online (both the master and the backups) and I expect that this will not change. Too many people fail to recognise that storage is becoming cheaper and more reliable over time. My first hard disk was 2MB and that was only about 25 years ago!
Why put yourself through hours of effort and pain to burn media and remove images from your master collection into a secondary collection with the risks inherent in doing so when you can simply buy more storage? Both the Windows and Macintosh platforms make it simple to move the entire system to larger storage when you need to.
If you do choose to do this then you have several considerations
Digital write once media (e.g. DVD and BluRay) have an unknown life span – even the archival quality ones may only last a few years. They can claim anything they like, but the extent of their warranty remedy is to give you another disc if they’re still in business when you make your claim. But what about your images – they’re gone!
Moving files from one location to another is dangerous. All sorts of failures in hardware and sometimes even those found between your brain and your fingers can cause you to lose the files or store them some place you would not expect to find them. This means you need a tool to verify that the move is complete. They usually work by doing a copy, verify then delete.
Copyright Support
I don’t require support in the workflow for Copyright. In Australia, all works are automatically copyright to the author. You do not need to register them. It is advisable to place a copyright notice and use instructions in the metadata or within a watermark on the image when publishing – this is covered during my publishing. I use and recommend Creative Commons license tailored for your needs. If you live in a country where it is advisable to register your images to ensure that you do hold the copyright then you should include this step in your workflow.
Backup
In my main photo workflow computer I have two RAID 1 (Mirror) arrays. I have three arrays
Array 1: 2TB Windows and non photographic stuff and all software
Array 2: 6TB Main photography working store
Array 3: 6TB Main photography in machine back up
Portable Drive 1: 6TB Offsite backup
Portable Drive 2: 6TB Offsite backup
I keep one off site away from my workflow computer and the other is connected. At the end of every major session I backup my catalogue using LR and my images using Microsoft SyncToy to both the internal backup and the external backup drive. Six monthly I format and do a complete copy. Roughly once a month I swap the external drives - ensuring the offsite copy is always reasonably up to date. If I do a big editing session or work on images particularly important to me I will swap the drives sooner. When I swap I take the onsite drive off site and bring the other one back so I am never in a situation where all copies of my images are in the same place. As my catalogue grows I ensure the storage keeps pace. Storage is cheap. At approximately AU$25/TB you can afford to have plenty.
When in the field on an extended trip I use a laptop and pair of USB 2 portable drives for the backups. The primary copy is on the laptop and this is synced to both of the portable drives. If I need to ensure a copy is safe I will send it home.
Note that a lot of people hit the fashionable and sometimes limitless cloud backup solutions. I believe these are fine for documents and the like and I use a service for that. However, for high volume I suggest you read the fine print. You can certainly store an enormous amount on a lot of cloud services but there are three common issues that I've seen discussed:
- It can be slow to establish a backup of a large existing library
- The safety of your backup is only as good as the business model of the company you use - what will you do if they go broke?
- Often (always?) to get a large restore you need to have the company create and ship media or drives to you. They usually (always?) charge quite a fee for this activity.
Tools
I use a number of tools in my workflow. I used to have quite the collection but I've cut this down over recent years.
- Lightroom (via Adobe Creative Cloud)
- PhotoshopCC (via Adobe Creative Cloud)
- Adobe Camera Raw
- Adobe DNG Converter
- Spyder Monitor Calibration
- Ingestamatic
- NIK Photo Tools Collection
I'm always open to new tools and processes and download tons of them to try but I always seem to come back to my favourites. You'll hear people suggest that there are better raw engines than Adobe blah blah blah. The truth is I want a seamless experience where the tools simply work. I don't want to spend time and disk space transferring giant image files between tools and avoid it wherever I can.
Folder Structure
My structure is really simple
Working
--- YYYY
--- yyyyddmm shoot-name
Each shoot has its own folder. These folders are created by Ingestamatic and used by LR for its photo store (I synchronise rather than Import).
Backup - same as working
Exports
--- social - files for social media go here
--- printing - large files to be sent to my print house go here
--- share - large files to be sent to my model go here
Make Your Own Path
Please don’t implement your workflow based on my say so – it works fine for me, or at least it does today. I'm sure I'll change it again. Change is constant. Change is one of the reasons I wrote this document – while it will help other people to make choices, it also helps me to remember why I made particular choices to help me evaluate if a change to my workflow will be beneficial.
If you follow my workflow and lose images then all you will get from me is sympathy! Don’t come chasing me because you followed my practices verbatim without first choosing if they meet your needs. When you implement a workflow make it yours. Put your own stamp on it.
