Automagisk blog

When image archives consisted of albums with negatives or cassettes with slides, having a good archiving system was crucial, otherwise it would be very difficult to find anything. With digital storage it is still very important to have a well-organized archive, but now you need to think about backup too. Luckily, it doesn’t require much effort. Here is what I do, using RAID drives, rsync, etc.

LaCie 2big Quadra 2TB, which I use as a 1TB mirrored RAID 1 drive.

How everything is organized

I store all images on the external RAID drive,  with files and folders organized like this:

• Images/
  • 1996/
  • 1997/
  • …
  • 2008/
  • 2009/
  • Export/ (files “developed” to JPEG and TIFF from Lightroom)
    • iPhone (portable, synchronized portfolio and albums)

On my MacBook I have a folder for all the Lightroom files, including the plugins and presets:

• Lightroom/
  • Lightroom.lrcat (the Lightroom catalog, almost 1GB now)
  • Lightroom Previews.lrdata (all Lightroom’s previews, can be very big, no need to backup)
  • Lightroom Settings/ (folder with all develop presets, etc)
  • Lightroom Plugins/ ( folder with all *.lrplugin files)

See Adobe Photoshop Lightroom: Workflow and tips for more details on importing images into Lightroom, naming files and folders, organizing, etc.

The backup methods for each of these are described below.

Backup of images

I store all my images on a LaCie 2big Quadra 2TB drive with RAID 1. It appears as one 1TB drive, but contains two mirrored drives. If one drive fails, I have an identical copy on the other drive. The chance that both drives fail at the same time is very small, so this is an easy and safe way to back up without really thinking about it.  The LaCie drive(s) can also be configured so it appears as one 2TB drive (RAID 0, or “striped”), but then all data would be lost if one drive failed. Not so good.. RAID 1 performance over USB 2.0 is good, and the drive also supports Firewire 800 and eSATA (which my MacBook does not).

Before I got the RAID drive, I used two identical 500GB USB drives (LaCie Quadra) and manually copied files between them (with rsync, see below). The two 500GB drives are now used for storing music, movies (that I hope to sometime finish editing) and other things. Maybe they can inherit the 1TB drives from the RAID drive when/if single drives with 2TB capacity become available? (update: Already available from Western Digital)

Backup of images to a different drive

If I should drop the RAID drive to the ground and damage both drives at once (not very likely, but still..), I do not want to lose everything stored on them. Therefore I also backup to a different drive, an Apple Time Capsule 1TB. The Time Capsule is never moved, it just sits in the corner providing WLAN and network storage.

Apple Time Machine automatically backs up all the important files on my MacBook every hour to the Time Capsule, such as documents, accounting files, etc. The image backup is not done with Time Machine, but as a manual operation. Using Time Machine for the Lightroom catalog would trigger a backup of the very large catalog every timew an image was added or changed.

I use Mac OS X’s built-in rsync. It can do a lot of things, but I use it in a very basic way:

• Backup all image files
• If some files have been deleted on the RAID drive, delete them on the Time Capsule too

The rsync command I use is to back up all the files and directory from SOURCE (for me, it’s “/Volumes/2big/images”) to TARGET (for me it’s “/Volumes/TimeCapsule/images/”) is:

rsync -av -delete SOURCE TARGET

The “-v” argument for “verbose” can of course be skipped. The rsync operation can be run automatically at specified intervals, but so far I have done it manually. The “-e –delete” argument makes sure that any files that have been deleted from the source will also be deleted from the target.

An extra backup stored in a different location

As if the above is not enough, I also back up to a portable USB drive that is stored in my parents’ house. Portable 320GB and 500GB USB-powered drives are quite cheap, and in case someone breaks into my house and steals everything, I will still have all the files, except the ones from the days after the last backup.

Lightroom catalog backup

I only use RAW (Nikon NEF), so my image files are not really ready to use, they just contain what the camera’s sensor registered, completely without any processing. The NEF files need to get a similar treatment as is done in-camera when photographing in JPEG: Adjust colors, contrast, etc.

All the changes to colors, contrast, sharpening, etc that should be done to the NEF files are stored in the Lightroom catalog, and the NEF files are never changed. So, the Lightroom catalog is almost as important as the images.

The Lightroom catalog contains all the information about the images:

• Develop presets
• Metadata (EXIF data, keywords, titles, captions, geotags, etc)
• All editing information

The Lightroom catalog is stored on my MacBook so I can use it even when I am not connected tot the RAID drive, but it is frequently copied to the RAID drive, and also rsync’ed to the Time Capsule. Similar backup as the images.

Lightroom can also be configured to ask when started whether it should back up the catalog. I have configured Lightroom to ask every time I start (Lightroom is normally running, so I don’t see the prompt that often), and back up to the same drive. Lightroom also checks the catalog for corruption during the backup. These backup catalogs are moved to the RAID drive (and rsync-ed to the Time Capsule) every now and then, and I normally keep a few old catalogs backed up, just in case.

Note: Lightroom also writes (if configured to) XMP “sidecar” files for each RAW/NEF file, containing mostly the same information as in the Lightroom catalog (metadata, adjustments, etc). These make it possible to send a RAW file with the adjustments, etc in a separate, file, and the XMP file can also be used for recreating a lost Lightroom catalog – at least some of it. I back up the XML files together with the images.

Backup of Lightroom presets, plugins, etc

Lightroom presets are really just simple text files, as you can see in this D300 preset: d300.lrtemplate. The same goes for file name presets (for import), metadata presets, etc. Lightroom saves all these files in a hierarchy inside a folder called “Lightroom Settings”.

I save all my Lightroom plugins in the same folder as the “Lightroom Settings” folder, and install them from there using Lightroom’s Plugin Manager.

This makes it easy to backup all presets and plugins together: Copy the “Lightroom Plugins” and “Lightroom Settings” folders to the RAID drive, and (quite frequently) rsync to the Time Capsule.

Soonr online backup

Soonr is an online backup service with 500MB 2GB free storage, and more storage available for purchase. I use Soonr for all my accounting documents (invoices, receipts, etc), address book and other non-humongous files.

It is very simple to set up: Just tell Soonr which folders (or files) you want to to back up, and it does it automatically in the background. Every time a files is changed, deleted or created, Soonr will update the online backup.

The files backed up by Soonr can also be accessed online using a web browser, and you can upload files to Soonr from anywhere with web access. Soonr also allows sharing, and I am sharing all my accounting files with my accountant. Every time I save an invoice or a receipt on my local drive on the MacBook, the file is also backed up on Soonr, where my accountant can access it.

I use this highly professional and advanced hierarchy for my accounting files:

• 2008/
• 2009/
  • Bank statements/
  • Invoices/
  • Receipts/

Deleting files

I delete the images that I am absolutely sure can not be used. I do not spend a lot of time deciding if an image should be deleted or not – if I’m not sure, I don’t delete it. Why? Because deleting takes time, and storage is cheap. I read somewhere(!) that storage becomes ten times cheaper every second year, which means that in 2011 I can buy ten 2TB drives for what I paid for the one I have now. Why spend time deleting?

Lightroom provides many ways to make navigating a large image archive easy:

• Keywords
• Collections
• Ratings (stars)
• Smart collections (example: All images with the keywords “ski” and “alpine” from 2009 with more than 3 stars)
• Metadata (example: All images taken with the D300 and the Nikon 10mm fisheye)
• Geotags (ok, not yet, but Adobe has to implement this in a future release..)

There’s no need to look at all the images all the time, just create views for what you need, or simply “hide” the lowest rated images. Lightroom also supports stacks, so that 20-frame sequence can be displayed as one image in grid view.

Summary

• Backing up the images is important
• Backing up the Lightroom catalog is equally important
• RAID 1 is good
• rsync is powerful, but also easy
• Consider having a backup in a different location
• Apple Time Machine and Time Capsule can backup your system and other files automatically in the background.
• Soonr can make your accountant happy

The above is what works for me, and I hope it may be useful for others too. It takes requires some effort setting it all up the first time, but after that this is all that’s required:

1. Answer “Yes” (sometimes) when Lightroom asks if it should check and backup the catalog.
2. Do an rsync from the RAID drive to the Time Capsule every now and then. This can be automated, or done with one aliased command or a small script.
3. Update the portable “somewhere else” drive every now and then.

That’s it. You will have a quite safe mirrored drive, a second backup on a separate drive, and a reasonably updated backup in a different location.

From the lab..

This is what my work desk looks like:

The LaCie drives are behind the MacBook. The MacBook’s display is mostly used for iCal and Terminal. The Moomin mouse mat is a must. I recommend a trip to Finland to buy it. Not sure why I bought the wireless keyboard instead of the wired one – I have to charge the batteries once a month.

LaCie drives:

From the left: LaCie 2big Quadra 2TB RAID, LaCie d2 Triple 500GB (discontinued) and LaCie d2 Quadra 500GB (now in a new design). The Time Capsule sits in a shelf in a corner of the room.

What about your backups?

How do you manage your backups? Ever learnt the hard way that backups are nice to have? What do you think of the backup methods mentioned here? Comments and suggestions welcome!

Update 2010-01

I now use ChronoSync for all backups, both manual and scheduled. Highly recommended!

Possibly related posts

1. Adobe Photoshop Lightroom: Workflow and tips I use Adobe Photoshop Lightroom for almost everything I do with my images: Import files from CF cards File renaming and directory building Adding metadata and default settings to...
2. Lightroom develop preset for Nikon D300 Adobe provides some good camera calibration presets for Lightroom (included in Lightroom 2.3), but you may want to tweak them some more. I have spent a lot of time tuning...
3. Adding Google Analytics to Lightroom web template I use Lightroom’s built-in web template for maintaining my portfolio on automagisk.no. The management of the portfolio can be one entirely from within Lightroom, which is very nice. Updating the...
4. MacBook with two drives: 160GB SSD + 640GB HDD I was tired of needing to connect an external drive to my MacBook to access my image archive, and was planning to buy a 1TB drive to swap with my...
5. Geotagging with Nikon GP-1 I want as much meta data as possible in my image archive, and I geotag all my images when possible. I recently bought the Nikon GP-1 GPS unit, which has...