If you operate anything like me, then it’s likely that you’ve got a massive amount of duplicate images tucked away in your photo library.  Through benign actions, technical ineptitude, or simple hoarding, it’s actually quite easy to amass a huge amount of unnecessary files in your image library.  While Aperture is superb at managing a library, it somehow lacks any internal method to identify and root out these doppelganger images.

My desire to nuke these duplicate images lead me to Duplicate Annihilator from Bratoo Software.  Finding no software reviews on the interwebs, I decided to take a plunge in and see if it could help my workflow.

Malignant Imaging

Lets establish a baseline here…at the start of this endeavor, my Aperture library contains 24,403 images in 357 projects, 213 of these images already in the trash.  My vault size is 275.9 GB, however the actual active library size is 128.55 GB.  By my own estimation, I’m assuming I’ve got somewhere around 10k duplicate images since I only had about 17k images in my library before cleaning up my hard disks, upgrading to Ap3 and consolidating all stray images into a single library.  Before we can start deleting supposed duplicates, we have to come to some understanding about how they get there in the first place.

First and most obvious would be that images are copied into your library multiple times.  Aperture doesn’t really prevent you from doing this, so it’s the easiest way to add the same image into your library as a duplicate.

If you’ve transitioned from an iPhoto library to Aperture, it’s highly likely that you’ve got duplicates sitting throughout your disks anyway.  The Aperture import function does not delete any photos from you iPhoto library even if you force it to copy the files instead of referencing them.  This results in two copies of each image residing on your disk in both your photo libraries.  Additionally, depending on the version of iPhoto you started with, it is also likely that you’ve got multiple copies of any image that you’ve performed edits upon.

Perhaps less obvious is the retention of JPEG and RAW pairs.  For me, I could care less about retaining any JPEGs I’ve shot if I’ve also retained a RAW master file, but a few years of shooting in paired mode has bloated my library with extra JPEGs that aren’t needed.

Finally, there’s a setting in Aperture’s preferences under the “advanced” tab that will create a new copy of each image every time you perform an adjustment.  In my opinion, this is stupid to leave enabled as it will create unnecessary bloat in your library.  Since aperture is performing “soft” edits, there’s no reason to create separate instances of images just to see a contrast adjustment.  A right click can quickly create a new image from the original master, so by NOT forcing aperture to create copies, you will keep a cleaner library while retaining flexibility to revisit all your images.

Duplicate Annihilator – Aperture Edition

The product page for Duplicate Annihilator focuses mainly on the iPhoto version of the program, and while it certainly looks impressive and well equipped, several of the iPhoto functions are not available in the Aperture Edition of the utility.  It’s important to keep in mind that these two products function somewhat differently, as I kept looking for features touted on the website only to find that they’re not available in the aperture edition.  It’s also important to ensure that you’re downloading the correct version, as the iPhoto version is not compatible with the Aperture library.

The utility is self contained and must run while aperture is closed.  There are vague warnings against opening aperture while the program is searching the library, so it’s probably safe to assume that the utility has the power to blow up your aperture database if you don’t take heed.

The trial version is a quick download and allows you to scan up to 500 images.  For a person like me with almost 25k images, the utility doesn’t really flex it’s muscles, but you can get a good feel for the general speed of the utility.  With a price of only $7.95, I decided to just cut the trial run short and buy a full registered version of the program.

Magic Mode – not so magical

After registering Duplicate Annihilator, I attempted to run the program but I found myself perplexed at the results.  The utility located 10,224 duplicates after a four hour process, totaling 39.04 GB.  The Annihilator doesn’t actually annihilate anything at all, instead all the program does is add a “duplicate” text tag to the keywords of each image it believes to be a duplicate.  The utility will either mark these duplicates running forward or backward through your database in a crude attempt to determine which image might be the original.

When I launched Aperture, it only found 9,854 images tagged with the word “duplicate”.  I started to go through the process of deleting each image marked as a duplicate and noticed that the logic of the duplicate annihilator wasn’t exactly as strong as I’d hoped.  In several instances, the program marked a RAW version of a file as the duplicate of a JPEG version.  Poking around the library further, I noticed several images that were visually dissimilar but taken in a close timeframe were also marked as duplicates.

This bothered me seriously, as a person could easily delete thousands of images that are mistakenly marked as duplicates.  Hoping to get a better handle on how the utility was working, I wrote the program’s developer for clarifications.

The first thing he suggested was to enable the feature to assign the “original” keyword to images the Annihilator believes to be original copies, then by flagging all images marked original and creating a smart album to pull all images marked with either of the keywords “Original” and “Duplicate”, you can safely compare and eliminate duplicate images.  This seemed logical in practice, but when after another four-hour run of the utility, I now found myself looking through a smart album with almost 20k images.  Since aperture can sort images in almost every possible way…with the EXCEPTION of visually sorting them (that’d be a neat trick huh?), there’s no way to place the duplicates along side the corresponding originals.

Additionally, I found that the utility was still marking images in sequence (with different, sequential filenames) as duplicates while also randomly marking either RAW or JPG images as origionals.  I asked the developer two questions in hopes of fine-tuning the search results, but the responses weren’t what I’d hoped.

How does the utility treat JPG and RAW versions of the same image?

“The Aperture masters database always point to one distinct image  file and that file will be the file that is compared no matter if it is a jpeg or a raw.”

There are two options to mark duplicates as walking forward or backwards through the library, is that more related to the positioning in the library structure or in the exif/file dating?

“In the case with Aperture that is actually backwards or forward through the aperture masters database where backwarrds results in keeping the most recently imported copy.”

From these responses, two core problems emerge with the function of the utility.  There isn’t any weighting function to enable a person to favor a RAW file over a JPEG and the process of identifying which file is original and which is a duplicate hinges entirely on the importing process of aperture.  If aperture imports the JPEG copy of an image before the RAW copy, the JPEG will be viewed as the original regardless of any other similarities.

Because of these core flaws, Duplicate Annihilator is essentially useless and I can’t recommend the Aperture version to anyone in it’s current state.  I would submit that there could certainly be some value in the iPhoto version of the utility, but since I don’t use iPhoto I can’t make a comparison.  Just from the product website alone, it seems like the iPhoto version is more advanced then the Aperture edition, so I’d still encourage people to check that out who are working in iPhoto.  At $7.95, it will only cost you two fancy starbucks coffees to see how the program works.

Although the Duplicate Annihilator couldn’t do anything for me at all, I did manage to eliminate 9,882 duplicate images from my library.  After all the poking and prodding with Duplicate Annihilator, I found myself sorting my entire library by file name and then I went through all 24k images one by one to delete the JPG half of RAW+JPG pairs, the duplicate edits and accidental extra imports.

My library is now sitting at a comfortable 14,792 images, for a size of 90.99 GB, and all I had to do was spend four days hunting through and comparing filenames.

Every second in front of a computer is a second I’m not shooting a picture of something…

While my adventures in Aperture 3 were torturous at times, the upgrade to Ap3 and Snow Leopard did force me to adopt a sensible backup strategy I’d lacked before.  I set up an external time machine, and I managed to consolidate my files and organize my data in an effort to slim down my boot disk.

After consolidating all stray images on my hard disk into the aperture library, I moved the entire library to a secondary disk in my mac tower.  I figured the library should probably reside on a secondary disk both for space and performance considerations, and after moving the library I realized that it was already consuming 130 gigs of space.

My library is now on a separate drive and I’ve recovered about 200 gigs of free space on my boot disk.  From a baseline, this is a pretty decent environment for photo editing/organization, but how is Aperture working for me?

The bad first…

I can say that the vast majority of stability and performance issues were cured with the second and third updates.  The endless hanging and beach ball spins were eradicated with these patches, which also re-enforced by belief that it was a shoddy first release and not a lack of RAM on my part that was leading to such poor performance.  That being said, even after the patches and drive work I’ve performed, the program is still noticeably slow.

I’m not talking about switching between project folders, or waiting for faces to cache, or applying brushed selective focus atop 19 other adjustments…I wouldn’t be surprised at any sluggishness in those operations, what I’m talking about is a noticeable delay in something as simple as the drop down file menus.

Places and faces both cause a good hang upon loading, but most annoying is the delay between the time the “loop” icon is pressed and the time it actually appears.

I don’t believe these issues are related to system horsepower, but instead are likely to be a result of the actual program.  My suspicion is that there might still be some performance gains to come from 64bit optimizations, so I’ll continue to judge aperture as a work in progress.

the good…

Perhaps not due to any feature of the actual program, but instead a result of installing it, I’ve become much more organized.  When I first installed Aperture 2, I was content with importing my iPhoto library and using it’s existing structure as the default organization.  Over time, this became a bit of a pain and the daunting task of locating a given image out of 300 some generically named projects meant that I often gave up out of frustration rather then actually working on any images.

Moving to Aperture 3 provided an opportunity to reexamine my organization and begin a cleanup process that might result in a photo library that I use instead of curse at.

and a current annoyance…

Aperture is great at organization in theory, but there is one area where it is severely deficient.  In consolidating all my pictures into a single library, it somehow ballooned to almost 130 gigs and 24789 images.  Since I started with somewhere around 17 thousand, I don’t believe that I somehow found an additional 7,000 images to add to my photo library.  Instead, I’ve managed to collect numerous duplicates of images from all corners of my hard disk.

Part of this is due to the way in which Aperture imports images from iPhoto.  When using the import utility, Aperture will gladly mine every image from your existing iPhoto library, however contrary to common sense, the iPhoto library will remain untouched even after you import them into aperture, resulting in a copy of the file in both Aperture and iPhoto.  While this may be perfect for some users, aperture gives a choice of either copying or moving images when importing from other locations…for me it seems stupid that iPhoto isn’t treated with the same options.  This difference in behavior resulted in me somehow copying my iPhoto library at least twice and perhaps more in my careless attempts at consolidation.

Which brings me back to Apertures glaring deficiency.  Aperture lacks any automatic method to locate duplicate images within it’s library.  Sure you can use a variety of smart album methods to discover similar EXIF data and use that to ferret out your clones, but there is no simple and reliable duplicate location function built into the platform.  Obviously there are many reasons why a person might want to retain several distinct copies of a given file, but there are many more reasons why someone would want to locate all distinct copies of a given file.  Aperture provides no real method to accomplish that task, which is one of the reasons why my library is at least 7k images fatter then it needs to be.

Searching google for a solution has lead me to Duplicate Annihilator from Brattoo Propaganda Software.  Since I’ve got this library problem and there aren’t a lot of reviews on google for this little utility, I’ve decided to give it a run for it’s money (at $7.99USD).  My next post will detail the results.

Let me say without reservations that I fully regret investing time and money in this software.

I have now also discovered the cause for my odd text entry quirk reported in the previous post.  Turns out that what was happening was even more stupid then I believed.  In tagging photos of “michael” I believe at some point I mistakenly tagged one of the shots as “Michelle”.  After it appeared in the main window, a face with the wrong name, I clicked on the name and corrected it to Michael.  All should be well, right?  Wrong…

In some instances still not fully understood by me, this can cause ALL of the photos of one person to be merged into the other name.  In my case, the reason why I could not seem to type Michael without having Michelle show up was that aperture had merged ALL of the photos of Michael into Michelle.  To further cloud this bug, after the merge of those two names, I could not even type the word “Mike” or “Miguel” without Michelle showing up.  I have Mike identified as well as Miguel in numerous photos, but this unexpected merging somehow caused those two names to disappear as options while still retaining them in the main “Faces” page.

I also discovered that “Greg” was eaten by “Gretchen”, “Jaina” was consumed by “James” and “Sara T.” completely absorbed “Sarah P.”  That’s as far as I’ve gotten…I don’t know how far the damage extends at this point, but with over 17k images…the usefulness of this feature kind of hinges on it behaving as expected.  There should be no way through accident, carelessness or software bug that two separately identified names can be irrevocably merged without so much as a dialogue box confirmation.

After reading this post regarding aperture’s database, I’m even less confident in the program as a whole.  I wonder if I’m just wasting my time organizing my library when I should probably be looking at migrating to a more stable platform.  How’s the Lightroom 3 Beta running for people out there?  Can it be worse then Ap3?

This guy writes about how extensive use of the “Brushes” feature causes memory leaks and brings his system to it’s knees.  While I don’t look forward to discovering new bugs and performance issues in Aperture, I can’t even get around to adjusting pictures when I’m forced to spend so much time struggling with the core faces feature.

This blog post details a few potential fixes, but unfortunately the majority of the advice out there focuses on turning off the Faces feature entirely.  Since that is one of the main reasons I wanted to upgrade to Ap3, I’ll just have to play the waiting game and see if Apple can actually get this working.

For those who haven’t tried out a demo or used iPhoto ’09 or Google’s Picasa, “Faces” is a facial recognition feature designed to find and identify all the people in your photographs.  The idea is that Ap3 will search your shots and find faces first, then ask you to identify the faces it has found.  After a face has been identified enough times, Ap3 will start to identify a given person in all your pictures and ask you for confirmation.

At least…that’s the way it’s supposed to work. 

In practice, Ap3 has found podiums, statues, crests, watches, dark blobs, paintings and billboards almost as much as it has found faces.  Ok, statues and paintings of people don’t bother me, but I don’t know how many times I’ve had to tell Ap3 that a wooden podium with sharp right angles is not a face.

I’d hoped that the 3.0.1 update would cure some of the ills I’d been experiencing with Faces, but all that seemed to do was stabilize the program a tiny bit, and everything is still as slow as tar on ice.  Ap3 still can’t find my wife’s face after tagging almost 2,500 pictures of her.  It still refuses to remember certain tags on certain shots, and in some random cases – it automagically ignores the keyboard entirely and refuses to recognize keystrokes.

I was working on a specific project containing pictures from a family vacation.  In Ap3′s defense, my wife’s family has some strong genetics, so it has a devil of a time knowing which cousin is which, but in tagging the faces I discovered the most infuriating bug.  I’d already identified sisters ”Megan” and “Michelle” a number of times in different projects, but in this one particular project I kept running into an issue with text entry.  When I would type “Meg…” on a given photo, Aperture would autocomplete Megan’s full name.

Easy peazy, right?

Except when I would start to type “Mic…” it would change the “I” into an “E” and autocomplete Megan again.  Backspacing the E and entering an I again resulted in an E.  I thought I was going crazy, or that my touch-typing skills had gone dyslexic, so I did this a few more times while watching my fingers hit the keys. 

No matter what I did, entering “M I” would result in “ME”.  My temporary solution was to misspell the name and then drag the misspelled photos onto Michelle’s existing icon in the main browser.  This is one of the rare times that I’ve actually filed a bug report on any of Apple’s feedback page, but I’m not entirely certain how this bug is happening so I have little confidence that they’d be able to duplicate the same issue.

That being said, I do now have some additional insight into the performance of Faces.  As a whole, it is still a load of crap and doesn’t work well…but I have discovered a marked improvement in performance when identifying faces on a per-project basis as opposed to using the global faces window.  The software still does not identify people it should already know, but when you tag a face, it does a much better job of finding all the other instances of that face within a project.

This seems to be completely hopeless in the global faces window, where it can’t seem to group instances at all.

Additionally, I have also discovered in working on a project by project basis, that there are TONS of faces that aperture is failing to discover in each picture.  I’d assumed it was doing a pretty good job of finding everything because it located so many blurry people in the dark reaches of a frame, but on a project level it is missing tons of obvious and clear faces right in the forefront of many photos. 

What does this mean?  It means that you’re probably going to still need to go photo-by-photo if you care about thoroughly identifying all the faces in your shots.  It is pretty simple to verify the tagged faces for accuracy, but without going shot-by-shot, you really have no idea of what aperture has missed.  For me, trudging through over 17,000 images of mostly people isn’t going to be very much fun.

Hey…at least “Places” works for the most part…right?

This update improves overall stability and addresses a number of issues in Aperture 3, including:

• Upgrading libraries from earlier versions of Aperture
• Importing libraries from iPhoto
• Importing photos directly from a camera
• Memory usage when processing heavily-retouched photos
• Face recognition processing
• Adding undetected faces using the Add Missing Face button
• Printing pages containing multiple images
• Printing photos and contact sheets with borders and metadata
• Editing photos using an external editor
• Display of images with Definition and Straighten adjustments applied
• Zooming photos in the Viewer and in the Loupe using keyboard shortcuts
• Accessing Aperture libraries on a network volume
• Selecting and moving pins on the Places map
• Adding and editing custom locations using the Manage My Places window
• Switching between masters when working with RAW+JPEG pairs.

The update is recommended for all users of Aperture 3.

For detailed information on this update, please visit this website:http://support.apple.com/kb/TS2518

First off, let me say that this update certainly improved the general stability of the program, but navigating around the library is still slower then it should be.  There may have been a minor improvement to the reliability of Faces, but it is still not terribly skilled in locating faces that have already been identified.  I have now tagged 2,128 photos of my wife and it still cannot find her in pictures taken under similar conditions to photos already tagged.

This leads me to believe one of two possibilities – that there is still something seriously broken in the faces feature, or that their underlying methodology for locating similar faces is hopelessly obtuse.

There is one final quirk that’s been bugging me this morning.  I’m not sure exactly what’s going on with color profile tagging.  I spent about 20 minutes working on a photo this morning with onscreen proofing turned on.  I exported a JPG and uploaded to a website and found the gamma to be completely off.  Now, I understand color profiling is difficult, but as far as I’m aware…Safari is color-profile aware.

Which brings me to my question, coming from photoshop I learned a lot about assigning color spaces, but I can’t find anywhere in Aperture that explicitly details what color space your image is in or what it is exporting to.  All I can find is onscreen proofing.  Thus, I have no idea why my image looks one way inside aperture and another outside of it…other then to assume that the color profile it is exporting is not the same that it is proofing.

Ugh…

Update 1: Spoke too soon.  I just spent 30 minutes tagging faces, turned to work on something else and came back to find aperture asking me to tag all the faces I’d just tagged.

At this point, I’m considering switching to Picasa for organization.

Update 2:  I’d forgotten where colorspace was set on export, but found it under “edit” when exporting.  Turns out I was already exporting to sRGB while doing my onscreen proofing in the same.  Still no idea why I’m seeing such a gamma shift when uploading the photo to a website.  I still have questions about how Ap3 is handling colorspace.

A few years back I was searching for an easy utility to complete a photobook project I was working on.  My search lead me to a free trial of Aperture,  a package I found to be surprisingly well designed and easy to use.

My initial enthusiasm was slowly tempered, as in the two years since I installed there was only been one update released.  No new features, no new improvements and worst of all…not even a hint that anyone in Apple even cared about the package.

When iPhoto’s semi-annual release added gimmicks like face recognition and integrated GPS photo mapping, I assumed that these features would quickly roll into it’s “professional” big brother, but there was still no update (free or paid).  My disappointment in Aperture’s development changed to downright frustration when I began to start playing with Google’s excellent and free Picasa on my wife’s laptop.  Seriously?  Google’s free photo software from has working face recognition?

Just when I was about to download and install the Lightroom 3 trial, Apple finally released Aperture 3!  Face recognition, yay!  GPS photomapping, yay!  Brushable adjusments, yay!  Entire codebase rewritten for native 64-bit processing, yay!

Judging by the press release, Apple released a feature-filled, high horsepower software package that can legitimately handle most photographic tasks previously owned by Photoshop’s full version.  Is Aperture 3 finally a worthwhile contender?

…that depends on if you can even get it to run.

Preface

For the purposes of comparison, I’ll baseline a few elements of my setup so my experiences with Aperture 3 can be put in context.  I start this journey with an 8-core Mac Pro tower running Leopard on a 320 GB hard drive with 2 GB of RAM.  I also have an external 500 GB firewire drive that I use with Final Cut Studio.

I shoot mostly with a now-ancient Canon 350D/Digital Rebel XT, with some additional images coming from a Nikon D70 or D200.  I’ve got a mixed library of RAW, JPG and RAW+JPG images all hovering around the 8 megapixel range.  I know I’ve got a few duplicates here and there, but my image library is sitting at a respectable 17,095 pictures.  Not too large, not too small…so I’d assume I’m a good median test case.

Speaking of mixed library, while I use Aperture 2 as my defacto librarian, I’ve been suffering under the weight of an unorganized photo library for years.  I’ve never had the time to sit down and start sorting through my photographs and thus they keep piling up in loosely organized areas according to what I’m working on at the time.  Part of my personal interest in upgrading to Aperture 3 was the potential to automate a large portion of that organization process through the implementation of face-recognition and GPS photomapping.

In preparation for upgrading to Aperture 3, I also started implementing some procedures that I should have been using all along.  Yes, I’m talking backups.  I’ve been living without them for nearly two years and that was another thing I was determined to change.

Installation

First things first – I went out and purchased snow leopard.  Honestly, I didn’t see much in the program of value to even bother obtaining the $29 upgrade, but with Aperture 3 running at 64-bit natively, I figured it might be a good time to upgrade across the board.  I also installed a 1TB drive inside the tower and partitioned it in half to serve as extra storage and backup areas during the upgrade process.

Step one was to backup my entire root drive to one of the partitions on my new backup drive.  I cloned it using CarbonCopy to have a baseline picture of my disk before I started all this work.  Next I spent an entire weekend going through my root hard disk cleaning and deleting outdated programs and unnecessary clutter. I’d been bumping against the final 20 GB of my 320GB hard drive for the past year, but with some focused cleaning, I was able to reclaim about 80 GB of space simply by finding multiple copies of the same files on my root disk.  Finally, I actually used Aperture’s Vault feature for the first time to make a legit in-software backup of my library.  I found about 8k images that were still being used as reference files, so I consolodated as part of the vaulting.

With my root disk in good shape, I made a carbon copy of it onto an external firewire drive and checked that it would be bootable.

I’d been debating for a number of hours whether to do a clean install of Snow Leopard, or to try and go the easy route and perform an upgrade.  After reading a well-reasoned post online about Apple’s OS upgrade process, I decided to just go the easy route with the understanding that my firewire backup was completely safe if I needed to perform a clean install.

I was surprised to find the snow leopard upgrade to be completely painless.  I left the computer alone for about an hour assuming that it would take a while, but when I came back to check, everything was installed, upgraded and fully functional.  What was almost as surprising was how there was almost no difference from regular leopard.  I guess I expected a few obvious tweaks, but there really isn’t much difference.

Next I decided to perform the same easy installation of Aperture.  Again, lighting fast and with a couple of clicks, it was ready to go.

Initial assessment…

As I’m telling myself how awesome this upgrading thing is, I start to realize that the process of converting my Aperture 2 library to the new software was going to take about 8 hours.  Oh, well…go do something productive I guess.  After a day of errands, my library converts just fine and aperture doesn’t crash, though in reading the forums I find that this may be an exception to the rule…

Next I decide to try and tag a few faces.  Oh…now the software has to go through all my pictures again and find the faces.  Two hours later it’s built a library of faces and random things that certainly aren’t faces.  Some are clearly identifiable people, some are patterns in walls and dresses, some are statues and some are blurred blobs of darkness in the far reaches of the night.  I tell myself that I can deal with this, it’s just a bit of sorting and the program will do the rest…this is where things start to crash.

Tag a few faces with names and…crash.  Jump from project to project and…crash.  Attempt to go full screen and…crash.

Detecting a theme here?  Something is certainly not right under the hood.  While aperture is now baked in 64-bit chewy goodness, it isn’t even half as fast as it’s older 32-bit version.  The spinwheel of death arrives at almost any alteration of any program state.  Switching between projects, tagging faces, searching geotags, adjustments, full screen…that which does not crash is painfully slow and nearly useless.

Speaking of faces…without tagging all the pictures in my library (it won’t let me yet), I can say with some reasonable certainty that I’ve got over 3k pictures of my wife.  That’s being pretty conservative.  After tagging hundreds upon hundreds of photos of her, it can’t even find her face in DUPLICATE images.  I’m not even talking about pictures that may have been taken moments apart under similar lighting conditions, I’m talking about IDENTICAL images in different projects with THE SAME FILENAME.  Even worse, I’ve now tagged the same photo a good 20 times and it refuses to save the tag.

I wish I could say something positive about the brushed alterations or the new full-screen layout, but to be honest…I haven’t gotten to use them yet.

My advise…don’t be an early adopter.  I know that .0 releases are famous for being half out of beta, but this is early adopter hell.  For a product that is hoping to take market share from the clear industry leaders, this is about as bungled a release as you could ever hope for.

If you’re interested in Aperture 3, wait a month and give the trial version a shot.  I wouldn’t lay down money until April just to ensure that the majority of the kinks are worked out.