I've been suing Microsoft's free SyncToy to do backups, and it does a good job. One of its problems, however, is a lack of understanding as far as xmp files go. It can't detect if they have changed or not, so after the first backup/echo/synch it never backs them up again. There is a reason for that explained in a MS forum somewhere, but the point is that it cannot see a change. One reason that I do remember now is that the date doesn't change on an xmp file, since you never open it. You would think you could use SyncToy's "check file contents" option, but that doesn't work on xmp files either.

The other problem with MS ST is that it's too dumb to compare files already in two places and echo/sync them. If you ever change hard drives then you will need to rebuild your echo from scratch--see you in about a day. For some reason it can't say, "Ok he wants me to echo the files on the left to the folder on the right. Now wait, there are files already in the folder on the right. Hmmmm, what shall I do. Well, if MS had it's head out of its ass, I would compare the files on the left with the files on the right, delete the files on the right that are not on the left and update the rest. Uh O, I don't understand that. I'll just copy all of the files over now and then leave all the new files on the left sitting there when I echo again in the future." Who knows what goes through MS's heads. So like I was saying, the other big problem with ST is that it gets confused when you tell it to echo from the left to the right when there are already files on the right.

So what you have to do is delete all of your files on the right, and then rebuild the echo using SyncToy, from left to right. I think Microbrains suggested that it wasn't a problem. You simply had to synchronize the folders instead of echo. So yeah that would work, but you can't change from Synchronize back to echo when you are done. Also, I'm a little paranoid to go back and forth like that. I know the files on the left are the ones I need. I don't need stupid software synchronizing my files on the left and deleting them accidentally because it's IQ is poor.

In a any event, what I want to do is echo right to left with software that can actually understand when an xmp file has changed.
So, what's your solution?

hey dwdallam, I feel your pain. Actually the reason you're having so much trouble with SincToy is because it was not created with the intent for actually being used as a backup utility. The idea behind it was a quick and dirty option for simple problems like folder on a pen drive or flash disk. If you want something more complete, but still free, I recommend syncback (CNET review here). Unfortunately it doesn't support file appends so it will always replace a file completely if it's been changed. Although this may not be a problem with small files such as raw files, it's a pain in the rear for big things like my outlook archive. Every week I have to wait a while while it copies over my 250mb pst just because I received new mail....

Anyway, it works well as a backup system and it's free. There are definitely better options, but in the freeware group it's the best I've found. You can even backup to a server via FTP (very useful for keeping an offsite backup).

KT

I use the free version of Syncback as well. I use the backup option, not synchronize. Due to user error, at one time I sync'd the wrong drive and since my files were't on that drive, they were erased from the target drive as well. Fortunately I got managed to recover them from the trash.

I use a version of a folder-compare utility that shows source and destination folders, and highlights the more recent file, if the file exists on both source and destination. It can display in three basic modes - 1) All differences, 2) Match-difference, when the files exist on both source and dest., 3) Exclusive-difference, when the files exist only on source or dest. The options at those points are - 1) Copy all newer to older, 2) Copy either direction, even older to newer, one at a time.

On Windows NTFS drives, backing up to FAT-32 drives, the Windows timestamp on files can shift by one hour compared to the FAT-32 drive, when Daylight Savings Time goes on or off. This utility compensates for that with an option that removes the difference highlight when the source/dest. files are exactly one hour apart, but the same size.

Lastly, I execute the utility once for each folder I need to back up, using a Windows 'batch' file to semi-automate the process.

Once I complete the visual backup process, if I need to verify the contents of the files, I execute another 'batch' file that's the same as the backup batch file, except it doesn't bring up a folder-compare screen for each folder - it just runs a content-difference utility in the background, and reports the final result in a text file.

I use Syncback locally. Mozy Home as an offsite backup.

As an aside it was Microsoft's Synctoy that happily synced 6 months of images into oblivion. I'm assuming it was pilot error but I'm not entirely certain of that. Very early version of the application, however.

As another aside there is a great comparison tool called Beyond Compare. You can also use that to make directory structures identical.

I use Clone 2.1 on the PC. Simple, cheap, a bit plain, but does a very good job at incrementally backing stuff up.
Clone 2.1

Hey thanks for the ideas. I had the SAME THING happen with a version of synchToy. It got rid of a set of nudes that I could never replace. After that I never used Sych again.

Well SychBack looks like the one for me. that looks exactly like what I want. I never do incremental since I keep my files backed up regularly and I don't want stem sets sitting everywhere. That depends on the time SB takes to do a full backup though. Actually I don't even want a backup. I just need to make sure files on my main drive are the same as on my backup drive.

Do any of you know if SynchBack actually backs up xmp files after they change, since the dates don't get updated--you never open them?

Thanks again for the help.

I think it depends on how you make it do comparisons. Haven't had any problems so far.

I can't get SyncBack to update directories, either create them or delete them. I read the manual and looked for options for that function, but cannot find it.

So when I create a new profile, it will copy (backup) all the directories from source to destination. However, if I delete a directory on the source side, it will not be deleted on the destination side. I must be doing something wrong because I can't imagine a backup program that would exclude updating directories, including deleting those on the destination side that are no longer on the source side, along with all files in the nonexistent diredctory.

Other than that huge limitation, which would make the software quite useless for me, it's exactly what I wanted (given it sees changes in xmp files, which I haven't tested yet.).

I'm using SynchBAck Free and so far perfect. Barring any unforseen problems, this is EXACTLY what I wanted. Much thanks.

I jsut did a full backup/synchronize (I set up a hybred) using SynchBack. F(%*&!

I can't believe how fast this thing is. This is the real deal here. It compared all my files and did it's job in like 2 minutes--on 181 GBs of data.

So much thanks.

I use SameDir without any problems. Give it a try. This is THE fastest program, best supporting update-mode backups, where only changes are transfered to create a full mirror.
I have recommend it in my article on Digital Photography Workflow at photo.net:
http://photo.net/learn/digital-photography...rt-photography/

Here is the download link:
http://web.archive.org/web/20071219234118/.../Samedir397.exe

I found it at this location:
http://web.archive.org/web/20071219121541/...samedir.sbn.bz/
http://www.boot-land.net/forums/index.php?showtopic=6031

So there's an option in the tab -> advanced
to delete files not in destination. BE CAREFUL!! I've already skrewed the pooch once and deleted a bunch of stuff without realizing i was backing up to a folder that had pics from a trip I hadn't yet copied over to my PC (I bkp to an external HD that goes between the PC and laptop). It works just like you tell it to...

there's also an option somewhere to delete empty directories on destination...

edit: One problem with syncback is it uses drive letters. That may change as you connect and disconnect drives from your pc. ALWAYS check your drive letters before you run it. Which brings me to another point. You can schedule a profile to run ata a specific time and day (say, mondays at midnight) and it will backup in the background. This can be great or a total disaster. If your drive letters change, you may run a full backup to somewhere that doesn't (and shouldn't) contain your files at all (and will now) like a pendrive ro whatever USB device is connected to that drive letter. You see where this is going...

I have grown so annoyed and tired with the vagaries of these tools, that I have gone back to manually copying my data to a second, external hard drive.

1) Backup an entire year(s) on a hard drive. Store offsite. (two hard drives in rotation)
2) Every month backup all of current year on a hard drive. Store offsite. (two hard drives in rotation)
3) At beginning of new year, make backup of all previous years on a new hard drive and store offsite.

I am risking the last month of data, but it makes more sense that way. All the backup programs had problems with my huge panoramic source files (1GB is my largest, but 800MB is very common). I also had problems with write protect failures in these huge file transfers to my WD Mybook 500GB. Now I just move it manually, it is slightly annoying every year, and a very minor task monthly.

I sometimes worry about the fact that my data isn't verified, but I am not sure that is as big a problem as it was 10 years ago. I manually check that the number of files transferred is equal, that the folder size is equal, and that randomly picked, very large files were transferred correctly.

Yeah, I know it is reckless, but I've been going without anything but free antivirus protection for three years. I guess that is living dangerously.

Good luck with that.

Does your ISP provide no Antivirus? Qwest coughs up Microsoft's One Care.

Seriously, the most problematic infections come from careless behavior, not ordinary html websites. I have made the mistake to click okay in a fake pop-up box once. There isn't a virus protection in the world that will prevent that kind of exploitation of my own stupidity. I knew what I had done immediately, and I got the right free tools to remove the malicious code.

Argh! You wrote this just as i was deleting it. Anyway, I have it figured out. Initially something was wrong with the Profile. It's all working correctly now. The drive letters are fine. I assign drive letters manually so when I plug in the media, they get assigned the same letter over and over again.

Fike. Try SynchBack. Like you I ws annoyed and pissed with all of the little idiocies of these programs. For doing backups SynchBack has done what all programs would do. It's really nice. Please give it a try and I think you'll really enjoy backing up your files everyday rom here on out. PLus, I don't have the luxury of losing even one days worth of files. Clients would be really unforgiving on that loss.

I use Aperture for my images and back up to vaults, I have my library split into 2 and make 2 backups of each (so 3 copies of each in total). Vaults are brilliant they just back up your changes. The rest of the computer is backed up with Time machine, that just happens automatically. Some important files like accounts also get backed up offline at my .mac account.
Time machine is excellent I've come to use it a couple of times, you can go back to various points in time on your computer and retrieve lost information, from deleted emails to Word documents. Aperture vaults take the pain out of backing up images, it's so easy, everything gets backed up, all your image versions, folders, projects etc.

Kevin.

I use the SyncBack pro version, and it allows you to specify source and destination based on on the volume label of the drives. So as long as you assign unique volume labels it doesn't matter if the drive letter change, it will get it right. The free version works great, but I found the pro version to be worth the cost.

Ronny

To be honest, I never even looked at the pro version. Lets you choose drive's by Name, eh? That might just be worth the money!
I'll look into that... thks!

KT

I actually maintain three versions of all my files too. I use a RAID 1 array for redundancy. When I make my monthly backups, I am actually getting the third copy for moving offsite.

I'll have to take a look at syncback.

Please don't think that RAID 1 gives you a backup. Prima-facia it would seem like "yeah I have backup in RAID 1," and you do in real terms. But in practicality a backup really means something that is away and secure from your original copy, such as your removable disk, or a server running RAID where the disk is populated from a different power source on a different server structure, or "real" redundancy.

So if you have a simple single disk mechanical failure, your RAID 1 will recover that since identical information is written to both disks. However, if you lose a disk due to electrical malfunction, such as a power supply or outside power fluctuation, it will most likely take out both disks. This happened to a friend of mine where his entire computer roasted in less than a second due to a power surge the power surge protector didn't protect. It is safer than a single drive for sure, but for different reasons than being safe like a removable disk.

For that reason, I bought a hot swap SATA drive docking station. You plug bare drives into it with out any housing enclosure. When I'm done backing up, I pull the drive out and put it upside down in the carriage, safe from electrical attacks. The technology for the docking station is just an extension of server farm technology to a single docking station. Check it out. It takes 2.5" and 3.5" drives. Cheap too.

http://www.overclockersclub.com/reviews/th...take_blacx_hdd/

Check out page two where you will see a lot more images of this unit. EVERYONE is using these now. If you Google: Thermaltake BlacX you'll see Fred Miranda pop up, all sorts of other photography sites, computer enthusiast sites, lots more. I think they cost 35.00 US. I've been waiting for this technology for years. BTW the link to that page is an older review when the BLacX didn't have eSATA, but only USB. The new ones all have USB and eSATA capability.

Really though check out SyuncBack. It really is the shinola.

Not sure that I can contribute anything terribly useful here but I've been using Robocopy for donkeys' years and have not had any problems that weren't attributable to setup stupidity on my part. Whether it deals with the xmp issue is another question however - I'll have to take a look. But it has switches that cover most of the configurations that I've ever needed - including in a mission-critical (ugh) production environment. I'll have a look at syncback though!

On eSata, a word of warning that may only apply to a dotard like myself. If your internal drives are configured as "dynamic disk", the addition of any new drive, including one mounted via the external eSata interface (which is just Sata on a long wire) your entire ntldr configuration wil be instantly ****erred resulting in a non-booting system... Of course the dynamic disc mode was an error on my part whilst building this pc. I'd been playing about with various RAID modes and got sick of re-installing the os, and left the hardware in this configuration. Bad mistake. I managed to find a fix but it's horrible.

I'm still using eSata for my secondary backup external drive (in a caddy as described above) but I leave the drive switched off and only switch it on after booting and when I run the secondary backup script.

Roy

You should pull the drive out of the caddy so it interrupts the flow of electricity in surge conditions. Just because the switch is off doesn't mean there isn't a direct wire connection to your drive. Electricity can arc aross switches, especially those very small distance between contacts on a little push button on of switch in our docking stations.

For those that do use Synch Toy, as I do, you need to look at the 'Contribute' option, not 'Synch' or others - this is where people are going wrong. 'Contribute' copies files one way only, no deletions at all. It simply copies files that are new or changed. If you delete a file locally, it does not delete it on the backup drive- it's one way only.

"Contribute will add changes from the left folder to the right folder. New and updated files are copied left to right. Renames on the left are repeated on the right. No deletions."

Check out the help files in Synch Toy and look for 'Contribute'. Read the other 4 methods and you'll see where people are going unstuck.

SynchToy is limited but does what it says it does. It's no use for backing up systems but for copying directories of images to a second location on a regular basis it's very effective. it runs at midnight every night and I have yet to have any issues - touch wood!

Which means that if you already have source and backup files and want to run SyncToy to recreate the profile, will need to delete the entire back up first, then run SynchToy. I've done some extensive research into what SynchToy will do and what it will not, and it will not do the things I and others have described here, such as seeing updated xmp files, which it cannot see.

On the other hand, SyncBack does it all. If you have files on the backup side and files on the Source side and you want to create a new profile and then run the job, deleting any files on the backup side that are not on the Source side any longer, while copying/overwriting new and updated files--SynchBack does that perfectly. ST can't, period. Go to MS's forum on SyncToy or look for articles on the MS website--and others--and you will see these two concerns are very real. SyncToy also leaves its stupid databases lying all over your drives in every directory, Source and Destination that you create. SyncBack keeps them all in one place, like it should.

Compared to SyncBack--a free program--SyncToy is not a toy, it's a joke.

You are right about the RAID limitations. To mitigate that possibility, in addition to the RAID, I have invested in pretty decent uninterruptable power supply for my PC unit.

I took a look at the synchback webpage and it seems to have all the same features as the genie-soft product I used. I particularly liked the Genie Soft product because its compressed files were in standard zip format. I liked that because if the software manufacturer went out of business, I would always have the zip files. The problem I had was that I kept getting Delayed Write Failures on my very large files (larger than 1GB). I also tried Acronis Drive Image and one other program; they all had the same problems. Any tool that compresses the file takes so long to finish the compression of these very large files that it seems that the hard drive (or OS) times-out and makes the backup fail. I expect SyncBack would have the same limitation.

So, now I backup manually.

That hard drive dock is cool. I have almost bought it a few times. I have a similar gadget that is even more flexible. On one USB cable it has data connectors and power for PATA 2.5 and 3.5, and SATA. With this cable I can plug-in any bare drive I want. This is nice because I still have some decent sized PATA drive laying around to use for backup.

This Hard Drive docking station enables eSATA and USB. eSATA is much faster than USB2. I might need one of thee.

eSATA HD Dock

Fike,

That's the one I was talking about. It's the BlacX. It also uses less CPU and is slightly faster than it's closest competitors.

As far as backing up and compressing, there are several ways to get around your problems, which is a limitation of the compression algorithms (if I remember correctly).

(1) Tell windows to compress your backup drive. This is probably the best choice.

(2) Use TrueCryp and create a volume on your backup drive, either partial or full. The entire drive will be encrypted and compressed. Once you mount the drive, copying, backing up, etc., will work like a regular drive, all transparent. Plus all of your information is encrypted. TrueCryp is Open Source and won't be going broke You can also use TruCrypt as a boot level security tool, meaning your entire hard drive is compressed and encrypted at the boot level. TrueCrypt writes to your disk MBR anbd sets up a boot level password login before the OS is booted. This is overkill.

(3) Use most any ISO creation software and backup your files to the ISO, which you mount as any other drive. The ISO would reside on your backup drive. You can create ISO's that are compressed. The only problem with that approach is that all of your files are in ONE container. if the container gets corrupted, you lose it all. I don't know how likely that is though, and I've never seen an ISO "go bad." ISOs images are really cool. I could probably come up with several even better options, other than (1), which is your easiest and most workable bet.

Once you get your backup work flow down, you will feel a lot of unknown stress disappear from your life. At least I do now that I started using SyncBack. I can't tell you all how much that reduced my stress level.

Again, thank much for passing SyncBack on to me.

OK it won't do that although I'm not exactly sure why you want to - but no matter, it's your workflow not mine and it obviously works for you.

OTOH, ignoring the issue of little files let lying around , it does exactly what I want it to, which is to make sure all the files on my internal storage drive are copied out to a second location if they are changed or added to. I don't want to ever automate the deletion of any files - too risky. I'll do that as a separate task, if I ever need to.

I'll check out Synchbabk in the meantime...

Nick,

It won't copy your xmp files or any files like that because it cannot see that the files have changed. I have thousands of hours locked up in xmp files. I sure don't want to lose that time.

I use both methods with SyncBack.

I use external cabinets with eSata drives, and have one drive that is synced (new ad and modified files are copied, and deleted files are deleted) and one drive where new and and modified files are copied (and old versions preserved) but no delete. If the main drive fails I just swap it with the synced drive and is back in operation in 5 min. And if I make an error and delete or modify files I can get them back from the backup drive.

Ronny

OK, I see.

I use DNG files almost exclusively so I don't have the same problem. One advantage of DNGs I guess. If I used sidecar files or other xmp stuff I'd be in trouble.

To repeat myself, try Clone 2.1. That can tell if a file has been modified in several ways. I've used it when working away with a PC laptop and need to keep my 2 ext HDs in sync.
It's a very neat and powerful, if basic looking bit of software.

Two questions related to this thread:

1) What exactly is stored in an XMP file?

2) Why doesn't SyncToy pick up changes to these files?

THanks,

Peter

I run all my images off an external drive, via FW800. When the images are downloaded to the main external drive ( using Lightroom) they are copied right away to a second drive using Chronosync. Only then do I erase images from the memory card.

Once I have sorted the images, and deleted the duds, the main drive is copied to the second drive again, thus eliminating the duds there too.

At this stage I burn copies of the originals to Archival Gold DVDs.

So at any time my Lightroom library is on 2 drives, and originals are on archival quality DVDs. ( Archival Gold DVDs)

I will soon add another drive to which I will back up the library once a week and keep off-site.

The final stage is an online backup of my favourite images.

Simple, effective and working like a dream for years (before Lightroom it was a main image folder).

For my Mac backups I use NTI Shadow or Retrospect for software.

My main focus when designing my backup system was not how to do it but what to use. There are so many methods and solutions out there and many work very well. I wanted two backups for redundancy and online storage in case something happens to both hard drives like an electrical surge or fire.

So here is what I came up with:

For hard drives I wanted redundancy so I looked for a RAID 1 mirrored hard drive enclosure so every time I backed up the data is stored on two drives. I ended up looking for an SAN enclosure that can always be accessible on my network regardless of what computer I am using. There are also cheaper enclosures out there that connect via USB/Firewire/eSATA that are only local hard drives but have RAID 1 mirrors built in.

SAN network hard drive setup (this is what I use):
http://www.dlink.com/products/?pid=509
Seagate Barracuda 7200 RPM drives

local hard drive setup:
http://www.tigerdirect.com/applications/se...ygtCjCVRqCjCVRq

For online storage I use iDrive. For $49.95 a year you can have one personal storage account for 150 GB. If you need more storage you can add up to a total of 5 150 GB accounts at $49.95 each per year. Not bad for 750 GB of storage compared with any other online storage service. I think you can store all or at least most of your keeper RAW files with 750 GB. You can purchase more storage but the cost goes up significantly since they consider it a business account after 750 GB.

http://www.idrive.com/

I've been using this system for a year with no faults or crashes yet. Worse case scenerio if my place burned down at least I have my network backup as well.

I hope this helps,

Jeremy

One thing to note:

Yes, a raid means that your data is on two drives. But note that should one of the drives become corrupted for any reason, the corruption is copied to the second drive automatically. It's healthier to manage the copying so that it only happens when you have confirmed the data on the first drive is healthy.

One (just one) of the major problems people have with backing up to DVD's is finding something when you need it. For that, you need to catalog the DVD's, i.e. each DVD and what's on it. We assume nothing can go wrong with an archive DVD, or do we make two? Back to the catalog. How do you describe what's on the DVD's? A general description like "Bob's wedding - 12/15/2008?" Or how about "Landscape shoot - redwood forest - 1/13/2009?" Sometimes you need to find something by a specific thing in an image that isn't in the catalog. I've found it necessary to file a collection/archive set in its own folder, and within each folder each image has a filename that describes the main point or object of the image, or possibly that plus a secondary subject/object in the image. Then, archiving all of those on a hard drive instead of diskettes, I can find precisely what I need quickly with a foldername/filename search, without having to maintain a hand-written catalog.

I make database software for process manufacturers, and what do you suppose their most requested (and least realized) issue is? To be able to easily get to all of the data they entered into their database, quickly and painlessly. Not happening yet.

Me Too!
And this Hard Drive accessory makes it easy!
http://eshop.macsales.com/item/Newer%20Tec...ogy/FWU2ES2HDK/

that is a great one, because it has firewire 800 which is very fast. the others brands are generally only usb2 or eventually sata. sata is a bit of a pain for mac users.

I was wondering, how do you guys store/protect the bare hard drives?

DriveBox
http://www.wiebetech.com/mm5/merchant.mvc?
$7.00 usd

Based on the recommendations here I tried Syncback Pro. It work superb, finding exterior drives based on the name that have been disconnected and reconnected. This is important to me as whenever I leave town, I remove all the drives, securely hide the originals and leave the backups at a relatives. Before plugging them back in reassigned them new drive letters and every program I used required a new backup profile then. Not with Synback Pro however-great. Also extremely fast writing from and to eSATA drives (much faster than other programs I had tried). Thanks all.

Hi!

I'm on Mac...

The way I do it is:

1) I have two 2TB hard disks in the Mac called Master and Backup. All originals go to "Master/DigitalPictures". "Backup"is used by TimeMachine for automatic backup.

2) I have a separate RAID 5 consisting of four older 1TB harddrives. This contains derivative works + a copy of "Master/DigitalPictures" and also of /Users/ekr (ekr that's me). This backup is done using "rsync" each day 23:00.

3) I have an offsite copy of the "Backup" disk updated now and than. on a 2TB extern hard drive I keep at work.

All the tools are part of the OS, except CarbonCopy Cloner which I use for cloning the backup disk.


Best regards
Erik

I have 4 x 1TB drives in my Mac Pro.

I have 2 different sets of 4 x 1TB drives (Set A & Set B ).

I use ChronoSync to backup my data to a Drobo drive with either Set A or Set B installed. The other set is stored in a Pelican Case off site. On a weekly basis or after a big shoot I will do a complete backup and then swap the drives and store these drives offsite again.

I'm thinking of adding a third set (Set C) to the equation as I don't like having all my data in one location at a time. By having two sets off-site and bringing one back for exchange I know that I still have at least one other set off-site.


I also found a site called BackBlaze.com that provides unlimited space for $5 per month. I'm doing some research on it first as they comment about encrypting the files before doing a backup to their server. I'm just hesitant of the encryption on my files.

Kelly

If you like the simple approach for software and are using a PC you can make a simple batch file in notepad or similar. Just type commands in and save as a .bat file. Simple but effective.

xcopy "path to files to be copied" "destination to be copied to" /s /d /h /y /i (/s is for all sub directories, /d is to check for date and time, /h is for hidden files, /y is yes to all when it asks to overwrite, /i create the directory if it doesn't exist)

This way you can just copy new or updated files to your external hard drive by executing the file. You can add a delete line if you use a directory like desktop as a temp place too so deleted files don't build up and more.

eg:

del "e:\backup\mypc\temp" /s /q
xcopy "C:\mypc\temp" "e:\backup\mypc\temp" /s /d /h /y /i
xcopy "C:\mypc\mypics" "e:\backup\mypc\mypics" /s /d /h /y /i

You can add as many lines as you like, works over networks too.

I wrote a utility program just for this purpose and giving it for free to the world;)
www.michaelezra.com check downloads, "Smart Disk Catalogue".
If you will use it, i would appreciate some feedback.