I'm not sure if I've posted this in the right section of the forum ...

I've been backing up my photo files to CD and now DVD. I make two copies and store one in an offsite location. I then remove most of my files from my PC. I've been reading about issues with the reliability of this medium for long term storage.(http://computerworld.com/printthis/2006/0,4814,107607,00.html)

What is the best option for reliable storage?
RobJ

A colleague uses high speed tape to backup his software (he's an IT engineer). The backup runs automatically at a user chosen time. Another colleague suggested hard drives. Apparently you can get a caddy that fits in an optical drive bay, and accepts hard disks. So you can swap hard disks at will. It might sound expensive, but a 200GB HD is not that much more expensive than 50 decent writable DVDs, the data integrity and longevity are higher, and copying is faster. So it might even be cheaper in that DVDs might only last a year or so. Make sure the main HDD and the backup are not on the same cable as that will slow transfers (or even better have one or both drives on higher speed SATA connections).

I use DVDs to store scanned images but it's a right royal pain, and I will probably go the HDD route at some stage. I already have two internal HDDs and backup my digtal camera images to the secondary HDD. But that is not secure in case of fire or whatever.

BTW buy yourself a surge protector if you haven't already done so. They're dirt cheap. It might help protect you from a HDD/PC crash.

Leif

The road to data longevity consists of:

- Redundancy (multiple copies of your data)
- Verifiability (checksums of the data files, stored with the copies and separately)
- Properly safe and stable storage (no light, no significant temperature changes, little or no air pollution)
- Regular and repeated verification (create new checksums and check against your old ones)
- Regular and repeated migration (copy your data to new media well within the life expectancy of the old media)

NASA learned much of this the hard way.

And, as others have suggested in other forums, take particular care with the data you care about. That could be e.g. photographs that are keepers, exhibit or portfolio grade, "memories" and family pictures. In these instances, back up RAW data as well as processed data.

Don't worry too much about the longevity of the CD or DVD media. Yes, there have been concerns, particularly with pre-printed media and rewritable media, but CD-R, DVD-R and DVD+R have - in most cases - many years of life expectancy, probamost likely greater than that of magnetic media, as long as they're stored properly. High speed tape backups sound real nice, but are also quite expensive for a single person, family or small business.

But you will have to migrate your data to new media once every few years anyway, as you upgrade your computer or hard disks. Take those opportunities to re-verify everything.

Now professionals like Jonathan or Michael with terabytes of data probably don't like the idea of verifying the integrity of their accumulated data all at once, so I expect it makes more sense to do this more frequently and of subsets of the data, perhaps "continuous" verification makes sense.

It's time consuming and expensive to take care of your data, as opposed to just putting it on a hard disk and forget about it. There are no fast roads, no cheap tricks. But even though it isn't fast or cheap, it's faster and cheaper than it was ten or twenty years ago.

Another little tip which helps preserve data integrity is to use NTFS file system rather than FAT32 on hard drives where you are archiving material. If you have Linux or Mac then they will have an equivalent journaling type of file system. The advantage of NTFS over FAT32 is that it is a much more robust indexing system such that if there is a problem with the disk then the OS may be able to recover portions of the data (it has worked for me on several occassions where a disk has 'crashed' and the OS has recovered a good proportion of the data).

If you are using NTFS on a removable drive remember to set the security to 'Everyone' - otherwise you will not be able to access the data if you put the drive in another machine.

Also, before archiving a large number of files to the archive disks remember to run CHKDSK (check disk) to map out any faulty clusters and ensure that the index is well maintained. This will prevent you copying files onto faulty clusters.

Yes, this is indeed a very critical step. I had an important customer who backed up his entire office during the night, every day of the week. They invested tens of thousands every month on leased lines and remote storage. I warned them that they must verify and not trust the system blindly. After using this system for a couple years, the big crash happened... And Murphy was invited to the party

Remote backup was a complete waste and the local was only partially usable. The boss of the office lost his job as well as people in charge of the computer system. They had to hire to manually re-type most of the lost info. Hopefully data was still available on paper.

So, backup every day to multiple locations and to multiple systems, verify your data often.

And we thought things would be simpler with digital, rather than using the film.

In the old days (not too long ago really) I did slides, scanned the slides and then stored the originals. I would back up the scanned images on a regular basis and all was well.

Things have certainly gotten more complicated now. Just keeping track of all your stuff (digital files) has become a job in itself. I used to only scan the keepers and would maintain just those files on computer, now everything is digital and needs to be managed.

Here is the latest that I use for a workable system. combination of Michael's workflow and many others.

In the field I use the computer, cf cards and a Firelight portable hard drive (two)
I copy files to computer first (nothing done to files until backups done) I make (based on trips) a folder with the name of the place and date and inside that folder I name subfolders by each day. I then backup each days folder to the portable hard drive. (you can then have a second hard drive to back up the back up).
When I get home I then make DVD's of each folder (usually the folder contains about 4 gb of data which one dvd will hold). I then date (In the center circle of the dvd with a sharpie and then place in a plastic case)
After then I place on a Maxtor Drive with the main computer system and also then put all raw files on a second Maxtor Drive (I then do as Michael suggested and use IVIEW Pro to place each raw folder into a catalog so that I can view what is in each folder).
I do all above before I erase the images off of my notebook computer.

My main computer also has a raid 3 system.

This is where I am now and will continue to work on better systems. Once you get your own flow down it should become second nature.

FWIW

Larry

I don't trust my backups to a single medium. My backup workflow looks like this:

I download the images from the camera to the online disk on my computer. A copy of the RAW files gets made to CD immediately. When I'm done post-processing that batch, the processed files get burned to CD. The CDs go off-site. My primary disk gets duplicated to a local copy (another disk) which stays at home. This is my primary backup. It also gets duplicated to a THIRD disk which goes offsite with the CDs.

Why all of this complication? I want BOTH CDs and removeable hard drives containing my data off-site. This protects against media-related failure and a fire/flood at home. My onsite backup is essentially a mirror of my online storage. Making copies to CD also protects against the case in which I corrupt my primary disk in some way and then duplicate the corruption to the onsite and offisite disk drives.

This process is cumbersome, but I don't want to lose data. Ever . The main disadvantage of my approach (aside from the labor involved) is the fact that I always have to buy more disk in groups of three (primary, onsite, offiste). It also implies that I'm willing to move all of my image files from one medium to another as the years progress.

I'm a sysadmin and I've been messing with computers for a long time. I honestly don't trust any single medium enough to entrust all of my backups to it for the long term. I'd rather keep lots of copies. In the last 15 years, I've seen everything (disk, tape, optical, floppies, etc) fail at least once.

The downside to using NTFS, is that recovering deleted files is more difficult without professional recovery tools, since the system also features "secure deletion".

FAT and FAT32 on the other hand, make it easy to restore deleted files, unless they've been overwritten.

This has security implications.

Then it's more complicated only because you imagine it to be more complicated.

What you're describing above, is something that adds another step to the entire archival process of images, or to put it bluntly: you had greater complexity if you wanted digital copies before.

So, in reality, it's less complicated today than it used to be.

In addition, you can now make backup copies of your "slides" (RAW files), and those backups are identical to the originals. You couldn't do that before, and therefore your eggs were all in the same basket.

One of my aims for 2006 is to become more organised and do a complete tidy up of everything - studio, home cupboards, filing systems, backup files. I am amazed at how much 'stuff' gets kept in all those places - just because there is room for it. Hoarding sort of creeps up on one...and it has developed in the field of my digital files as well.

I have been asking myself with every item- "Do I need this?". If the answer is no - it goes, if the answer is yes- I ask myself why. Very interesting process, especially with all my folders of thousands of images. I was astounded at how many I kept - 'just in case one day....'.

Do I keep them because one day I might be able to edit them better? Do I keep them because I like them, and one day when I am bored I might feel like going through my backup files just to look at all my images? Do I keep them for our children as part of family history, so they can wade through the hundreds of folders one day when they are bored, to show their own children what life was like 'back then'? Do I keep them as a record of my artistic development as a professional artist?

A recent browse through some of my grandparents and parents family albums over Christmas with the family, made me decide on a couple of things. I have made a committment to go through ALL my photos and select ones for printing which I feel would be wothwhile to stand the test of time. It didn't matter that some of the photos from over 60 years ago were a little faded - they were readily accessable after such a long period of time...AND they did NOT require any technology to view them.

No updating was required, no remembering to update on different media every few years was required. No concern for viewing technology becoming obsolete. Just a good album and a clean, dry dark cupboard. If a flood or fire is going to damage an album, it probably would damage any electonic storage media at the same time as well.

So...apart from external hard drive, one of the ways I now back up - is in print - for both personal and professional reference ..... just for one of those days I am bored and want to go through and have a look at my images.

Julie

Some slight confusion in terminology above. Actually, I think your question is more about archival storage than backup.

To be technically correct, backup is a dynamic and ongoing process whereby in most cases the files being stored to hard disk are checked at regular intervals to see if there are new files in existing folders or new folders. These are then migrated to alternative media and the process is repeated on a regular basis.

Archival storage is movement of files to a media which can be expected to preserve the data for extended periods without the necessity of the user to check the integrity of these files on any systematic and regular interval.

Perhaps the safest medium for archival storage is magneto-optical.. This is a very expensive way but very reliable having true 100 year or greater life expectancy. Of course it would be naive to think that hardware to actually read the data would be available that far into the future so a means of migrating the archived files to more currently usable formats would be necessary.

For true dynamic backup, the most commonly used media today for commercial purposes is tape. There are a number of extremely reliable tape systems. Probably the least expensive of truly reliable formats today is DLT (Digital Linear Tape). DLT is extremely robust, has a storage life expectancy of around 35 years and comes in a variety of flavors. The leading edge drives such as Super DLT are quite expensive and have compressed capacities of as much as 600 gigabytes per tape (300 native) but older technology which is just as reliable and much less expensive can be purchased for around $300-400 for a drive with 40 gigabyte per tape native capacity. These drives will back up a 100 gigabyte hard disk in under 1 hour (not counting the additional and suggested step of checking the data integrity). Assume another hour per 100 gigabytes for the purpose of extreme reliability in data integrity.

Personally, I use Magneto-Optical for near-line storage of my critical files (available as quickly as I can change out a 9 gigabyte cartridge). I use LTO Ultrum tape drives with a 100 gigabyte uncompressed capacity for my backups. These drives are newer technology and though mine had an original cost of about $5000 can now be purchased for around $3000 or even less. As you can see, unless you are doing it professionally you would be better off with an older DLT drive which is equally reliable but has lower native capacity.

The software to do unattended backups is very inexpensive so for under $600 one can have a great backup system (you'll need a fast/wide SCSI interface card). This is an initial "hit" for many who paid less than this for their cameras, but necessary if you truly want to insure the preservation of your photos.

Lin

My main concern is archival.

I appreciate all of the feedback.

Bob

One of the key things NOT to do is write on the surface of an optical disk (CD/DVD) nor apply any stickers to it. Inks and glues slowly eat away the protective surface of the disk. I make paper labels for the disks and slip them into the transparent case.

There are also pens made specifically for writing on disks that aren't supposed to damage the disk over the long run (or at least do so much less than regular pens). You can order them from places like office supply places, amazon, etc. I use the Maxell's "Disk Writer" pens, but I'm sure there are other manufacturers too.

LIsa

Michael did a review of "Archive Creator", a tool for building data sets and subsets from the bulk storage directories. Anybody still using this? Comments?

Peter

One of the advantages of still using film is that you can always scan it again if you lose the digital file. I can still easily get my 14 year old slides scanned if I need to. No worries.

If you lose your RAW file, then that is it, gone. I only archive my TIF files from scanned slides twice, in two DVDs.

I archive my RAW files thrice: hard disk, and two DVDs.

But you can only have one copy of your film, and it can be lost or damaged too. Once that one copy is gone, it's gone. With digital, you can make as many backup copies as you want.

Film also fades and gets color casts over time. Digital never does.

Lisa