I would love to use S3, and I only have around 1TB of images at the moment.

But that's $150 per month.

An external 1TB USB drive costs a lot less than that to buy and can be "sneakernetted" to a friend's house or the office. Not as good, but a whole lot cheaper, though drives do need to be checked periodically.

Maybe S3 is a solution for the portfolio shots however. For many or even most of us we would probably be reasonably happy if we could save our best 100 images from a fire. At an average of 20Mb per image that's only around 2Gb or $0.30 per month - much better!

I think there is a market for online archives though, as opposed to online always-available storage, at a fraction of the cost. If my house burned down and my offsite USB drive wouldn't fire up, I would happily wait for a week to get my files back from an online archive service.

I agree - I don't want online 'backup' ... I want online archives.

I have working files on RAID 0, backups locally on RAID 5 and archives on a disk in the closet. I want the archive off-site and managed.

I looked at S3 and came to the conclusion that it was way too expensive for my needs.

In the end I opted for BackBlaze: $5 a month or $50 a year for unlimited storage.

As with all these things, initial upload can take months but thats about the only downside - it just works.
The advantage over sneakernet is you don't have to remember to do a backup & transport the disk every few months.

That was an interesting article, but agree that to store 1TB of data at Amazon is too expensive. For that price, I can buy lots of hard drives for local backups. I generally keep two backup disk sets using 1TB or 1.5TB drives. One backup package is stored at my cabin in the mountains, while the other is kept in a hot-swap SATA bay on the front of my PC, continuously backing up my main data drive that is in RAID 1. Every month or two, I swap disk sets.

All of my data is simultaneously on 3 or 4 drives. If my house burns down, the most I could lose would be a month of data. If my house burns down, honestly a couple of weeks of photos will be a minor issue. (OCD pros will disagree with me.)

Here is the setup:

* Active Internal 1TB Drive: Archive of RAW files from prior years (2002, 2003....2008)
* Active Internal RAID 1 Array of 1.5TB Drives: Current year RAW archive, working directories, and Final Copy directories for all years.
* Removable 1TB Hard Drive in Drive-bay: Daily backup of Archive of RAW files and Current Year
* Removable 1TB Bare Drive stored offsite: Less than 30 day old Backup of Archive of RAW Files and Current Year

Using Amazon for Final Copies may be an interesting idea. I will need to look into that. Otherwise, I think I am covered. All data is on a minimum of three drives at all times. One of those copies is offsite.

I have considered NAS systems like Drobo, but I am not enthusiastic about proprietary systems like that, and I don't see how it would be an improvement over what I am doing.

What I do for my backups is pretty "primitive". I use LR as my database, and put all my files into numbered hds and then carbon copy them into another disk the same size and put in a safe. Works like a charm. But, Geoff's article made me paranoid. What can I do to make sure that my backup is Ok? Technology definitely DOESN'T make life simpler. I find shooting digital liberating in some aspects, but not in others. Converting my images into a zillion 1s and 0s doesn't make it easier.

I use Mozy for my backups. But with the connection I have it takes a while to back things up. If I wasn't a photo packrat this wouldn't be a problem.

I also backup to an extra internal HD and several external HDs.

Edit: As an aside a metadata aware backup application would be great. I really only need to backup images with a rating of 2 stars and above.

$8K raid setup, or $300 monthly at S3 for my 2 TB?
This solution is for somone in a different tax bracket than I'm in.
I need some creative solutions for regular folks. Currently I keep 2 off-site external drives, one full backup, one just raw and jpg backup. my home computer is no raid, regular SATA hard drives, plus an external USB drive backup.
Who can afford to keep updating systems with costs like those mentioned in the article?
Not me.

One option I have not heard mentioned anywhere, you may already have a lot of online storage space that you are already paying for. If you have a website, you likely have gigabytes of unused storage on your account, simply ftp files to a directory on your account, possibly encrypting them before uploading. Might be a good solution for 5 star images.

You mean not everyone on this forum shoots with a Linhof 679CS + P45, and they're not *gasp* well-heeled (former) executives at Fortune 500 companies? LOL.

Shoot! And to think, I thought I needed at least $50k in gear just to call myself a photographer.

after years of messing around with mozy, elephant trunk and others i finally found crashplan....and it really works....the biggestt advantage is that it not only allows you to put everything in the cloud, but also to sync the data between remote computers.....
i have a fios with 20mbps upload and crashplan actually uses most of it....all others cap the upload speed...mozy with 140kbp is a joke....
i have 2TB in the cloud right now and keep adding....price is 50$/year unlimited storage, no cap on upload and the backup software is rocksolid in 10.6.1.....can't beat that....

We recently configured a similar promise cabinet for a medical imaging application (we used SAS) - I can attest to the build quality and ease of use. Only downside is noise, these things are LOUD.

I get up to 700kbps with Mozy. What speed are you getting to crashplan?

I found a much easier solution to RAID. What the article fails to mention is that when you want to add more storage to the array, you need to destroy all the data (so really you need to move it all to another system). I bought a Windows Home Server, it is not a true RAID system but it uses a software solution to ensure that two copies of you data is stored on separate drives and you can simply add a new drive to the system and add it to the pool in order to increase your storage space. It was very easy to set up and has a great interface plus lots of other useful features. I started with a 640gb and 400gb drive and have since added 500gb and 1tb drives to the pool. With a gigabit network switch the file transfer from the WHS is as fast as keeping everything on the internal drive of my Mac.

I only keep about 90gb of files on Amazon S3, about 60gb of photos and 30gb of videos - it's a good solution if you don't store massive amounts of data but it could get expensive if I needed more space. Hopefully prices will come down over time.

when i tried mozy the cheap version had a capped upload speed, the business solution is not capped but much more expensive....crashplan does not cap the upload....i get almost 1mb/sec upload....they also have an option to send them a harddrive with data to avoid the initial upload.....
i have about 2TB images, some video, my itunes and my aperture library (which was a problem with several services...) no problem at all, everything gets updated every day and new shoots get uploaded overnight.....

i had looked at them before but they did not offer online backup....only computer to computer....which is great as well btw....and i might put a raid box somewhere else just to make sure....this is easier of course because the initial backup can be done local, the box moved and crashplan takes care of the updates....

i am in no way affiliated but it just works....

i almost signed up with photoshelter (1TB for 1000$/year which also includes a lot of stuff i don't need, website,...) i tried that and the upload speed was insane...i got 2mb/sec....but the price was too much and crashplan really works for me....

This topic has me considering going back to film for important work. At least for medium format. I probably have some 20, 30 or more tb in positives and negatives (not including my fathers Kodachromes from the late 1930s or 616/626/120/sheet films from earlier times both here and in Europe)!

A very long hmmmmmm.

I would not want to pass my storage to a cloud - it might rain!

my film is actually much harder to file and keep safe...takes up a ton of room and is (obviously) always only in one place...fire, water, theft....and just storage (humidity, temperature,...) who can afford an actual proper film archive?

I am interested in other users' experiences with Crashplan & Backblaze. Amazon's S3 is amazing, and you know the the company is not going anywhere; but again, pricey, if you are dealing with large amounts of data. 4TB would cost $600/month just for the storage fees.

So, have other users out there had good success with multi-terabyte amounts of data and online services?


Andrew.

That article was cursory at best, ill-informed at worst. Not to mention missing the price bracket for most people.

The article seemed to imply that RAID is a backup solution. It's not. RAID should be used for uptime and/or speed (depending on the mode). Sure there's redundancy, but that should not be confused with reliability. Your RAID card is toast, your data is likely toast. User error? Same result. Theft, fire, water damage? Toast.

S3 and comparable services are a rich man's offsite solution. Most of us have terabytes of data to backup, and using those services costs hundreds of dollars per month. Just put your data on external hard disks or tape, and take it to the office or a friend.

As for Windows Home Server, that's an even worse backup solution. It shipped with a crippling data corruption bug which MS took quite a while to acknowledge, and several months to fix. It appears to be stable now, but again, it's not backup as it shares all of the shortcomings of RAID for backup use.

Depending on the value of the data and level of paranoia, you should have at least two different backups, at least one of them off-site. You should test that you can actually recover the data periodically; CRC check is not enough, actually try to recover the data. And you should occasionally take snapshots of your data, instead of only using incremental backups.

I didn't see any implication that RAID was being used as backup in the article. You really should look at the Promise cabinet; it goes far beyond any consumer RAID solution. The individual unit I worked with contains 4 independant SAS channels connected via two separate controllers. Failure of up to 3 of the cannels will slow down data transfer but will not affect operations:

http://www.promise.com/upload/datasheet/1_...0s_DS112807.pdf

Windows home server is just another implementation of dynamic disk capabilities starting Server 2003 (with a *nice* interface) - like any RAID solution, there are bad, better and best practices:

http://support.microsoft.com/kb/816307

I agree with your conclusion regarding Amazon's data backup pricing, but one point missed, is reliability: these folks know how to warehouse data! I probably wouldn't store *every* file in my possession there, but certainly the inportant ones....

I don't see the need for a RAID,you cannot keep adding new drives, they are all kept running and so more likely to fail, if you want to work between 2 locations you have to cart the whole thing around and all disks must be replaced when full and so are a pain to bring back online unless you have another cabinet.
Drives are getting larger and cheaper, I use incremental daily back up onto 2 internal drives for live work which then gets copied onto 2 drives in a JBOD SATA sonnet cabinet and stored 1 at home and 1 in the studio. iView media pro is my thumbnail database so an image is easily found, the catalogued off line drive can be quickly located and slotted into the cabinet.

All these spinning discs in Raids, failing, burning up power, noisy, hot, no need.

I think there is a lot of misinformation about RAID. Earlier in this thread someone said that if your RAID card goes bad, your data is unrecoverable. That isn't true of all implementations. Another person said that you need to tear down and recreate your RAID array if you want to add a drive for increasing size. That, too, isn't true for all implementations. It is true that older RAID systems had both these faults, but these issues have been addressed.

RAID is a practical every-day solution for redundancy so that your backups don't have to be done hourly. If you want to work from a laptop, they aren't practical, but doing all your photo work from a laptop isn't a very practical solution anyway (i'm not even going to go into it).

One suggestion, that I haven't heard mentioned is to make sure to replace hard drives before they get too old. It is kind of wasteful, but I rarely have trouble finding someone to take and use my old drives for USB drives or something similar. Another use for old hard drives is to put them into RAID 0 as a fast scratch disk. I would say that I rarely keep a data drive more than 12-14 months.

I agree that the Amazon service pricing is a bit out of reach for most of. I am not sure that whether I need to use it in addition to a decent backup and recovery plan that includes offsite storage.

I didn't say such thing. I'm aware of modern RAID capabilities, but that doesn't change the fact that RAID is not backup.



For those of us who don't like throwing good money away, it's much cheaper to use RAID for redundancy and speed, and rotate two or more copies of backups until the drives start failing. And replace a drive at the first sign of failure.

For those of you considering fike's suggestion of putting old drives into a RAID 0, ensure it is indeed a scratch disk with nothing critical on it. If one of your drives fails in RAID 0, all data in the whole array is hosed for good. In other words, the failure rate increases with the number of drives in the striped array. And if you have old likely-to-fail drives in the array, that's a recipe for disaster.

Such old drives should be put into a scratch disk striped array, or a redundant RAID array so the failure is not catastrophic.



I'd like to raise a very recent Microsoft subsidiary Sidekick debacle. Latest news is that they might be able to recover "most, if not all" data, but lesson learned: catastrophic failures do happen to even the most experienced people and companies.


As recap: I'm not saying not to use clouds or RAID, but don't put your eggs in one basket. Any one system is not enough to protect your priceless data. A friend of mine lost years of photographs, and doesn't have a single photo of his son growing up. Data backups should be taken seriously, as it is easy to ignore and forget, but recovery is often impossible without proper backups.

I'd like to raise a very recent Microsoft subsidiary Sidekick debacle. Latest news is that they might be able to recover "most, if not all" data, but lesson learned: catastrophic failures do happen to even the most experienced people and companies.
[/quote]

We'll probably never know *exactly* what happened to the data managed by MS's recently purchased Danger unit - but suffice to say, there are bad, better and best practices to everything, including Data Warehousing.

IMHO I see fairly robust competition ahead for cloud backup and data services; Amazon S3 http://aws.amazon.com/s3/ , Microsoft Azure http://www.microsoft.com/windowsazure/ and potentially Google (no formally defined services covering our needs - but here is what they currently offer http://code.google.com/apis/gdata/ ) All of these providers offer geographically dispersed data storage, no eggs in one basket (data center) ala Danger.....

Interestingly the most open about methodology is Microsoft. Here's one specific example

http://blogs.msdn.com/olavt/archive/2009/0...ial-images.aspx

This was a timely article as my website host apparently disallows uploading backup files to them (I even paid more for the extra space, but they don't want my money apparently).

3 GB of data is a slow week here... much of that is working data though, PSDs etc and not delivered selects. These are backed up daily across the network. I work in an aging studio though and flood or fire is always in the back of my mind. So I had been storing business data, and extensive keeper image portfolio (LZW compressed tiffs) encrypted off-site on my web host. Currently around 30 GB of data.

I'm now using IDrive for this - $50 / yr for 150 gb. Transfer rate is holding steady at 472 kb/s, which I believe is about all my connection can achieve. 2 mb/s would be nice but not available here. Mozy is technically unlimited, but I've heard the transfer rates can be spotty.

S3 seems a little expensive for equal service and is more complicated to use.

One worry I still have though is a fire that wipes out the current job being worked on. Insurance will cover the monetary damage, but there's always reputation damage and the opportunity cost from the lost images. Given the slow connection speed it's not realistic to upload the post-production dailies offsite, but perhaps I will schedule a backup of raw selects from each job everyday, so at least the raw files are protected.

For fast off-line storage an external disk will be a cheap and easy solution.

Either a 2.5" usb drive, I've seen 500GB for €75 and only weigthing 170 gr, easy to take with you and store in you're car or another place. Even a usb stick could be big enough and those are very small.

You could use 3.5" drives which can hold up to 2 Tb, given the size a esata docking is very nice to have. A lot heavier and I've seen 1.5 TB samsung drives for €89, but those weigth 650 gr without an enclosure. I use those last one's for my primary backup. One is always stored off site.

I use http://www.backblaze.com

It's 5/month, unlimited usage and works on both windows and mac. It took me about 2 weeks to get all 500GB of my data transfered but man its so much cheaper. If you need to restore you have 3 different options.

By far, the most likely event that will cause you to lose data is a drive failure. RAID and WHS type data redundancy is a good way to insure yourself from a drive failure because your data is duplicated as soon as it is stored on the system and if you do get a drive failure you can recover from it without having to download all your data off the internet.

This won't protect you from the worst case scenario though, if your house/studio burns down then RAID isn't going to save your data. You need an offsite solution to protect against this, the cheapest way to do this is to have an external drive that you backup to. The problem I found with this is that I usually created a backup every month so a lot of data goes unprotected for a long period. You could also have a failure in your backup drive which could go undetected until you need that data.

My WHS is set to run a backup at a certain time every day so new data does not sit on the system long until it is backed up off site. A professional service should protect your data from a drive failure by continually checking the data that is stored on it's servers and resolving any issues it finds.

Thanks for recommending Crashplan, I'm going to move from S3 to it soon.