Help me devise a backup strategy
|
| Author | Content |
|---|---|
| Sander_Marechal Mar 30, 2007 1:21 AM EDT |
I am in need of a good backup strategy for the data on my server(s). I have taken care of the destop sides. When a PC shuts down it rsyncs /home and some other essential files to my server. Once a week the server creates a GPG encrypted tar.gz (about 800 Mb) and sends it to my friend's server. He does the same thing on my end. However, for the server data this doesn't work. There's too much of it. I have about 40 Gb of stuff on my server that I need to backup. My friend has about 100 Gb of data. On average the datasets grown by 1-2 Gb but it fluctuates (10 Gb in one week, then a couple of weeks nothing, etcetera). There's a 1024 kbps internet connection between my server and my friend's server. And to boot, neither me nor my friend currently have enough harddrive space in our servers to store both datasets. I am looking for a strategy that will minimize costs as well as manual labour. We're going to *have* to buy new hardware for this. Whether that's a NAS box, new, bigger SCSI drives, whatever (sadly, digging in a fiber line from my house to my friend's house isn't possible :-) I would also love for the backup to be automated because you all know how easily you forget making manual backups. Any advice? The best we can come up with at the moment is buying a NAS an lugging it around, or rsyncing the weekly changes and set a QoS on the routers so the 48-72 hour rsync doesn't interfere with regular internet usage too much. |
| Scott_Ruecker Mar 30, 2007 1:40 AM EDT |
Have you thought about hiring another company to store it for you temporarily off-site until you get the hardware you need? Using other peoples storage capacity, even for the short term is getting cheaper by the day. For example, you could set it up so that once a day (during an average slow time) your servers could call out to the storage facility and upload the newest changes/additions. The initial upload would be big but after that it would only be what is new or different which would not take nearly as long. I am assuming that you have high speed access. Hope I helped. |
| Sander_Marechal Mar 30, 2007 1:47 AM EDT |
Haven't thought about that yet. One problem would be the initial upload. Assuming approx. 10% network and OS overhead, it would take about 350 hours, over two weeks, to transfer. The weekly updates would take about 3-24 hours to transfer. That's a lot :-/ Oh how I wish I had 100 Mbps up and down to the internet. |
| Scott_Ruecker Mar 30, 2007 1:58 AM EDT |
The other option would be to rent the storage hardware needed to give you the room you need. It depends on how desperate your storage problem is right now. If it is hitting critical then rent what will get you through and then start buying servers. Is there any way to condense the data? |
| Sander_Marechal Mar 30, 2007 2:59 AM EDT |
All the data is already compressed. We're not desperate for storage at the moment. Combined there's about 30-40 Gb left, and if it does get desperate we can always put in an 80 Gb IDE for extra storage. The problem is that there simply aren't any backups. The 40 Gb volume is reasonably safe sitting on RAID5 so I can afford to have one drive die, but the 100 Gb volume is all on a single IDE disk. If it dies then the data is gone. |
| Sander_Marechal Mar 30, 2007 3:18 AM EDT |
Out of curiosity I went to newegg and had a look at the prices for stuff that's compatible with my server. Not funny :-( A 36.4 Gb harddrive costs between $150-$250. Larger sizes go up to $399. I would need at least two of 200 Gb. Alternatively, there's an Ultium tape drive in the system. A 200 GB tape costs about $30. That's not bad, but another tapedrive (either for the other server, or in order to read the tape if the server has burned to a crisp) costs well over $1500,- And that's excluding the 18% VAT I'd need to pay from the US to The Netherlands. It's all nice and well buying decommissioned server equipment, but replacement parts are expensive! If more than one drive dies I'm cheaper off buying a whitebox server than trying to replace the HP ProLiant parts :-( |
| Scott_Ruecker Mar 30, 2007 3:50 AM EDT |
And with all that I have said I should have gone with my gut in responding to your call for advice.. Get some older Pentium or AMD white boxes for cheap and then spend some good money on battery back-up. You will be able to add hard drives to them for extra storage and parts or whole "new to you" boxes are cheap and cheaper |
| cyber_rigger Mar 30, 2007 6:22 AM EDT |
Lacie makes a 500GB external usb drive for about $200 (also NAS, SATA and firewire). I've been happy so far. It does require its own power supply. http://www.pricescan.com/01140106.asp [url=http://www.pricegrabber.com/search_attrib.php/page_id=11/form_keyword=lacie 500/rd=1]http://www.pricegrabber.com/search_attrib.php/page_id=11/for...[/url] |
| jdixon Mar 30, 2007 7:51 AM EDT |
Well, 500 GB SATA hard drives are in $125-150 range now. Two of them set up in a raid would probably meet your back up needs with no problems. 250 GB drives are almost exactly half that price, and they might do. I recently priced a barebone Sempron system with a Biostar motherboard for Rat's. Half my post got cut off. Anyway, condensing: A home built system will run you $400 or so for 250 MB raid or $600 for 500 GB raid. Western Digital sells a 1TB system non-raid NAS box which can apparently be configured as a 500 GB raid box for just over $400. |
| Sander_Marechal Mar 30, 2007 2:19 PM EDT |
Yep, NAS it's going to be. Anything else is just too expensive. I'm probably going to build it myself so I can have RAID as well. |
| jdixon Mar 30, 2007 3:37 PM EDT |
Here's a link to a US source for the Western Digital unit: http://www.mwave.com/mwave/skusearch.hmx?SCriteria=4293427&C... I'm sure you can find other sources at an equivalent price. |
| Sander_Marechal Mar 30, 2007 4:00 PM EDT |
That one looks nice. User servicable. Possibility for RAID1. Gigabit ethernet. Pretty much all I want. The reviews on newegg aren't too positive though, citing firmware troubles. |
| jdixon Mar 30, 2007 8:32 PM EDT |
> The reviews on newegg aren't too positive though, citing firmware troubles. Ouch. I've found the reviews on NewEgg to fairly reliable. There were a number of other units available, it's just that this one looked like a good match for your needs. :( |
| jimf Mar 30, 2007 9:17 PM EDT |
Not sure this one's a match, but it sure is Linux friendly and gets great reviews. http://www.newegg.com/Product/Product.aspx?Item=N82E16833124... |
| jdixon Mar 30, 2007 9:34 PM EDT |
The Lynksys looks OK, but it's only for USB drives, which I'd expect to be quite a bit slower than either IDE or SATA drives, and it doesn't include the drives. It also doesn't look like it supports raid. Still, it may meet Sander's needs. |
| jdixon Mar 30, 2007 10:32 PM EDT |
Sander: Extreme tech has a review of the Western Digital unit here: http://www.extremetech.com/article2/0,1697,2106424,00.asp They're quite a bit more favorable than the web based reviews I found. Their main complaint was one of the ones mentioned in the web reviews: The fan is extremely loud. As they noted, it should be easy enough to replace the fan though. They did not report the network problems the various web reviews listed. I'm wondering if the GB ethernet on the WD might not have compatibility problems with some networks/routers. I've seen that before with various network interfaces, and playing with the speed/duplex settings usually cures the problems. Except for the speed/noise problems, the unit seems to work fine, and it seems it runs Linux, so you may be able to modify it. I think I'd give it a try if you can guarantee you can return it for a full refund. |
| jimf Mar 30, 2007 10:33 PM EDT |
Yeah, I saw all that, but, it will take two (cheap) drives, and the reviews rave about it. |
| Sander_Marechal Mar 31, 2007 12:08 AM EDT |
jimf: That looks very nice. Plus you can upload your own firmware which is a definite plus for me. jdixon: Yesterday morining the 500GB Western Digital wal also on newegg (it wasn't there anymore last night). One of those reviews said that the Debian-based firmware isn't hackable. Too bad it disappeared from newegg :-/ Good stuff people. Thanks :-) |
| ggarron Apr 03, 2007 5:43 AM EDT |
I think you can do a cross backup on your same network (buy a cheap old PC), using rsync, or use mondo backup (too much disk switchings unless you buy a new harddisk and send the image to it). Here are some links Mondo Backup http://linux.go2linux.org/node/32 using Rsync http://linux.go2linux.org/node/37 using duplicity http://linux.go2linux.org/node/41 And backing up a VMware machine http://linux.go2linux.org/node/17 Hope could be usefull |
| devnet Apr 03, 2007 5:52 AM EDT |
I'd go external hard drive with bacula
http://www.bacula.org/ Works quite well for our 30GB of Music and our 100GB of Movies. |
You cannot post until you login.