I made this thread a long time ago on the BSDnexus forums. I was thinking about this tonight thanks to lucas's rsync thread, so I thought I'd make a thread where everyone could share.
how do you accomplish data redundancy or backup or whatever?
my backup location is hard drive partition wd1a on my router/server. it's mounted at /home/brain/files (maybe I should change this to /backup; I forget why I didn't do that in the first place.) and receives backups both locally and from my MacBook over wifi.
local backups of ~/documents, /var/www/users (user web directories), /var/www/htdocs, and /root/bin are made via a shell script run daily by cron.
MacBook backups of ~/Desktop ~/Documents ~/Library ~/Movies ~/Music ~/Pictures, and ~/Software are made via a shell script run daily by anacron (anacron rules on laptops because they're not on 24 hours a day; read about it if you have a laptop and want to schedule tasks!).
I had to set up ssh without a password for this script to work when run in the background such as by anacron. I don't know how much of a security issue this is, but oh well.
ok, I think that's it. what do you do?
how do you accomplish data redundancy or backup or whatever?
my backup location is hard drive partition wd1a on my router/server. it's mounted at /home/brain/files (maybe I should change this to /backup; I forget why I didn't do that in the first place.) and receives backups both locally and from my MacBook over wifi.
local backups of ~/documents, /var/www/users (user web directories), /var/www/htdocs, and /root/bin are made via a shell script run daily by cron.
#!/bin/sh rsync --delete -avz /home/brain/documents /var/www/users /var/www/htdocs /root/bin /home/brain/files/crashoverride/
MacBook backups of ~/Desktop ~/Documents ~/Library ~/Movies ~/Music ~/Pictures, and ~/Software are made via a shell script run daily by anacron (anacron rules on laptops because they're not on 24 hours a day; read about it if you have a laptop and want to schedule tasks!).
#!/bin/sh rsync -e 'ssh -p 23' --delete -avz /Users/brain/Desktop /Users/brain/Documents /Users/brain/Library /Users/brain/Movies /Users/brain/Music /Users/brain/Pictures /Users/brain/Software brain@crashoverride:/home/brain/files/cerealkiller/
I had to set up ssh without a password for this script to work when run in the background such as by anacron. I don't know how much of a security issue this is, but oh well.
ok, I think that's it. what do you do?
wow.. i wish i could back up my shit wirelessly. i desparately need to rethink my organization especially of my photos. Right now I only have one copy of my photos on an external hd, and I have some important photos copied on my macbook hard drive. can someone help me out? I really need it. I just bought another 300gb external that I have not touched yet. open for suggestions :D
tape drives!
foldershare -> raid 1 on desktop
is your MacBook drive too small for all of the photos, andyp?
if so, I'd invest in another external or upgrade the drive in your MacBook.
if so, I'd invest in another external or upgrade the drive in your MacBook.
yeah its an 80gb hd. and I'm constantly throwing shit onto my external hd to make room to import new photos. I have 10gb on deck for that, but with my new camera it gets eaten up quick. I'm more worried about how i'm going to deal with my new blank external hd. is there a way to set it up to automatically back everything up every time i plug it in?
Just get an enclosure. Firewire is the way to go.
I'm not sure of any way to automatically back up when you plug it in, but I have a feeling there's gotta be a program out there that does that.
it would be tough having two externals that you want to mirror unless you got a case designed to do that.
this might be overkill, but you could set up a computer somewhere in the house as a backup server type thing. this would make more sense if you already had a computer set up that was acting as a router or a server.
it would be tough having two externals that you want to mirror unless you got a case designed to do that.
this might be overkill, but you could set up a computer somewhere in the house as a backup server type thing. this would make more sense if you already had a computer set up that was acting as a router or a server.
assuming it's windows, you could just do an autorun script on the external.
i read today about someone who lost their 400gb music collection - it was on an external hd.
that is so terrifying. i need backup... when i come home i'm doing raid 1 again
that is so terrifying. i need backup... when i come home i'm doing raid 1 again
s3
s3
s3
s3
s3
that's why you never store things you care about in only one place.
i can't s3 100gigs, can i?
of course!
i currently store about 15gb, for a cost of about $2/month.
i currently store about 15gb, for a cost of about $2/month.
get one of these external enclosures--everyone loves them:
http://www.newegg.com/Product/Product.aspx?It … 6817173010
http://www.newegg.com/Product/Product.aspx?It … 6817173010
I haven't been doing backups lately, but I've got a new network setup at home. my router/firewall is now a smaller, less powerful box with only one hard drive. I still have my old router/firewall box so I'm going to use it as a file server or something else cool (suggestions?). I'll use it for backups too; right now I have a problem with NFS so I've gotta work that out first. then I'll probably do something similar to what I was doing before.
added another 750gb to my workstation yesterday
i'm making a raid 1 array with two 200 GB disks for backup, possibly encrypted
my hd setup:
2x250 gb raid 1 (wd2500ks)
750gb (wd7500)
80gb raptor (wd800adfd)
all sata2
2x250 gb raid 1 (wd2500ks)
750gb (wd7500)
80gb raptor (wd800adfd)
all sata2
fileserver:
maxtor pata disks:
300 GB for root, home &c
160 GB for incoming torrents
2x200 GB
samsung sata disks:
2x1 TB for music, movies &c. these have ffs2 filesystems
i've got room for 8 more sata disks
maxtor pata disks:
300 GB for root, home &c
160 GB for incoming torrents
2x200 GB
samsung sata disks:
2x1 TB for music, movies &c. these have ffs2 filesystems
i've got room for 8 more sata disks
==================================================================== wlw:~/svn/classes s3:lucas-vc every 6h wlw:~/svn/journal s3:lucas-vc every 6h wlw:~/svn/larz s3:lucas-vc every 6h wlw:~/svn/ttf s3:ttf-vc every 6h wlw:~/svn/wlw s3:wlw-vc every 6h mysql@wlw:ttf s3:ttf-db, ttfbackup@gmail.com every 6h mysql@wlw:wlw s3:wlw-db every 6h wlw s3:wlw, ext_hdd on demand ~\private s3:lucas, ext_hdd on demand D:\private s3:lucas, ext_hdd on demand D:\ap ext_hdd on demand D:\music ext_hdd on demand D:\programs ext_hdd on demand ====================================================================
I installed Leopard and have been doing backups via Time Machine lately. my firewall has not been getting backed up though :/. I should make a cron job to back up my pf.conf at least.
I don't think I've done a backup of my workstation since the last time I posted in this thread. pretty stupid. I think I'm going to return to using a tower-size machine for my routing and backup needs and maybe sell my nettop computer that I've been using as a router. I can't justify the electricity usage of having the nettop do routing and the other machine just do backups.
just bought two 500 GB drives from newegg. not sure if I'll do RAID or just keep one as a backup drive in the case of hardware failure. I thought about upgrading my entire router/server box with a new mobo/cpu and stuff but decided against it. not worth the money. the only problem is that it's going to be hard to find non-SATA hard drives pretty soon.
i think 500gb hdds are a good size to go with. the 1tb hdds don't seem to have a good review average.
i'm disappointed that i went with 250gb hdds when i started on my setup. luckily it's a very reliable drive and it's pretty damn cheap.
i'm disappointed that i went with 250gb hdds when i started on my setup. luckily it's a very reliable drive and it's pretty damn cheap.
the only reason I didn't go for a TB is because there were no non-SATA TB drives.
Since rdiff-backup is at different versions on some of the machines I use, and not installed at all on others, here's my new solution:
1. Mirror files from all my machines to their per-machine directories on atlas.
2. Run rdiff-backup on the mirrors on atlas to another backup directory on atlas for versioning.
3. Goto 1.
1. Mirror files from all my machines to their per-machine directories on atlas.
2. Run rdiff-backup on the mirrors on atlas to another backup directory on atlas for versioning.
3. Goto 1.
My new backup solution.
We are all going to die. The earth will burn. Our star will collapse. Our galaxy will be consumed by itself. The universe will go dark and die in heat death.
We are all going to die. The earth will burn. Our star will collapse. Our galaxy will be consumed by itself. The universe will go dark and die in heat death.
I just use subversion for backing up stuff, it's easy to set up, easy to manage, widespread, and keeps previous versions.
In the past I used cvs, which also works OK except that it can mangle binary data.
In the past I used cvs, which also works OK except that it can mangle binary data.
Why not back up the universe?
But brain, where will we find rubber pants that size?
> atlas
my atlas?
<3
my atlas?
<3
Yes, your atlas. Also, OH SHIT I FORGOT TO REALLY IMPLEMENT IT!!! :O
working on a new one of these now. just made a list of all the data stored in various places. I'm thinking about storing encrypted tar archives on amazon s3 to have off-site backup of certain data like my Documents and Pictures directories. I'll obviously exclude all my mp3s and stuff because that would be way too much disk space to pay for and I don't care that much about them.
amazon is pricy there are better backup centric alternatives.
so I've heard. tarsnap is more expensive, but looks pretty cool.
But it uses Amazon S3 itself. I'd cut out the middle layer.
yeah, I didn't notice that until carpetsmoker pointed it out on twitter.
backing up all the configs and important data from my server to s3 now via a cron job! weekly.
I've been jumping os's and storage/backup solutions so much lately that I'm sitting here next to two disassembled computers, one running openbsd from a usb stick and copying files from a sata drive (the sole residence of my documents, photos, music, etc.) to an external usb hdd.
I think I'm going to run freebsd as a file server. two 500 gb ide drives and a 1tb sata drive. os and file shares on the tb drive and copies of everything on both of the 500gb drives.
I think I'm going to run freebsd as a file server. two 500 gb ide drives and a 1tb sata drive. os and file shares on the tb drive and copies of everything on both of the 500gb drives.
I've been doing hourly backups for nearly three months now. It finally filled my drive. What to delete? :O
three-month old backups?
i think atlas has some room for your crazy backups (then delete it locally)
or if they're mission critical, put them on aws s3 (then delete locally)
or if they're mission critical, put them on aws s3 (then delete locally)
speaking of atlas, I should back up to there. same with s3. I did that for about a month.
I'm thinking I'll just keep monthly snapshots for stuff that's more than a month old.
backing up to atlas now!
is scp'ing a gigabyte of data per week ok on atlas?
of course
if you're happy with the speeds (15Mbps/2Mbps), go for it
if you're happy with the speeds (15Mbps/2Mbps), go for it
cool! yeah, it took somewhere around an hour going 200-250 kB/s. I can live with that. it's automated anyway, and I just need at least one copy of important stuff stored away from home.
n.b. availability will be spotty at best as i relocate to santa barbara in four weeks
no problem; I'll probably have s3 backups running by then.
well, what i stress is that while it isn't always available, if you put something there, it is reliably archived. you might have to wait a bit to get it back, but you'll get it. that's the promise of larz and two enterprise disks in sync.
i'm also willing to do imports by sending me a usb drive. i can also have a 160 GB usb disk drive to which i am willing to export and ship to you.
also, while s3 is great for highly-available archives, it's expensive for retention of large amounts of data. accordingly, if you have 50 GB to archive and it need not be highly available, send it to atlas. that's why i have atlas in the first place.
i'm also willing to do imports by sending me a usb drive. i can also have a 160 GB usb disk drive to which i am willing to export and ship to you.
also, while s3 is great for highly-available archives, it's expensive for retention of large amounts of data. accordingly, if you have 50 GB to archive and it need not be highly available, send it to atlas. that's why i have atlas in the first place.
good pitch! that is basically what I need in a backup server. I already have my data on three disks, two of them in a raid-1 array, so I really only need access to the offsite backup in the event of something major like my house burning to the ground.
weekly tar/7zip/aes files of important files scp'ed to atlas (thanks, lucas!).
and now I'm doing hourly backups to an external drive with rsync. the logic in that script is kind of fucked and doesn't make it easy to adjust for, say, daily backups instead of hourly. and also requires more and more conditions for weekly/monthly/yearly backups. but it's working and taking up like zero disk space thanks to --link-dest.
and now I'm doing hourly backups to an external drive with rsync. the logic in that script is kind of fucked and doesn't make it easy to adjust for, say, daily backups instead of hourly. and also requires more and more conditions for weekly/monthly/yearly backups. but it's working and taking up like zero disk space thanks to --link-dest.
no problem! wow, atlas still has 732 GiB available.
hahahaha
that was my backup strategy for a long time! and it worked out quite well, luckily.
that was my backup strategy for a long time! and it worked out quite well, luckily.
That's also been mine since my backup drive started failing. :(
is that supposed to point to the "do nothing" section?
bluet: not into atlas backups anymore? i'm glad asemi is getting use out of it... :)
bluet: not into atlas backups anymore? i'm glad asemi is getting use out of it... :)
> is that supposed to point to the "do nothing" section?
Yes.
> bluet: not into atlas backups anymore? i'm glad asemi is getting use out of it... :)
I've got plans, I just need to get a circular tuit.
Yes.
> bluet: not into atlas backups anymore? i'm glad asemi is getting use out of it... :)
I've got plans, I just need to get a circular tuit.
look at this! cold storage.
i've bought a blu-ray writer and some discs that i've been meaning to back up to. write once, read many! :D
that's cool. i think tapes would be fun. probably more fun than useful.
tapes are cool, for sure. but they probably only make sense at huge scales where the cost of the equipment is dwarfed by the cost of media
blu-rays are good for distributing to your friends, for dirt cheap off-site backups
blu-rays are good for distributing to your friends, for dirt cheap off-site backups
damn, now you've made me want to make a reel-to-reel storage system for my pc
i wonder how many bits you can store on one of these: http://www.ebay.co.uk/itm/TEAC-X10R-AUTO-REVE … 2506203052
i wonder how many bits you can store on one of these: http://www.ebay.co.uk/itm/TEAC-X10R-AUTO-REVE … 2506203052
haha yeah!! we had a reel-to-reel just like that at the radio station. used to all the time for temporary recording
ideally you'd take data, run it through symmetric encryption, and somehow encode it and lace it into audio. record this on a bunch of tapes. then you have plausible deniability.
this is coolest so far: punched tape!
http://www.ebay.com/itm/FACIT-4070-DESKTOP-PR … 2142506899
https://www.youtube.com/watch?v=61uG-y0e3wE
http://www.ebay.com/itm/FACIT-4070-DESKTOP-PR … 2142506899
https://www.youtube.com/watch?v=61uG-y0e3wE
Dunno ... Don't think the tech is quite ready yet. Maybe in two or three years? It's definitely the future though!