think tank forum

technology » backup solutions

asemisldkfj's avatar
12 years ago
link
asemisldkfj
the law is no protection
I made this thread a long time ago on the BSDnexus forums. I was thinking about this tonight thanks to lucas's rsync thread, so I thought I'd make a thread where everyone could share.

how do you accomplish data redundancy or backup or whatever?

my backup location is hard drive partition wd1a on my router/server. it's mounted at /home/brain/files (maybe I should change this to /backup; I forget why I didn't do that in the first place.) and receives backups both locally and from my MacBook over wifi.

local backups of ~/documents, /var/www/users (user web directories), /var/www/htdocs, and /root/bin are made via a shell script run daily by cron.

#!/bin/sh

rsync --delete -avz /home/brain/documents /var/www/users /var/www/htdocs /root/bin /home/brain/files/crashoverride/


MacBook backups of ~/Desktop ~/Documents ~/Library ~/Movies ~/Music ~/Pictures, and ~/Software are made via a shell script run daily by anacron (anacron rules on laptops because they're not on 24 hours a day; read about it if you have a laptop and want to schedule tasks!).

#!/bin/sh

rsync -e 'ssh -p 23' --delete -avz /Users/brain/Desktop /Users/brain/Documents /Users/brain/Library /Users/brain/Movies /Users/brain/Music /Users/brain/Pictures /Users/brain/Software brain@crashoverride:/home/brain/files/cerealkiller/


I had to set up ssh without a password for this script to work when run in the background such as by anacron. I don't know how much of a security issue this is, but oh well.

ok, I think that's it. what do you do?
andyp's avatar
12 years ago
link
andyp
nothing is wrong - what are you scared of?
wow.. i wish i could back up my shit wirelessly. i desparately need to rethink my organization especially of my photos. Right now I only have one copy of my photos on an external hd, and I have some important photos copied on my macbook hard drive. can someone help me out? I really need it. I just bought another 300gb external that I have not touched yet. open for suggestions :D
Chiken's avatar
12 years ago
link
Chiken
Don't Let Your Walls Down
tape drives!
nestor's avatar
12 years ago
link
nestor
nestor
foldershare -> raid 1 on desktop
asemisldkfj's avatar
12 years ago
link
asemisldkfj
the law is no protection
is your MacBook drive too small for all of the photos, andyp?

if so, I'd invest in another external or upgrade the drive in your MacBook.
andyp's avatar
12 years ago
link
andyp
nothing is wrong - what are you scared of?
yeah its an 80gb hd. and I'm constantly throwing shit onto my external hd to make room to import new photos. I have 10gb on deck for that, but with my new camera it gets eaten up quick. I'm more worried about how i'm going to deal with my new blank external hd. is there a way to set it up to automatically back everything up every time i plug it in?
DaGr8Gatzby's avatar
12 years ago
link
DaGr8Gatzby
Drunk by Myself
Just get an enclosure. Firewire is the way to go.
asemisldkfj's avatar
12 years ago
link
asemisldkfj
the law is no protection
I'm not sure of any way to automatically back up when you plug it in, but I have a feeling there's gotta be a program out there that does that.

it would be tough having two externals that you want to mirror unless you got a case designed to do that.

this might be overkill, but you could set up a computer somewhere in the house as a backup server type thing. this would make more sense if you already had a computer set up that was acting as a router or a server.
 
12 years ago
link
dbrown
assuming it's windows, you could just do an autorun script on the external.
nestor's avatar
11 years ago
link
nestor
nestor
i read today about someone who lost their 400gb music collection - it was on an external hd.

that is so terrifying. i need backup... when i come home i'm doing raid 1 again
lucas's avatar
11 years ago
link
lucas
i ❤ demo
s3
s3
s3
asemisldkfj's avatar
11 years ago
link
asemisldkfj
the law is no protection
that's why you never store things you care about in only one place.
nestor's avatar
11 years ago
link
nestor
nestor
i can't s3 100gigs, can i?
lucas's avatar
11 years ago
link
lucas
i ❤ demo
of course!

i currently store about 15gb, for a cost of about $2/month.
lucas's avatar
11 years ago
link
lucas
i ❤ demo
get one of these external enclosures--everyone loves them:
http://www.newegg.com/Product/Product.aspx?It … 6817173010
asemisldkfj's avatar
11 years ago
link
asemisldkfj
the law is no protection
I haven't been doing backups lately, but I've got a new network setup at home. my router/firewall is now a smaller, less powerful box with only one hard drive. I still have my old router/firewall box so I'm going to use it as a file server or something else cool (suggestions?). I'll use it for backups too; right now I have a problem with NFS so I've gotta work that out first. then I'll probably do something similar to what I was doing before.
nestor's avatar
11 years ago
link
nestor
nestor
added another 750gb to my workstation yesterday
bluet's avatar
11 years ago
link
bluet
i'm making a raid 1 array with two 200 GB disks for backup, possibly encrypted
nestor's avatar
11 years ago
link
nestor
nestor
my hd setup:

2x250 gb raid 1 (wd2500ks)
750gb (wd7500)
80gb raptor (wd800adfd)

all sata2
bluet's avatar
11 years ago
link
bluet
fileserver:

maxtor pata disks:
300 GB for root, home &c
160 GB for incoming torrents
2x200 GB

samsung sata disks:
2x1 TB for music, movies &c. these have ffs2 filesystems

i've got room for 8 more sata disks
lucas's avatar
11 years ago
link
lucas
i ❤ demo
====================================================================

wlw:~/svn/classes   s3:lucas-vc                         every 6h
wlw:~/svn/journal   s3:lucas-vc                         every 6h
wlw:~/svn/larz      s3:lucas-vc                         every 6h
wlw:~/svn/ttf       s3:ttf-vc                           every 6h
wlw:~/svn/wlw       s3:wlw-vc                           every 6h

mysql@wlw:ttf       s3:ttf-db, ttfbackup@gmail.com      every 6h
mysql@wlw:wlw       s3:wlw-db                           every 6h

wlw                 s3:wlw, ext_hdd                     on demand

~\private           s3:lucas, ext_hdd                   on demand
D:\private          s3:lucas, ext_hdd                   on demand

D:\ap               ext_hdd                             on demand
D:\music            ext_hdd                             on demand
D:\programs         ext_hdd                             on demand

====================================================================
asemisldkfj's avatar
11 years ago
link
asemisldkfj
the law is no protection
I installed Leopard and have been doing backups via Time Machine lately. my firewall has not been getting backed up though :/. I should make a cron job to back up my pf.conf at least.
asemisldkfj's avatar
10 years ago
link
asemisldkfj
the law is no protection
I don't think I've done a backup of my workstation since the last time I posted in this thread. pretty stupid. I think I'm going to return to using a tower-size machine for my routing and backup needs and maybe sell my nettop computer that I've been using as a router. I can't justify the electricity usage of having the nettop do routing and the other machine just do backups.
asemisldkfj's avatar
10 years ago
link
asemisldkfj
the law is no protection
just bought two 500 GB drives from newegg. not sure if I'll do RAID or just keep one as a backup drive in the case of hardware failure. I thought about upgrading my entire router/server box with a new mobo/cpu and stuff but decided against it. not worth the money. the only problem is that it's going to be hard to find non-SATA hard drives pretty soon.
lucas's avatar
10 years ago
link
lucas
i ❤ demo
i think 500gb hdds are a good size to go with. the 1tb hdds don't seem to have a good review average.

i'm disappointed that i went with 250gb hdds when i started on my setup. luckily it's a very reliable drive and it's pretty damn cheap.
asemisldkfj's avatar
10 years ago
link
asemisldkfj
the law is no protection
the only reason I didn't go for a TB is because there were no non-SATA TB drives.
bluet's avatar
9 years ago
link
bluet
Since rdiff-backup is at different versions on some of the machines I use, and not installed at all on others, here's my new solution:

1. Mirror files from all my machines to their per-machine directories on atlas.
2. Run rdiff-backup on the mirrors on atlas to another backup directory on atlas for versioning.
3. Goto 1.
nny's avatar
9 years ago
link
nny
M̮͈̣̙̰̝̃̿̎̍ͬa͉̭̥͓ț̘ͯ̈́t̬̻͖̰̞͎ͤ̇ ̈̚J̹͎̿̾ȏ̞̫͈y̭̺ͭc̦̹̟̦̭̫͊̿ͩeͥ̌̾̓ͨ
My new backup solution.

We are all going to die. The earth will burn. Our star will collapse. Our galaxy will be consumed by itself. The universe will go dark and die in heat death.

Carpetsmoker's avatar
9 years ago
link
Carpetsmoker
Martin
I just use subversion for backing up stuff, it's easy to set up, easy to manage, widespread, and keeps previous versions.

In the past I used cvs, which also works OK except that it can mangle binary data.
Carpetsmoker's avatar
9 years ago
link
Carpetsmoker
Martin
Why not back up the universe?
nny's avatar
9 years ago
link
nny
M̮͈̣̙̰̝̃̿̎̍ͬa͉̭̥͓ț̘ͯ̈́t̬̻͖̰̞͎ͤ̇ ̈̚J̹͎̿̾ȏ̞̫͈y̭̺ͭc̦̹̟̦̭̫͊̿ͩeͥ̌̾̓ͨ
But brain, where will we find rubber pants that size?
lucas's avatar
9 years ago
link
lucas
i ❤ demo
> atlas

my atlas?

<3
bluet's avatar
9 years ago
link
bluet
Yes, your atlas. Also, OH SHIT I FORGOT TO REALLY IMPLEMENT IT!!! :O
asemisldkfj's avatar
9 years ago
link
asemisldkfj
the law is no protection
working on a new one of these now. just made a list of all the data stored in various places. I'm thinking about storing encrypted tar archives on amazon s3 to have off-site backup of certain data like my Documents and Pictures directories. I'll obviously exclude all my mp3s and stuff because that would be way too much disk space to pay for and I don't care that much about them.
nny's avatar
9 years ago
link
nny
M̮͈̣̙̰̝̃̿̎̍ͬa͉̭̥͓ț̘ͯ̈́t̬̻͖̰̞͎ͤ̇ ̈̚J̹͎̿̾ȏ̞̫͈y̭̺ͭc̦̹̟̦̭̫͊̿ͩeͥ̌̾̓ͨ
amazon is pricy there are better backup centric alternatives.
asemisldkfj's avatar
9 years ago
link
asemisldkfj
the law is no protection
so I've heard. tarsnap is more expensive, but looks pretty cool.
bluet's avatar
9 years ago
link
bluet
But it uses Amazon S3 itself. I'd cut out the middle layer.
asemisldkfj's avatar
9 years ago
link
asemisldkfj
the law is no protection
yeah, I didn't notice that until carpetsmoker pointed it out on twitter.
asemisldkfj's avatar
9 years ago
link
asemisldkfj
the law is no protection
backing up all the configs and important data from my server to s3 now via a cron job! weekly.
asemisldkfj's avatar
8 years ago
link
asemisldkfj
the law is no protection
I've been jumping os's and storage/backup solutions so much lately that I'm sitting here next to two disassembled computers, one running openbsd from a usb stick and copying files from a sata drive (the sole residence of my documents, photos, music, etc.) to an external usb hdd.

I think I'm going to run freebsd as a file server. two 500 gb ide drives and a 1tb sata drive. os and file shares on the tb drive and copies of everything on both of the 500gb drives.
bluet's avatar
8 years ago
link
bluet
I've been doing hourly backups for nearly three months now. It finally filled my drive. What to delete? :O
asemisldkfj's avatar
8 years ago
link
asemisldkfj
the law is no protection
three-month old backups?
lucas's avatar
8 years ago
r1, link
lucas
i ❤ demo
i think atlas has some room for your crazy backups (then delete it locally)

or if they're mission critical, put them on aws s3 (then delete locally)
asemisldkfj's avatar
8 years ago
link
asemisldkfj
the law is no protection
speaking of atlas, I should back up to there. same with s3. I did that for about a month.
bluet's avatar
8 years ago
link
bluet
I'm thinking I'll just keep monthly snapshots for stuff that's more than a month old.
asemisldkfj's avatar
8 years ago
link
asemisldkfj
the law is no protection
backing up to atlas now!
asemisldkfj's avatar
8 years ago
link
asemisldkfj
the law is no protection
is scp'ing a gigabyte of data per week ok on atlas?
lucas's avatar
8 years ago
link
lucas
i ❤ demo
of course

if you're happy with the speeds (15Mbps/2Mbps), go for it
asemisldkfj's avatar
8 years ago
link
asemisldkfj
the law is no protection
cool! yeah, it took somewhere around an hour going 200-250 kB/s. I can live with that. it's automated anyway, and I just need at least one copy of important stuff stored away from home.
lucas's avatar
8 years ago
link
lucas
i ❤ demo
n.b. availability will be spotty at best as i relocate to santa barbara in four weeks
asemisldkfj's avatar
8 years ago
link
asemisldkfj
the law is no protection
no problem; I'll probably have s3 backups running by then.
lucas's avatar
8 years ago
r3, link
lucas
i ❤ demo
well, what i stress is that while it isn't always available, if you put something there, it is reliably archived. you might have to wait a bit to get it back, but you'll get it. that's the promise of larz and two enterprise disks in sync.

i'm also willing to do imports by sending me a usb drive. i can also have a 160 GB usb disk drive to which i am willing to export and ship to you.

also, while s3 is great for highly-available archives, it's expensive for retention of large amounts of data. accordingly, if you have 50 GB to archive and it need not be highly available, send it to atlas. that's why i have atlas in the first place.
asemisldkfj's avatar
8 years ago
link
asemisldkfj
the law is no protection
good pitch! that is basically what I need in a backup server. I already have my data on three disks, two of them in a raid-1 array, so I really only need access to the offsite backup in the event of something major like my house burning to the ground.
asemisldkfj's avatar
6 years ago
link
asemisldkfj
the law is no protection
weekly tar/7zip/aes files of important files scp'ed to atlas (thanks, lucas!).

and now I'm doing hourly backups to an external drive with rsync. the logic in that script is kind of fucked and doesn't make it easy to adjust for, say, daily backups instead of hourly. and also requires more and more conditions for weekly/monthly/yearly backups. but it's working and taking up like zero disk space thanks to --link-dest.
lucas's avatar
6 years ago
link
lucas
i ❤ demo
no problem! wow, atlas still has 732 GiB available.
Carpetsmoker's avatar
6 years ago
link
Carpetsmoker
Martin
http://www.freebsd.org/doc/en_US.ISO8859-1/bo … dp88123568
asemisldkfj's avatar
6 years ago
link
asemisldkfj
the law is no protection
hahahaha

that was my backup strategy for a long time! and it worked out quite well, luckily.
bluet's avatar
6 years ago
link
bluet
That's also been mine since my backup drive started failing. :(
lucas's avatar
6 years ago
link
lucas
i ❤ demo
is that supposed to point to the "do nothing" section?

bluet: not into atlas backups anymore? i'm glad asemi is getting use out of it... :)
bluet's avatar
6 years ago
link
bluet
> is that supposed to point to the "do nothing" section?

Yes.

> bluet: not into atlas backups anymore? i'm glad asemi is getting use out of it... :)

I've got plans, I just need to get a circular tuit.
lucas's avatar
2 years ago
link
lucas
i ❤ demo
look at this! cold storage.
bluet's avatar
2 years ago
link
bluet
i've bought a blu-ray writer and some discs that i've been meaning to back up to. write once, read many! :D
lucas's avatar
2 years ago
link
lucas
i ❤ demo
that's cool. i think tapes would be fun. probably more fun than useful.
bluet's avatar
2 years ago
link
bluet
tapes are cool, for sure. but they probably only make sense at huge scales where the cost of the equipment is dwarfed by the cost of media

blu-rays are good for distributing to your friends, for dirt cheap off-site backups
bluet's avatar
2 years ago
link
bluet
damn, now you've made me want to make a reel-to-reel storage system for my pc

i wonder how many bits you can store on one of these: http://www.ebay.co.uk/itm/TEAC-X10R-AUTO-REVE … 2506203052
lucas's avatar
2 years ago
link
lucas
i ❤ demo
haha yeah!! we had a reel-to-reel just like that at the radio station. used to all the time for temporary recording
lucas's avatar
2 years ago
r1, link
lucas
i ❤ demo
ideally you'd take data, run it through symmetric encryption, and somehow encode it and lace it into audio. record this on a bunch of tapes. then you have plausible deniability.
lucas's avatar
2 years ago
link
lucas
i ❤ demo
this is coolest so far: punched tape!

http://www.ebay.com/itm/FACIT-4070-DESKTOP-PR … 2142506899

https://www.youtube.com/watch?v=61uG-y0e3wE
Carpetsmoker's avatar
2 years ago
link
Carpetsmoker
Martin
Dunno ... Don't think the tech is quite ready yet. Maybe in two or three years? It's definitely the future though!