/eternalarchive/ - The Eternal Archive Project

I can only offer my experience, but I'd like to emphasize one of the truisms that everyone will tell you:

RAID is _not_ a backup.

I had everything I'd ever decided to save (~7tb of stuff), including things I had ripped from TV myself, recorded from my gameboy, notes to myself in the future, everything on a RAID 5 array.

I decided to migrate it to RAID 6 as rebuilding a several TB raid 5 is risky; but one of the drives just timed out and the controller lost its shit.

I paid over 5k for professional recovery, they were able to get about 80%. All of my porn was in a truecrypt archive so people couldn't just windows search "xxx" and find it; they were not able to find the file record for that it was lost entirely. Literally porn downloaded via kazaa over a 56k modem, edits I'd asked for, things from /b/ years ago that are no longer anywhere, thing I may have had the only copy of. Gone.

I thought as long as I didn't fuck up and accidentally delete things myself, if a drive died a could recover, if other hardware died I could replace it, essentially 'it couldnt happen to me'. I'm trying to impress upon you if you do not have two full copies of your data somewhere; you will lose it.

I'm planning on rebuilding; a couple 4TB drives (5x in raid 6 gets you 12tb) + bottom tier PC parts + linux to do software raid for a backup server. My raid controller on my main PC with the same sized array, synced over the network. If that is too expensive, I'd recommend encrypting everything then using a backup service.

But whatever you do, you have to remember if you don't have two copies you will lose it. Don't make my mistakes.

>>259

Nigga just use freebsd at that point.

>>260

Not sure what you are reffering to. My most recent experience with FreeBSD culminated with my entire filesystem getting corrupted and wrecked. I did not pursue recovery as much as >>259 did, as it was not really that important what was there, and all that was important was elsewhere.

>>252

I would have to say, if it is valuble, if it is unique, back it up. Perhaps it is me but 'the cloud' does not seem a good option. The cloud can disappear, the service you are working with can go under. You cannot really trust hardware that you do not physically control yourself.

It would be far simpler and more reliable to make duplicates on hard disks. It might be a little more hassel upfront but in the end it will safegaurd the data better.

>>252

> What would be the best method to back this up?

Buy a single 4 tb hard drive and put everything on it, wrap the hard drive in the grey anti-static faraday-cage bags they often come with, and store it at a friend's house whom you trust not to just go "lol what's this junk?" and throw it away.

Getting a 4TB HD this week, I'll keep a copy of important shit in there but it's not that fail-safe.

http://www.lto.org/ looks good as fuck and it's cheap, too bad the reader is expensive as fuck.

Having 4 kinds of backups simultaneously is the best case. First layer, a redundant raid array. Second layer, on-site powered off backup, power it up reguarly and do point in time backups, keep at least 3. Third layer, off site backup whether using some sort of backup service or something manual.

I dont know what a good backup service is thesedays, wouldnt trust their own encryption either, do it yourself. If anyone has suggestions for backup services with good reputations put them here.

Any backup that you keep connected to your computer is not a backup, it's just storage.

Backups are kept away from your computer, where fire and electric failures can't touch them, and updated as much as you can spare.

>>287

>A more exotic option is tapes, but the problem is that the drives are obscenely expensive

And extremely fragile. Seriously, tapes are not permanent storage media at all. They're meant to be overwritten regularly.

> FreeFileSync to handle my backups

Good.

>does anyone know a good solution (on Windows) for monitoring file integrity?

No, but I wouldn't trust it anyway. You could periodically format your backup drive and completely re-write all data on it if you want the data to be totally re-written. I've done it a few times.

>>259

>they were not able to find the file record for that it was lost entirely

NTFS? I suggest ext4 zfs reiserfs or other better file systems

>>287

will it kill if my AC power blacked out?

>>359

>NTFS

Indeed. In fairness, at 1TB out of 8TB I'm sure some of it is corrupted anyway, but man, fuck highpoint. Disk times out, migration stops, what do we do on next reboot? Just plow ahead like nothing happened, obviously.

>>252

I just discovered this board, while in the process of archiving stuff, and as it contains mostly books, I came to realize the problem is two-fold: it's part size, and part usage.

Starting with the usage, I came to realize I was hoarding more shit than I can possibly consume in my entire life. Worse, there's plenty stuff I kept "just in case", but really, in case I actually need it, I'll have other things to do than sit and read a book. Most of the stuff, I'd rather read and assimilate than keep it around.

Then, obviously a size problem. I have a lot of books, but most of that stuff doesn't have to be in fucking PDFs and shit. Simple text files are much better, and are more easily compressed.

So I started to fucking optimize everything, acquiring knowledge that needs to be acquired beforehand, and turning the rest into plain compressed TXT. There's a 0.17 compression ratio on average for plaintext PDFs, and 0.051 for OCR'ed scanned books. Which is huge.

With proper compression of all content, my core, most important archive is 17MB, and the extended one is 4GB. So obviously backup isn't an issue anymore.

>>764

I've come to a similar conclusion.

I'm still dealing with clinging to music though. Right now, I have the power to listen to almost anything I want but that may be taken away from me in the future. I can see how that fear can drive other people to hoard.

>>765

Indeed music is quite problematic, especially the temptation to store it all in FLAC. That shit DOES takes a lot of space.

It's less of a problem to me now though, mostly because the way I listen to music has changed quite a lot. It doesn't occupy the same place as before, and I mostly have a couple themed playlists with select few songs from various artists, and ditch the rest, and I can even content myself with Youtube quality, because I have no money to invest in a high end sound setup anyway, and not all music deserves it either, plus I listen to it in the backgroud most of the time so quality isn't that important as long as it's not atrocious.

My base music collection is sitting there on the side without any backup, at least until prices drop and I can get a bigger harddrive that can contain everything. The only thing I do backup is my 15GB core music collection of MP3 VBR2.

That's clearly not a solution for everyone though, but still, I'm happy to have detached myself enough of it that I don't feel the same need to hoard it, nor worry about it.

How should I go about building a backup server?

What backup drives are the best?

>>793

"Best" meaing fastest, most durable, largest? All going to be different responses. How much data are you storing? How important is the data to you? What's your budget?

>>794

About 8TB. It's my own archive. I'm a student on welfare, but I didn't get the money for the textbooks I needed last semester until just about a week ago, so that's a surplus of a bit over half a thousand.