[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]

/eternalarchive/ - The Eternal Archive Project

Fighting the TPP

Catalog

See 8chan's new software in development (discuss) (help out)
Infinity Next Beta period has started, click here for info or go directly to beta.8ch.net
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Options
dicesidesmodifier
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 8 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.

Submit a list of your archived shit regulary in the meta thread

File: 1448299626173.png (1.12 MB, 1024x640, 8:5, faraday.png)

8cd835 No.307

SECURE STORAGE AND SHELF-LIFE: https://archive.is/vgIQy

Let me introduce myself briefly before I present this key information. I am a professional archiver and a librarian. I have archived mass amounts of data/media for over 8+ years now. I have dealt with and overcome a vast amount of problems in my past, including: computer crashes, witnessing data degradation of certain storage platforms including (VHS/Betamax/floppies/cassettes/CDs/DVDs/flashdrives/HHD/etc.), malicious hacking attacks against my network(s), to even simply misplacing my storage from time to time.

There are two things every archiver must know about, because this is key for long-term storage, the defense and integrity of your storage. SECURE STORAGE AND SHELF-LIFE.

The good news is external (offline) storage platforms have become more robust over the years, the bad news is even CDs & DVDs will inevitably face natural data degradation at some point in the future. Although widely debated, the average DVD-R (4.7GB) has a shelf-life of up to 25 years max. After 25 years, natural data degradation will begin to occur to the point the DVD-R will become unreadable and useless. CDs also fall into similar expectancy. Mind you, this is if you take proper care of the optical discs such as simply leaving them in binders and rarely using them. Data degradation is also natural due to aging, not just physical neglect.

Optical discs tend to be more robust than most magnetic mediums of storage, and so are flashdrives, but even flashdrives WILL NOT last forever! Flashdrive shelf-life can vary depending on the manufactures making the EPROM hardware.

Now you may ask: how do we deal with these inevitable problems with long-term storage? There currently are several ways to mitigate these threats! Here are my recommendations:

(1) For every computer you have, you should have a physical bootable ISO backup of the operating system. I recommend having two computers, one with online access and one air-gap (offline).

(2) After you get done downloading an archive of media online, you should make a physical backup of it: be it a flashdrive, a DVD-rom, a CD-rom, or on an external harddrive and after that media or data is backed up, air-gap it (disconnect it) from the online computer system.

(3) Place all physical backups into a Faraday Cage. Make sure DVDs and CDs are protected inside binders BEFORE placing them into a Faraday Cage. Make sure you put your flashdrives or external harddrives in a bag or a cigar box BEFORE placing them into a Faraday Cage. A Farady Cage will protect the storage mediums from two threats: one, collecting dust and two, (God-forbid) if there was an electromagnetic pulse (EMP) attack from a foreign enemy.

Faraday Cages must be isolated from the storage on the inside. You cannot have an effective Faraday Cage if the storage mediums are physically touching the inner rims, as during an EMP it acts as a conductive barrier. Any electronic touching that conductive barrier would be fried.

Here is an online video tutorial of DIY Faraday caging:

watch video » https://www.youtube.com/watch?v=0yZz5pXBC1k

download mp4 » http://www3.zippyshare.com/v/1mXAmXtR/file.html

(4) Last, but not least, you must consider stocking up on BLANK store mediums (unused and keep in the case). Flashdrives, external harddrives and optical discs! Why? One example: Once you rip a DVD, you will have around 20 to 25 years before natural data degradation occurs, so you will have to remember to re-backup those old DVD and to do that you will need a good storage of blanks.

Flashdrives and even external harddrives are the same, and you should be ready to re-backup any old storage before data degradation occurs. Remember that you will only have to do this a few times in your lifetime so its not a big deal.

Here is a new technology that you might want to look into, fellow archivers: http://www.mdisc.com/

These discs are built to last for a long, long time! However they are not cheap.

00adc7 No.308

>>307

what about HDD's in RAID 1?


677353 No.313

>>307

>you must consider stocking up on BLANK store mediums

wouldnt those degrade as well? shouldnt you be buying the blank storage mediums once the old ones are close to begin degrading?


372a26 No.314

>>308

I don't know the difference between the RAID partitioning so I really can't say. Its best to assume that data degradation will *inevitably occur, even if more robust than others.

>>313

The optical discs themselves will be fine. Its the data burnt onto those discs that face degradation over time even if the disc itself is kept in mint condition.


10e0fa No.317

>>308

RAID 1 is just for redundancy, so if one drive fails the system will continue to operate normally using the other drive.


000000 No.471

>>314

First, blanks: Yes, they SEEM TO degrade. I have not seen good info on this or made an a test of it. An EXCELLENT topic of study.

On storage of written optical media: The disks should be stored at appropriate atmospheric conditions of humidity, temperature (but pressure range isn't a worry on Earth, for optical media).

They should be stored upright, like music records or books.

For the longest-lived consumer-available storage solution consider M-disk:

>"M-DISC is an archival-quality storage solution that preserves […] for 1,000 years or more."

Now available in BD-XL (100gb).

http://store.mdisc.com/Verbatim-M-Disc-100GB-BDXL-5-pack-jewel-case_p_20.html

$100 for a 5 pack, so $.20/gb.

smaller disks look the same $/gb, so may as well go big.

Drives:

http://www.mdisc.com/m-ready/

From the FAQ:

"Can the M-Disc DVD be read by regular DVD drives?

Yes. The M-READY™ Drive engraves the M-DISC™ using a compatible data format that can be read by most quality DVD drives.

Can the M-Disc DVD or M-Disc Blu-ray be written using regular optical disc (Non-M-READY) Drives?

The M-Disc Blu-ray meets all of the specifications for the Blu-ray recordable (BD-R) format. This means it should work in any Blu-ray writer. Our testing indicates that it will work in over 80% of Blu-ray drives made after about 2011.

The M-Disc DVD requires an upgraded disc drive that can deliver the increased laser power required to burn an M-Disc DVD. Therefore, only an M-READY Drive can burn data to an M-Disc DVD."


03e05d No.478

File: 1448601758936.jpg (121.88 KB, 947x960, 947:960, 10524607_703477013020722_5….jpg)

>>308

>what about HDD's in RAID 1?

>>314

>I don't know the difference between the RAID partitioning

You may have read it around online, but if you haven't, let me be the first to say it:

RAID is not a backup

>Why? RAID 1 means that I've got a perfect, identical, live copy of data at all times?

Yes, so what happens when you accidentally delete that file you want? Oops! The RAID controller propagated that deletion on the slaved drive at the exact same moment. Unless it is recoverable using conventional data-recovering methods it is lost forever.

>but I never ever make accidental deletions, anon. It is physically impossible for that to ever happen to me.

Alright, so you're hot stuff. But what happens when your computer is hacked or infected and your data becomes corrupted by a malicious process? You guessed it: the changes are effected simultaneously across both drives.

Are you kind of getting the picture? Raid 1 only protects against a single kind of failure: Head crashing. Physical, mechanical failure of a drive.

>Okay, so I went to the store and picked up 2 drives that were right off the shelf, they were even right next to each other! Super safe, right?

Wrong. If the failure of drive 0 was caused by a manufacturing defect present in that batch, it is highly likely that the exact same defect is present in drive 1, and even more likely that because they were sitting next to each other on the shelf, that they were shipped in the same container, from the same batch.

So are you getting the picture, here? RAID is not for use by people who don't fully and completely understand how and why they work. If you have 2TB of storage, go out and buy another 2TB and copy the files to it. If you want a hot-swappable disk for your OS drive in case it crashes, you can do what I do:

Say my boot drive is 500GB and I have 250 GB of storage. I go out and buy a 1tb hdd, which gives me 1000GB of storage stfu if you say 1024. After formatting you don't get that much, and that's not the point of this thought exercise So what I'm going to do is connect the 1TB hard drive to my computer and use SATA, not this USB shit. Buy a damned bare drive and install it. and use software (preferably FOSS) to clone (not "backup" or "image"!) the 500gb drive my 1TB drive. This will create a 500GB partition on the 1TB that is exactly the same as the boot drive in every way.

>What do you do with the other 500GB partition that's blank, anon?

Initialize and format the remaining space on the drive, then copy my 250GB of files over to it, also leaving room for expansion. Then I disconnect it and store it safely.

What does this enable me to do?

If my 500GB boot drive crashes, I can immediately swap it out for the 1TB drive, point my BIOS to boot from the OS partition, and I'm golden - so long as there hasn't been any major hardware changes that could introduce complications (new video card/processor/memory, etc.)

Once I am booted into the OS, the 250GB of files on the second partition will still be accessible. They will also be accessible and able to be updated independently of the OS files as any external hard drive would be.

Note that this will revert your computer to the state it was in whenever you first cloned the drive. If you install any programs since then, they will obviously not be on the cloned image unless you update the image.

To update your OS image after installing an important piece of software, or whatever, go back into your disk management/cloning software, format the old OS clone partition of 500GB, and then re-clone your OS disk to that same partition. Boom, updated.

This is if you want to kill two birds with one stone on the cheap. You can obviously adjust this to whatever size/level of redundancy that you want. However, I try to keep in mind that yes, magnetic media does degrade, so rather than just copy over more recent files, what I want to get more into is using the linux tool/module/thing "dd" which can compare data that is supposed to be "identical" across multiple drives, and automatically fix corruption using the better data from either source, if I interpreted some of its more advanced functions correctly.

Hope this helps. Pic unrelated, rats are just cute.


000000 No.479

>>471

Oh, and light. Of course optical media should be kept in darkness, typically in some kind of holder that has the little middle "snap" to keep the edge of the media from resting on the shelf.


e13bd0 No.524

Does anyone know a way to generate hashes for every file in a particular directory in Linux? I'm not too familiar with bash. Basically a way to automatically do "echo sha256sum filename.ext > .filename.ext.sha256sum" recursively. I plan to build a NAS with ZFS when I have the funds, but for the time being I'd like to be able to tell if a file is corrupted.

Eventually you're going to run into flipped bits or corrupted blocks. Backups can help, but not if it's already corrupted before it's been backed up or if it's corrupted on one drive and not on the other and you can't reliable tell which is which. Sometimes you don't know it's corrupt until it's too late.

How do you guys deal with bit-rot?


ba75af No.527

If you're buying harddrives then I suggest goharddrive.com

They've got good prices.

6TB for $150.


b77d0d No.529

>>527

>goharddrive.com

Yeah, nah. Nobody buy from there. Believe it or not, if a lot of storage (I'm talking like the $150 for 6TB) is cheap then it probably won't last long. Some shady vendors sell HDDs with SMART errors. Also some brands of HDDs are worse than others, as in, last a lot shorter than others. Mainly Seagate are the worst for long term storage and reliability, choose Western Digital if you are serious about this.


b77d0d No.530

>>478

>stfu if you say 1024.

Storage manufacturers sell their products in solid numbers like 1000, 500, 250 etc. Not 1024 (which is TiB, an actual Terabyte)


3d01a9 No.539

I'll tell you a secret, which is what I do regularly— buy Army ammo cans, burn data files on DVD-DL disks, put them in CD sleves, then put them inside the Army ammo can all bound up with tape, then hide 'e, in a cave in the desert!


3eb42a No.624

File: 1449028600232.jpg (447.52 KB, 3000x2400, 5:4, wallpaper-615829.jpg)

>>530

I know, which is why I was inb4ing because people with autism like to say, "lol pleb it's not 1000 it's 1024" as if they have some kind of special hacker insight or something, which pisses me off. I guess I still carry the scars from /g/.


248c70 No.626

1. Why "air-gap" it? Just so that nobody can use a virus or hack in and see the files?

2. If the actual data on the drives will deteriorate, what good will switching the corrupted data to a blank drive do? Does the data only start to deteriorate after a decade or two?


8292e0 No.637

>>626

>so that nobody can use a virus or hack in and see the files

Yes.

>what good will switching the corrupted data to a blank drive do

>Does the data only start to deteriorate after a decade or two?

It's not the data that deteriorates per se, it's the medium it's on. You're supposed to re-backup your data into a new medium before the current one starts deteriorating.


64c5a0 No.692

>>637

>You're supposed to re-backup your data into a new medium before the current one starts deteriorating.

Correct.

>It's not the data that deteriorates per se, it's the medium it's on.

Well, from my research it's the data that gets burnt that deteriorates after a long period of time (studies show between only 10 to 35 years expectancy). To be fair, there are some who do claim that blanks will only last as long as those that have data copied to them (some claim only 5 to 10 year life expectancy although I don't know if its actually been proven true yet). The optical discs can still be in mint condition if you securely store it (and rarely use it) but the data will face degradation to a point where the lasers in CD/DVD-Rom drives will not be able to interpret all of it. Here is where I got some information about these claims:

http://www.researchgate.net/profile/Matthew_Linford/publication/246856696_Optical_Disc_Life_Expectancy_A_Field_Report/links/541b325a0cf2218008c04304.pdf


e30c85 No.705

File: 1449757621391.png (70.48 KB, 785x1017, 785:1017, dhui_the_dog_by_karzahnii-….png)

>>307

good info, thanks bub


eaacee No.717

Taiyo Yuden is a company that makes the most reliable and longest lasting DVDs and CDs. They're not a brand name but specialty sites sell CDs direct from them.

http://www.supermediastore.com/brand/show/taiyo+yuden

http://www.mediasupply.com/taiyoyuden.html

Great for back ups.



[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]