Considerations

Storage market focus may deviate from your (=preservation) use case.
Business “longterm”: 3-5 years (=product lifecycle)
Cross vendor compatibility? Standards? Documentation?
Spare parts?
This market focus affects prices, features and reusability.
Who owns your storage = owns your work.

Digital Storage = Layers

A Digital File
Application required to read/write data
Operating system
Hardware it runs on
Filesystem
Drive (e.g. for tapes, optical media)
Physical carrier (medium)

Physical carrier

Classic “Spinning Platters” harddrive LTO: Linear Tape Open cartridge Optical Disks SSD: Solid State Disk (no cover) SD cards

Different carrier media have different properties when it comes to error resilience, longevity - and of course: price.

Magnetic: Susceptible to electro-magnetic issues.
Optical: light, temperature, warping
Electronics: Failing capacitors
Mechanics: Wear and tear
Supported/obsolete interfaces or drives?

It’s usually good to mix storage media to distribute their pros/cons properties.

I won’t go into great detail here, as this is possibly covered in many other preservation-storage presentations.

This here is for the interested reader.

Examples for different use cases:

preservation / backup:
tape (affordable size, good preservation properties, less need for super-fast access)
access: No need for long-term endurance, so e.g. optical would be sufficient.
production: HDD-based storage for nearline files - tape for offline storage. Or: tape-robot with HDD cache (same concept, but fully automated)

More properties:

Data Access Speed?
SSD > HDD > Tape > Optical
Cost/size ratio?
SSD > HDD > Tape > Optical
Widespread?
- SSD, HDD: Excellent!
- Tape: almost exclusive professional use case Drives not that “easy” to get (for particular LTO generation)
- Optical: Declining (except for “cold storage” in large data centers)

In greater detail:

Optical Disks

Pros:

Cost/size ratio: Relatively cheap (cheapest still?)
Not susceptible to electro magnetic (dis)charge
No mechanics/electronics in carrier - only in the drive

Cons:

Error prone
Suffers easily from material degradation (scratches, data layer falling off, light exposure, temperature, etc…)
Becoming less and less common = less support, less choice, etc.

HDD: Hard Disk Drive

Pros:

Reasonable cost/size ratio
(yet) cheaper than SSD
well known, well supported

Cons:

Mechanical wear (moving parts)
Susceptible to electro magnetic discharge
Electronics on carrier

Data Tape

Pros:

Cost/size ratio: Tape cartridge “cheaper” than HDD
Tape preserves well (and there is much experience with dealing with tape degradation/aging from analogue)
Supported in “tape libraries” (aka “tape robots”) for removing “tape-jockey” limitations.
Larger capacity per cartridge than per HDD (yet)

Cons:

Not common outside certain domains (large institutions, broadcast preservation, …)
Therefore support for tapes very “specialized”
Drives quite expensive (>3000 EUR)

Flash Memory

Pros:

cost/size ratio: getting cheaper, but still more expensive than HDD
No moving parts = no mechanical wear
SD cards: almost no electronics in the carrier
Speed: SD cards are not sooo fast, whereas SSDs (or M.2) are extremely fast (compared to HDD).

Cons:

Currently most expensive (still)
Lack of long-term experience
Limited number of write/erase cycles per flash block (becoming less of an issue) =Should not be used for large number of write operations. But preservation storage case is mostly “read”.

More about Flash memory on Wikipedia. SD card image by Steve Meirowsky (Wikipedia) - Own work, CC BY-SA 3.0

Dvdisaster: Optical Rescue.

Dvdisaster: Success!

HDD vs SSD

Good to know:

Higher data density = more impact of an error.
Example: 1mm hole in CD vs DVD vs BluRay - or SD vs MicroSD, etc.
HDD: The longer it is active, the shorter is lives.
But: “Hardware that lies, dies.”
Be aware if your HDD is “Shingled (SMR)” or not.
Mixing carrier/storage types: good idea!

Density:

Think of a 1mm hole on an optical carrier: CD vs DVD vs BluRay
All 3 have the same dimension, but different data density.
When choosing a carrier: Do you really need it to be as small/dense as possible?

Hardware that lies, dies:

Got external HDDs stored on shelves?
You may want to power them up every once in a while (every 1-2 years or so), and copy the data off them before the hardware gets old. Because: Electronics fail (capacitors), lubricants harden out or movable parts get “lazy” or stuck.

Das Werkstatt Rule #2 (German): “Hardware die liegt, stirbt” Useful “Das Werkstatt” principles

Shingled HDD:

“The more time a shingled disk spends in band compaction, the more it is wasting resources.”

Source: Classifying Data to Reduce Long-Term Data Movement in Shingled Write Disks This means, SMR disks are active when idle, causing them to potentially wear out faster.

Tape & Drives

LTO Generations

Imagine you find an old LTO tape…

Not every drive can read every tape.
New LTO generation release: ~every 2-3 years.
< LTO-7: Read=2 gen. / Write=1 gen.
BUT: LTO-8: Read/write=1 gen.!

Oh, and: Which filesystem was it written with…?

LTO improvements include new LTO generations: Higher data density at same form factor. This allows the mechanics (of drives and robots) to stay compatible, with only requiring certain parts to be updated to support a new generation.

The LTO consortium guarantees a certain level of downwards compatibility. This used to be:

Read: 2 generations down
Write: 1 generation down

Example: LTO-6 drive should be able to read LTO 6,5,4 - but write only 6 and 5.

With LTO-8, this was reduced to 1 generation backwards compatibility, so an LTO-8 drive can not read LTO-6, only back to LTO-7

New LTO generation release: About every 2-3 years. This must be considered for migration!

This also means that you might want to keep a few drives of different generations in house?

Oh, and if the application you use

The File System

Popular File Systems

FAT{16,32} (Microsoft)
NTFS (Microsoft)
EXT{2,3,4} (Linux)
ZFS (Sun)
HFS+ (Apple)
LTFS (IBM)

FAT:

FAT introduced in 1977.
Widespread, well-known and documented
Default for SD cards and embedded devices
FAT32: 32bit limitation (max 4GB filesize, limited max. device size)
No access permissions (=everyone can read/write everything)
Very simple and straightforward filesystem
Non-free / license fees by Microsoft.

NTFS:

Introduced in 1993.
max volume size (different implementations): 256 TiB (pre 2016), now 8 PiB (2019 or later)
Non-free / license fees by Microsoft.

EXT:

ext2 introduced in 1993.
ext4 standard for Linux-based OS.
ext4 and NTFS supported by digital cinema standard. (Because the projectors probably run Linux ;))
max volume size: 1 EiB (1024 PiB)
No licensing fees

ZFS: * Introduced in 2005 * More than “just a filesystem”: combines RAID (redundancy, handle harddisk failure) Logical Volume Management (LVM) and filesystem in one. * Integrated integrity checks * (recently) well known among IT administrators of unixoid systems. * Requires slightly more experience to handle (because of complexity. * Very interesting filesystem (also for preservation). Too much to describe here. You may want to read more on Wikipedia :)

HFS+: * HFS Introduced in 1985. * max volume size: 8 EB * Successor of “HFS” * Officially only supported on Apple OS

LTFS: * LTFS first prototype (Linux and Mac OS X) during 2008/2009 * Only for LTO tapes. * See separate page/slide on LTFS for details.

LTFS: Linear Tape File System

Open specification = vendor neutral
Better for preservation, but may not support “convenience” features.
All implementations must:
- Correctly read media that was compliant with any prior version.
- Write media that is compliant with the version they claim compliance with.

The Linear Tape File System (LTFS) is a good solution to avoid vendor lock-in by tape filesystems. Please make sure that you can read the data written onto the tapes under the following conditions:

Different drive
From different vendor
With (FOSS) software (and/or from different vendor)

Otherwise, you won’t really know if you can access your data as technology- and vendor-neutral as desired.

By default, different applications may write data in their own way on LTO tapes. This allows them to implement some non-standard features that often provide more “convenience” by offering additional features. Using such vendor-specific features, instead of LTFS has the downside of reducing the possibilites to read that data under different conditions in the future.

Therefore it is advised to use LTFS for long term preservation use cases.

File system: Disaster relevant?

Deleted files are still there (maybe fragmented).
Different FS = different error resilience and recovery options.
Does it scale? (moving files is when they’re most vulnerable…)
Tools/knowhow to deal with recovery of broken filesystems in your setups?
Logical Volume Management (LVM) snapshots

Deleted files: Deleting files is just flagging them in the index and their disk space is marked as available again. The data is still where it was - until a new write operation (=creating new data) uses the same space and overwrites the old data.
Related articles:
- Get Your Data Back with Linux-Based Data Recovery Tools
- [How to Install and Use TestDisk Data Recovery Tool in Linux]((https://www.tecmint.com/install-testdisk-data-recovery-tool-in-linux/)
- Scalpel: Extract lost files from disk/card image
Different FS have different error resilience and recovery options: Go ask your IT guys! ZFS is pretty awesome, but “simple and well known” is also nice for recovery.
FS recovery requires understanding the FS format. Got documentation? Source code?
Logical Volume Management (LVM) Remember VM-snapshots? You can do the same on storage level (below filesystem): LVM Snapshots
Does it scale?
- Total number of files/folders?
- Length of path?
- Suitable for NAS/SAN/cluster?
- FS vs Blocks vs Object-based?

Failsafe mechanisms

S.M.A.R.T.

Self-Monitoring, Analysis and Reporting Technology

“[…] is a monitoring system included in computer hard disk drives (HDDs), solid-state drives (SSDs),[1] and eMMC drives. Its primary function is to detect and report various indicators of drive reliability with the intent of anticipating imminent hardware failures.”

Source: Wikipedia: S.M.A.R.T.

S.M.A.R.T. Graph

RAID

Redundant Array of Inexpensive Disks
Example of a disk chassis

“[…] is a data storage virtualization technology that combines multiple physical disk drive components into one or more logical units for the purposes of data redundancy, performance improvement, or both.”

Source: Wikipedia: RAID

RAID

Different RAID levels:
- Stripe = Fast, but dangerous!
- Mirror = Perfect for OS system disks
- RAID5/6 = 1 or 2 disks fault tolerance (parity bits)
RAID is not a backup.

Sad, but true:

With > 4TB disks, RAID-6 may be insufficient…
Long rebuild times may “burn out” the left-over disks.
ZFS supports 3 disks parity, but … future?

btw: How does “The Cloud” do it?

The algorithm RAID-6 uses to provide data redundancy for rebuilding in case of harddisks failing, may have come to an end with HDD capacities beyond about 4TB per-disk.

The reason for that is, that in case of a disk failure, the RAID mechanism reads the parity data from all remaining disks to reconstruct the one that “has gone”. With larger disks, the time for this rebuild to happen is getting longer. During that period, all remaining disks are constantly “stressed” until the required data has been read and the new disk rewritten.

Considering that disk failures more likely occur in RAID arrays that have been running for quite some time, it can be assumed that the other disks may be stressed too hard during that rebuild, causing other disks to fail due to the recovery - resulting in a complete data loss.

Cloud Storage

No internet = no service.
Make sure your online bandwidth is sufficient.
Have an exit plan (+contract?) for migration.
What if their conditions change?
Where is your data actually stored?
Does it matter for you? ^{(e.g. legally)}
Again: Consider mixing.

The term “Cloud” in computing came from marketing, and describes nothing else than digital services that already existed for over 20 years - but now had a nicer GUI and focus on convenient usability.

There’s “cloud storage” (e.g. Owncloud/Nextcloud, Dropbox, etc) and “cloud computing/services” (GDocs, etc).

In this context we will only deal with “cloud storage”.

Storing data in large data centers is very convenient. All the same challenges regarding large-scale storage still apply, but they’re someone else’s problem now :)

What’s still relevant for you though is:

What happens if the provider changes their conditions?
What happens in case of unstable political situations, war, etc?
What if you want to migrate your stuff to another storage/provider?
Does it matter (e.g. legally) if the provider could access your data?

Storage Debate!

Storage Debate: Summary?

Online/Cloud: For “smaller” files (eg Access copies)
Large storage: You may want/need to get external support.
Decide which level of control you have over your data.
Mixing (again) is a good idea.
Should we archives outsource all knowhow/skills about digital storage?

Cluster & replication

Multiple copies on multiple servers
(“cluster nodes”)
Synchronized (replicated) automatically
Off-site replication possible
Replication is not a backup!
What if a node goes down?

Erasure Coding

More parity possible than RAID
Data can be spread across network cluster nodes.
Does not suffer from rebuild “burn out”.
More complex to set up & compute.
Not that widely known/supported/used yet.
A FOSS implementation: MinIO

“protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in MinIO erasure code you can lose as many as half of drives and still the data remains safe.”

Source: MinIO Erasure Code Quickstart Guide

“Compared to data replication, erasure-coding approaches have better performance at reducing storage redundancy and data recovery bandwidth.”

Source: Reliability Assurance of Big Data in the Cloud (2015)

Links:

Object storage

Beyond classical, hierarchical filesystems

Object storage

Files become “objects”.
There are no (actual) folders. (Foldername = Plain Metadata)
There’s just an identifier.
Arbitrary metadata may be assigned to each “object”.
Scales infinitely (theoretically).
Clusters of block storages with a clever abstraction layer.
Quite a complex construct of APIs, layers and components. Fascinating.

Object storage = The future?

If implemented well, the storage itself could be the DAM/MAM.
Allowing links/relationships between objects = 🤯🤠🥳
The object filesystem is an actual catalogue.
Combined with LoD, MD Standards & APIs = Wikidata for/of your files.
If implemented well…

Object storage = The future?

Object based storage may very likely replace hierarchical filesystems, but things need to be rewritten/adapted to properly support it.

It’s not yet (really) plug-compatible with existing programs.

Software Defined Storage (SDS)

More “Classical”, yet Supersize-able.

Again: Onions, eh Layers.

Does it scale?

Does it scale?

What if you run out of diskspace?
Need to copy/move everything?
Or can you simply add “more space”?
What about your file/folder positions?
How to know if everything is still “there” and intact?

Data Integrity?

Fixity information & Hashcodes :)

Hashcodes: fixed size number that’s like a fingerprint for data.

CRC =
4294967295
MD5 =
d41d8cd98f00b204e9800998ecf8427e
SHA256 =
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
xxHash =
e4c191d091bd8853

Hashcodes are fixed-length numbers that are generated in a way that if even a single bit in the source data changes, that number will be completely different. They are often written in hexadecimal, therefore including the characters 0-9 and A-F.

If you have a hashcode for a set of data, it can be used to verify the bit-exact integrity of that data, by calculating the same-algorithm hashcode again and comparing them. If they’re identical, the data is intact. If two distinct sets of data have the same hash, it’s called a "hash collision. Hash algorithms are designed particularily to keep the chance for collision as low as possible.

Anyways: hashcodes are a must for safe data transfer and integrity checks.

Different hashcodes algorithms/implementations may have significantly different runtimes. When dealing with large quantities of data, this may matter.

MD5 is the most popular one around: Well known and widely supported by different applications/systems, etc.

Anyone transferring lots of uncompressed film? You may want to look at xxHash. It’s designed for speed.

In case someone has heard that MD5 is broken, fear not: For plain checking of file transfer or stored data integrity, MD5 is sufficient. It was cryptographically “broken” - which is relevant for security, but not for data integrity checking.

Important: Hashcodes are not sufficient for proving authenticity! In case you need to deal with important originals/documents and you need to make sure they’re originals and not forged, etc. please check out this:

Digital signatures are good, but could be signed with a date in the past (backdating).

If this is a concern, you may consider blockchain mechanisms: Blockchain cipher became popular for digital currency (like Bitcoin), but it can also be used to proof data authenticity and avoid backdating.

AFAIK there are currently no systems that implement this productively yet, but some have already researched into prototyping blockchain use for storage.

Do it early, do it continuously.

Create hashcodes early in the lifecycle.
Use them when transferring files.
Storage: Ongoing integrity checks matter.
Consider runtimes & interference with daily work.

Ongoing integrity checks on your storage: Know where your collection is at.

It also makes sense to verify your backup that way.

Consider runtimes:

In order to generate/verify a hashcode, the whole file must be read at least once.
The chosen hashcode algorithm and its implementation may have impact on processing speed.
Do some tests, calulations to find a sweet spot for how often you can do integrity checks.
Consider throttling throughput data to avoid interference with daily work.

Real world example:

Archival Institution had hashcodes (ingest system created them), but never used them. They told me: “We don’t need to verify hashcodes, we have no problems” So I checked. More than 1000 items had invalid hashcodes. Since they were never checked before, it is unknown when or what caused them - and also their impact on the material.

It’s good practice to document the result of integrity checks somewhere. It helps finding issues in your setup, or at least narrowing things down.

Hashcode manifest

As simple as that. Plain text.

Using (=validating) hashcodes

HashCheck: GUI tool showing data validation

Some tools

Our storage has integrity checks built in!

Good! 🥳
But it can only do so as soon as data has arrived.
Did you verify that the data got there intact? 🧐

Exercise

The “cursed” digital archive folder.

What hashcode manifests can tell you beyond “it’s borken”, and how to interpret them.

Backups

The 3-2-1 Backup Rule

Keep at least three copies of your data.
Store two backup copies on different devices or storage media.
Keep at least one backup copy offsite.

Backup integrity?

btw: Is your backup copy (still) intact…?
Did you check? ;)
Tape: Backup drives on shelf vs robot?
Handout: Backup checklist

Migration?

Eternal Migration

Always consider how to get your data out of any system before you get into it.
Or at least while it’s still running.
As vendor- and technology neutral as possible.
Try that before you buy.
Include migration in your planning.

💾 Which Storage is Right for YOU? 🤩

Must	Should	Could	Won’t
______	______	______	______
______	______	______	______
______	______	______	______
______	______	______	______

Choose a use-case and try to phrase your wishes and requirements.

Storage: Summary

There’s more than just one right solution…
Mixing is usually a good idea.
RAID6 might be at its limits :(
Consider LTFS and LTO generations
Embrace hashcodes!
Consider which level of control you have/want.
Have a backup (plan)

- fin -

Questions? Comments?

Peter Bubestinger-Steindl

Peter @ ArkThis.com

Storage Terms Collection

S.M.A.R.T. Self-Monitoring, Analysis and Reporting Technology
NAS Network Attached Storage
SAN Storage Area Network
RAID Redundant Array of Inexpensive Disks
Object Storage
JBOD Just a Bunch Of Disks
SDS Software Defined Storage

Archival Storage

Considerations

Digital Storage = Layers

Physical carrier

Dvdisaster: Optical Rescue.

Dvdisaster: Success!

HDD vs SSD

Good to know:

Tape & Drives

LTO Generations

The File System

Popular File Systems

LTFS: Linear Tape File System

File system: Disaster relevant?

Failsafe mechanisms

S.M.A.R.T.

S.M.A.R.T. Graph

RAID

RAID

Sad, but true:

Cloud Storage

Storage Debate!

Storage Debate: Summary?

Cluster & replication

Erasure Coding

Object storage

Object storage

Object storage = The future?

Object storage = The future?

Software Defined Storage (SDS)

More “Classical”, yet Supersize-able.

Again: Onions, eh Layers.

Does it scale?

Does it scale?

Data Integrity?

Fixity information & Hashcodes :)

Do it early, do it continuously.

Hashcode manifest

Using (=validating) hashcodes

Some tools

Our storage has integrity checks built in!

Exercise

The “cursed” digital archive folder.

Backups

The 3-2-1 Backup Rule

Backup integrity?

Migration?

Eternal Migration

💾 Which Storage is Right for YOU? 🤩

Storage: Summary

- fin -

Questions? Comments?

Storage Terms Collection