Reality check

“Just storing files” is not preservation
Storage market focus may deviate from your use case
There’s more than just one right solution…
Mixing is usually a good idea.

Physical Media Types

Optical Disks

HDD: Hard Disk Drive

Data Tape

Pros:

Cost/size ratio: Tape cartridge “cheaper” than HDD
Tape preserves well (and there is much experience with dealing with tape degradation/aging from analogue)
Supported in “tape libraries” (aka “tape robots”) for removing “tape-jockey” limitations.
Larget capacity per cartridge than per HDD (yet)

Cons:

Not common outside certain domains (large institutions, broadcast preservation, …)
Therefore support for tapes very “specialized”
Drives quite expensive (>3000 EUR)

LTO Generations:

LTO improvements include new LTO generations: Higher data density at same form factor. This allows the mechanics (of drives and robots) to stay compatible, with only requiring certain parts to be updated to support a new generation.

The LTO consortium guarantees a certain level of downwards compatibility. This used to be:

Read: 2 generations down
Write: 1 generation down

Example: LTO-6 drive should be able to read LTO 6,5,4 - but write only 6 and 5.

With LTO-8, this was reduced to 1 generation backwards compatibility, so an LTO-8 drive can not read LTO-6, only back to LTO-7

New LTO generation release: About every 2-3 years. This must be considered for migration!

Flash Memory

SSD: Solid State Disk (no cover) SD cards

Media Types: Overview

Classic “Spinning Platters” harddrive LTO: Linear Tape Open cartridge Optical Disks SSD: Solid State Disk (no cover) SD cards

HDD vs SSD

Good to know:

Higher data density = more impact of an error.
Example: 1mm hole in CD vs DVD vs BluRay - or SD vs MicroSD, etc.
HDD: The longer it is active, the shorter is lives.
But: “Hardware that lies, dies.”
Be aware if your HDD is “Shingled (SMR)” or not.

Density:

Think of a 1mm hole on an optical carrier: CD vs DVD vs BluRay
All 3 have the same dimension, but different data density.
When choosing a carrier: Do you really need it to be as small/dense as possible?

Hardware that lies, dies:

Got external HDDs stored on shelves?
You may want to power them up every once in a while (every 1-2 years or so), and copy the data off them before the hardware gets old. Because: Electronics fail (capacitors), lubricants harden out or movable parts get “lazy” or stuck.

Das Werkstatt Rule #2 (German): “Hardware die liegt, stirbt” Useful “Das Werkstatt” principles

Shingled HDD:

“The more time a shingled disk spends in band compaction, the more it is wasting resources.”

Source: Classifying Data to Reduce Long-Term Data Movement in Shingled Write Disks This means, SMR disks are active when idle, causing them to potentially wear out faster.

Storage Types

External Hard Disks

External Hard Disks

Ideal for:

* Collections ≤ [size of largest HDD] * Accessed by only 1 computer at a time * Moving data * Quick access

Advantages:

* Relatively low cost (100-300 EUR) * Portable

Disadvantages

* Risky: Drives may fail * Backup is manual and easily out of sync * Access is limited

Network Attached Storage (NAS)

Network Attached Storage (NAS)

Ideal for:

* Collection < ca. 40 TB (if standalone). * Larger is possible, if clustered. * Quick access (incl. multiple networked users) * Networks with ≤ 1 GbE for large files (10 GbE for film) * Organizations with some IT support

Advantages:

* Multiple users can access in parallel * Relatively affordable (200-1000+ EUR)

Disadvantages:

* Requires heating / cooling * Potentially less secure (always on) * Less portable * Requires IT skills for problem solving

Data Tape

Ideal for:

* Collections from 10 TB to x PB * Collections that don’t need to be accessed in seconds * Back up scenarios

Advantages:

* Relatively low cost for tape stock * Scalable * Portable * Low failure rates

Disadvantages:

* Offline or nearline * Management & migration can be challenging * Proprietary tape filesystems

Data Tape Library (Robot)

LTO Generations

Imagine you find an old LTO tape…

Not every drive can read every tape.
New LTO generation release: ~every 2-3 years.
< LTO-7: Read=2 gen. / Write=1 gen.
BUT: LTO-8: Read/write=1 gen.!

The Cloud

Ideal for:

* Collections from any size * Institutions with limited IT support * Collections that don’t need to be accessed immediately * Fast access to smaller resolution files (streaming)

Advantages:

* Only pay for what you use * Scalable * Reduces the day-to-day management needs * AV: Low-resolution access scenarios

Disadvantages:

* HTTP access can be very slow for large files * Requires careful planning to ensure the correct services are being purchased * Depends on Internet connection * Vendor migration/lock-in

Storage Debate!

Networks

Consider: Layers!

The File System

File- / foldernames, size, timestamp, access rights

The File System?

Raw storage = unstructured chain of bits
File system (FS) = file/folder structure
Different FS = different formats:
- FAT{16,32} (Microsoft)
- NTFS (Microsoft)
- EXT{2,3,4} (Linux *)
- ZFS (Sun *)
- HFS+ (Apple)
- LTFS (IBM *)

(*) Open formats

FAT:

FAT introduced in 1977.
Widespread, well-known and documented
Default for SD cards and embedded devices
FAT32: 32bit limitation (max 4GB filesize, limited max. device size)
No access permissions (=everyone can read/write everything)
Very simple and straightforward filesystem
Non-free / license fees by Microsoft.

NTFS:

Introduced in 1993.
max volume size (different implementations): 256 TiB (pre 2016), now 8 PiB (2019 or later)
Non-free / license fees by Microsoft.

EXT:

ext2 introduced in 1993.
ext4 standard for Linux-based OS.
ext4 and NTFS supported by digital cinema standard. (Because the projectors probably run Linux ;))
max volume size: 1 EiB (1024 PiB)
No licensing fees

ZFS: * Introduced in 2005 * More than “just a filesystem”: combines RAID (redundancy, handle harddisk failure) Logical Volume Management (LVM) and filesystem in one. * Integrated integrity checks * (recently) well known among IT administrators of unixoid systems. * Requires slightly more experience to handle (because of complexity. * Very interesting filesystem (also for preservation). Too much to describe here. You may want to read more on Wikipedia :)

HFS+: * HFS Introduced in 1985. * max volume size: 8 EB * Successor of “HFS” * Officially only supported on Apple OS

LTFS: * LTFS first prototype (Linux and Mac OS X) during 2008/2009 * Only for LTO tapes. * See separate page/slide on LTFS for details.

LTFS: Linear Tape File System

Open specification = vendor neutral
Better for preservation, but may not support “convenience” features.
All implementations must:
- Correctly read media that was compliant with any prior version.
- Write media that is compliant with the version they claim compliance with.

The Linear Tape File System (LTFS) is a good solution to avoid vendor lock-in by tape filesystems. Please make sure that you can read the data written onto the tapes under the following conditions:

Different drive
From different vendor
With (FOSS) software (and/or from different vendor)

Otherwise, you won’t really know if you can access your data as technology- and vendor-neutral as desired.

By default, different applications may write data in their own way on LTO tapes. This allows them to implement some non-standard features that often provide more “convenience” by offering additional features. Using such vendor-specific features, instead of LTFS has the downside of reducing the possibilites to read that data under different conditions in the future.

Therefore it is advised to use LTFS for long term preservation use cases.

File system: Disaster relevant?

Deleted files are still there (maybe fragmented).
Different FS = different error resilience and recovery options.
Does it scale? (moving files is when they’re most vulnerable…)
Tools/knowhow to deal with recovery of broken filesystems in your setups?
Logical Volume Management (LVM) snapshots

Deleted files: Deleting files is just flagging them in the index and their disk space is marked as available again. The data is still where it was - until a new write operation (=creating new data) uses the same space and overwrites the old data.

Related articles:
- Get Your Data Back with Linux-Based Data Recovery Tools
- [How to Install and Use TestDisk Data Recovery Tool in Linux]((https://www.tecmint.com/install-testdisk-data-recovery-tool-in-linux/)
- Scalpel: Extract lost files from disk/card image
Different FS have different error resilience and recovery options: Go ask your IT guys! ZFS is pretty awesome, but “simple and well known” is also nice for recovery.
FS recovery requires understanding the FS format. Got documentation? Source code?
Logical Volume Management (LVM) Remember VM-snapshots? You can do the same on storage level (below filesystem): LVM Snapshots
Does it scale?
- Total number of files/folders?
- Length of path?
- Suitable for NAS/SAN/cluster?
- FS vs Blocks vs Object-based?

Errors? Backup!

Production Backup

The 3-2-1 Backup Rule

Keep at least three copies of your data.
Store two backup copies on different devices or storage media.
Keep at least one backup copy offsite.

Statistics of HDD failure

https://www.backblaze.com/blog/how-long-do-disk-drives-last/

The Story of ToyStory2

https://www.youtube.com/watch?v=8dhp_20j0Ys

Data errors

Digibeta Dropouts

Audio Bit-Errors

Small Bit, Big Problem

JPEG header “signature” = FF D8 FF DB

0xFF = 0b11111111
0xBF = 0b10111111

Corruption / Bit rot

“Bit rot can be caused by a number of sources but the result is always the same – one or more bits in the file have changed, causing silent data corruption. The ‘silent’ part of the data corruption means that you don’t know it happened – all you know is that the data has changed (in essence it is now corrupt).” — Jeffrey B. Layton, Linux Magazine, June 2011

Data scrubbing, Fixity checking

How do you know your data is intact?

A check a day keeps the bitrot away…

Headcount

Hashcode manifest files can also be used to check if all files expected are present or additional ones exist that are unaccounted for.

Storage: Challenges / Risks

Storage media failure
Obsolescence
Humans
Catastrophes / War

RAID

Redundant Array of Inexpensive Disks
Example of a disk chassis

“[…] is a data storage virtualization technology that combines multiple physical disk drive components into one or more logical units for the purposes of data redundancy, performance improvement, or both.”

Source: Wikipedia: RAID

RAID

Different RAID levels:
- Stripe = Fast, but dangerous!
- Mirror = Perfect for OS system disks
- RAID5/6 = 1 or 2 disks fault tolerance (parity bits)
RAID is not a backup.

Sad, but true:

With > 4TB disks, RAID-6 may be insufficient…
Long rebuild times may “burn out” the left-over disks.
ZFS supports 3 disks parity, but … future?

The algorithm RAID-6 uses to provide data redundancy for rebuilding in case of harddisks failing, may have come to an end with HDD capacities beyond about 4TB per-disk.

The reason for that is, that in case of a disk failure, the RAID mechanism reads the parity data from all remaining disks to reconstruct the one that “has gone”. With larger disks, the time for this rebuild to happen is getting longer. During that period, all remaining disks are constantly “stressed” until the required data has been read and the new disk rewritten.

Considering that disk failures more likely occur in RAID arrays that have been running for quite some time, it can be assumed that the other disks may be stressed too hard during that rebuild, causing other disks to fail due to the recovery - resulting in a complete data loss.

Cluster & replication

Multiple copies on multiple servers
(“cluster nodes”)
Synchronized (replicated) automatically
Off-site replication possible
Replication is not a backup!
What if a node goes down?
How does “the cloud” do it?

Inside = The same stuff

Combining a mix of most (if not all) the above mentioned technologies.
Mostly unix-like operating systems (e.g. Linux).
And lots of carriers.
maintaining it = other people’s problems.

So what if cluster nodes fail?

Another node takes over.
Seamlessly. Transparently.
The faulty node is removed & replaced.
Life continues.

Erasure Coding

“protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in MinIO erasure code you can lose as many as half of drives and still the data remains safe.”

Source: MinIO Erasure Code Quickstart Guide

“Compared to data replication, erasure-coding approaches have better performance at reducing storage redundancy and data recovery bandwidth.”

Source: Reliability Assurance of Big Data in the Cloud (2015)

Erasure coding is a different (algorithmic) approach to provide data redundancy to counteract device/node failures.

Quote from the Minio website:

“Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. MinIO uses Reed-Solomon code to shard objects into variable data and parity blocks. For example, in a 12 drive setup, an object can be sharded to a variable number of data and parity blocks across all the drives - ranging from six data and six parity blocks to ten data and two parity blocks.

By default, MinIO shards the objects across N/2 data and N/2 parity drives. Though, you can use storage classes to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures.

In 12 drive example above, with MinIO server running in the default configuration, you can lose any of the six drives and still reconstruct the data reliably from the remaining drives.”

More math at: Wikipedia: Erasure Code

Quote from Wikipedia:

“Example: In RS (10, 4) code, which is used in Facebook for their HDFS,[4] 10 MB of user data is divided into ten 1MB blocks. Then, four additional 1 MB parity blocks are created to provide redundancy. This can tolerate up to 4 concurrent failures. The storage overhead here is 14/10 = 1.4X.

In the case of a fully replicated system, the 10 MB of user data will have to be replicated 4 times to tolerate up to 4 concurrent failures. The storage overhead in that case will be 50/10 = 5X.

This gives an idea of the lower storage overhead of erasure-coded storage compared to full replication and thus the attraction in today’s storage systems.”

Erasure Coding

More parity possible than RAID
Data can be spread across network cluster nodes.
Does not suffer from rebuild “burn out”.
More complex to set up & compute.
Not that widely known/supported/used yet.

Object storage

Files become “objects”.
There are no folders ^{(as we’re used to)}
There’s just an identifier.
Arbitrary metadata may be assigned to each “object”.
Scales infinitely (theoretically).
Clusters of block storages with a clever abstraction layer.

Object storage

Beyond classical, hierarchical filesystems

Object storage = The future?

Object based storage may very likely replace hierarchical filesystems, but things need to be rewritten/adapted to properly support it.

It’s not yet plug-compatible with existing programs.

Object Storage = MAM?

The object filesystem is an actual catalogue
If implemented well, the storage itself may be a future DAM/MAM.
Links/relationships between objects = 🤯🤠🥳
Combined with LoD, MD Standards & APIs = Wikidata for your files.
No more distance between file and its catalogue entry.

Just an idea so far, but maybe…?

There is great potential in Object Based Storage.

OBS are already in production usage for a few years by major (online cloud) storage service providers. Yet, the “catalogue” world, including the WWW are still mostly running and “thinking” in hierarchical filesystems. DAMs and MAMs are currently a middleware layer for trying to link files and “other stuff related to them” in a way that solely serves: Search and Retrieval.

A basic condition for this to work well is that it is implemented using open and interoperable standards.

Then, any metadata, any relationship - and additional description or history that would be assigned to a data-file over its lifecycle would simply accumulate as part of its object-record. On filesystem-level.

The default file manager would function as a very basic asset-management system. It already does so, but in a crude, not-so-interoperable way. And the metadata fields available are currently very file-format related, since often stored inside the file itself.

If moving the paradigm from “metadata in a file”, plus the “rest in a separate, decoupled database” to a “no metadata inside the file, since the file is now an object, and that object can have certain basic metadata fields by its new”nature”:

objects can be file-data or metadata-only.
+keywords / tags
+name = value pairs (as much as your hardware can handle)
+relationships between objects (as many as your hardware can handle)
relationships are themselves just regular “objects”

This is currently just an idea…

Software Defined Storage (SDS)

Again: Onions, eh Layers.

The Cloud

No internet = no service.
Make sure your online bandwidth is sufficient.
Have an exit plan (+contract?) for migration.
What if their conditions change?
Where is your data actually stored?
Does it matter for you? ^{(e.g. legally)}
Again: Consider mixing.

Does it scale?

Does it scale?

What if you run out of diskspace?
Need to copy/move everything?
Or can you simply add “more space”?
How to know if everything is still “there” and intact?

More Storage Terms

S.M.A.R.T. Self-Monitoring, Analysis and Reporting Technology
NAS Network Attached Storage
SAN Storage Area Network
RAID Redundant Array of Inexpensive Disks
Object Storage

Good practice

At least 1 backup (=2 copies)
Preferrably 2 backups (=3 copies)
Geographic separate locations (with different threat profile)
Mix storage media
Migrate timely and with a plan (~5-7 years)
Use the right tech for your needs
Periodically check fixity of content and backup
Work with IT to implement and maintain technology.

Topic 5 - Archival Storage

Reality check

Physical Media Types

Optical Disks

HDD: Hard Disk Drive

Data Tape

Flash Memory

Media Types: Overview

HDD vs SSD

Good to know:

Storage Types

External Hard Disks

External Hard Disks

Network Attached Storage (NAS)

Network Attached Storage (NAS)

Data Tape

Data Tape

Data Tape Library (Robot)

LTO Generations

The Cloud

The Cloud

Storage Debate!

Networks

Consider: Layers!

The File System

The File System?

LTFS: Linear Tape File System

File system: Disaster relevant?

Errors? Backup!

The 3-2-1 Backup Rule

Statistics of HDD failure

The Story of ToyStory2

Data errors

Digibeta Dropouts

Audio Bit-Errors

Small Bit, Big Problem

Corruption / Bit rot

Data scrubbing, Fixity checking

A check a day keeps the bitrot away…

Headcount

Storage: Challenges / Risks

RAID

RAID

Sad, but true:

Cluster & replication

Inside = The same stuff

So what if cluster nodes fail?

Erasure Coding

Erasure Coding

Object storage

Object storage

Object storage = The future?

Object Storage = MAM?

Software Defined Storage (SDS)

Again: Onions, eh Layers.

The Cloud

Does it scale?

Does it scale?

More Storage Terms

Good practice

Comments?

Questions?

Links