Storage

Considerations

Storage market focus may deviate from your (=preservation) use case.
Business “longterm”: 3-5 years (=product lifecycle)
Cross vendor compatibility? Standards? Documentation?
Spare parts?
This market focus affects prices, features and reusability.

Physical carrier

Classic “Spinning Platters” harddrive LTO: Linear Tape Open cartridge Optical Disks SSD: Solid State Disk (no cover) SD cards

Different carrier media have different properties when it comes to error resilience, longevity - and of course: price.

Magnetic: Susceptible to electro-magnetic issues.
Optical: light, temperature, warping
Electronics: Failing capacitors
Mechanics: Wear and tear
Supported/obsolete interfaces or drives?

It’s usually good to mix storage media to distribute their pros/cons properties.

I won’t go into great detail here, as this is possibly covered in many other preservation-storage presentations.

This here is for the interested reader.

Examples for different use cases:

preservation / backup:
tape (affordable size, good preservation properties, less need for super-fast access)
access: No need for long-term endurance, so e.g. optical would be sufficient.
production: HDD-based storage for nearline files - tape for offline storage. Or: tape-robot with HDD cache (same concept, but fully automated)

More properties:

Data Access Speed?
SSD > HDD > Tape > Optical
Cost/size ratio?
SSD > HDD > Tape > Optical
Widespread?
- SSD, HDD: Excellent!
- Tape: almost exclusive professional use case Drives not that “easy” to get (for particular LTO generation)
- Optical: Declining (except for “cold storage” in large data centers)

In greater detail:

Optical Disks

Pros:

Cost/size ratio: Relatively cheap (cheapest still?)
Not susceptible to electro magnetic (dis)charge
No mechanics/electronics in carrier - only in the drive

Cons:

Error prone
Suffers easily from material degradation (scratches, data layer falling off, light exposure, temperature, etc…)
Becoming less and less common = less support, less choice, etc.

HDD: Hard Disk Drive

Pros:

Reasonable cost/size ratio
(yet) cheaper than SSD
well known, well supported

Cons:

Mechanical wear (moving parts)
Susceptible to electro magnetic discharge
Electronics on carrier

Data Tape

Pros:

Cost/size ratio: Tape cartridge “cheaper” than HDD
Tape preserves well (and there is much experience with dealing with tape degradation/aging from analogue)
Supported in “tape libraries” (aka “tape robots”) for removing “tape-jockey” limitations.
Larger capacity per cartridge than per HDD (yet)

Cons:

Not common outside certain domains (large institutions, broadcast preservation, …)
Therefore support for tapes very “specialized”
Drives quite expensive (>3000 EUR)

Flash Memory

Pros:

cost/size ratio: getting cheaper, but still more expensive than HDD
No moving parts = no mechanical wear
SD cards: almost no electronics in the carrier
Speed: SD cards are not sooo fast, whereas SSDs (or M.2) are extremely fast (compared to HDD).

Cons:

Currently most expensive (still)
Lack of long-term experience
Limited number of write/erase cycles per flash block (becoming less of an issue) =Should not be used for large number of write operations. But preservation storage case is mostly “read”.

More about Flash memory on Wikipedia. SD card image by Steve Meirowsky (Wikipedia) - Own work, CC BY-SA 3.0

HDD vs SSD

Good to know:

Higher data density = more impact of an error.
Example: 1mm hole in CD vs DVD vs BluRay - or SD vs MicroSD, etc.
HDD: The longer it is active, the shorter is lives.
But: “Hardware that lies, dies.”
Be aware if your HDD is “Shingled (SMR)” or not.

Density:

Think of a 1mm hole on an optical carrier: CD vs DVD vs BluRay
All 3 have the same dimension, but different data density.
When choosing a carrier: Do you really need it to be as small/dense as possible?

Hardware that lies, dies:

Got external HDDs stored on shelves?
You may want to power them up every once in a while (every 1-2 years or so), and copy the data off them before the hardware gets old. Because: Electronics fail (capacitors), lubricants harden out or movable parts get “lazy” or stuck.

Das Werkstatt Rule #2 (German): “Hardware die liegt, stirbt” Useful “Das Werkstatt” principles

Shingled HDD:

“The more time a shingled disk spends in band compaction, the more it is wasting resources.”

Source: Classifying Data to Reduce Long-Term Data Movement in Shingled Write Disks This means, SMR disks are active when idle, causing them to potentially wear out faster.

Tape & Drives

LTO Generations

Imagine you find an old LTO tape…

Not every drive can read every tape.
New LTO generation release: ~every 2-3 years.
< LTO-7: Read=2 gen. / Write=1 gen.
BUT: LTO-8: Read/write=1 gen.!

The File System

Popular File Systems

FAT{16,32} (Microsoft)
NTFS (Microsoft)
EXT{2,3,4} (Linux)
ZFS (Sun)
HFS+ (Apple)
LTFS (IBM)

FAT:

FAT introduced in 1977.
Widespread, well-known and documented
Default for SD cards and embedded devices
FAT32: 32bit limitation (max 4GB filesize, limited max. device size)
No access permissions (=everyone can read/write everything)
Very simple and straightforward filesystem
Non-free / license fees by Microsoft.

NTFS:

Introduced in 1993.
max volume size (different implementations): 256 TiB (pre 2016), now 8 PiB (2019 or later)
Non-free / license fees by Microsoft.

EXT:

ext2 introduced in 1993.
ext4 standard for Linux-based OS.
ext4 and NTFS supported by digital cinema standard. (Because the projectors probably run Linux ;))
max volume size: 1 EiB (1024 PiB)
No licensing fees

ZFS: * Introduced in 2005 * More than “just a filesystem”: combines RAID (redundancy, handle harddisk failure) Logical Volume Management (LVM) and filesystem in one. * Integrated integrity checks * (recently) well known among IT administrators of unixoid systems. * Requires slightly more experience to handle (because of complexity. * Very interesting filesystem (also for preservation). Too much to describe here. You may want to read more on Wikipedia :)

HFS+: * HFS Introduced in 1985. * max volume size: 8 EB * Successor of “HFS” * Officially only supported on Apple OS

LTFS: * LTFS first prototype (Linux and Mac OS X) during 2008/2009 * Only for LTO tapes. * See separate page/slide on LTFS for details.

LTFS: Linear Tape File System

Open specification = vendor neutral
Better for preservation, but may not support “convenience” features.
All implementations must:
- Correctly read media that was compliant with any prior version.
- Write media that is compliant with the version they claim compliance with.

The Linear Tape File System (LTFS) is a good solution to avoid vendor lock-in by tape filesystems. Please make sure that you can read the data written onto the tapes under the following conditions:

Different drive
From different vendor
With (FOSS) software (and/or from different vendor)

Otherwise, you won’t really know if you can access your data as technology- and vendor-neutral as desired.

By default, different applications may write data in their own way on LTO tapes. This allows them to implement some non-standard features that often provide more “convenience” by offering additional features. Using such vendor-specific features, instead of LTFS has the downside of reducing the possibilites to read that data under different conditions in the future.

Therefore it is advised to use LTFS for long term preservation use cases.

File system: Disaster relevant?

Deleted files are still there (maybe fragmented).
Different FS = different error resilience and recovery options.
Does it scale? (moving files is when they’re most vulnerable…)
Tools/knowhow to deal with recovery of broken filesystems in your setups?
Logical Volume Management (LVM) snapshots

Deleted files: Deleting files is just flagging them in the index and their disk space is marked as available again. The data is still where it was - until a new write operation (=creating new data) uses the same space and overwrites the old data.
Related articles:
- Get Your Data Back with Linux-Based Data Recovery Tools
- [How to Install and Use TestDisk Data Recovery Tool in Linux]((https://www.tecmint.com/install-testdisk-data-recovery-tool-in-linux/)
- Scalpel: Extract lost files from disk/card image
Different FS have different error resilience and recovery options: Go ask your IT guys! ZFS is pretty awesome, but “simple and well known” is also nice for recovery.
FS recovery requires understanding the FS format. Got documentation? Source code?
Logical Volume Management (LVM) Remember VM-snapshots? You can do the same on storage level (below filesystem): LVM Snapshots
Does it scale?
- Total number of files/folders?
- Length of path?
- Suitable for NAS/SAN/cluster?
- FS vs Blocks vs Object-based?

Linking Metadata and Content

Metadata: Database (DB)
Content: Storage
Connection: By identifier and/or path.
Stored in DB.

File-naming and structure

Høvegaard, Björn Clément/16mm 571,1 Jahr 1978 - Mågst im Wald tanz’n - MPEG-4 ProRes.mov
f23bfeb9-7558-4a9c-bfb4-dd2d5a0409de.mxf
VX/00/VX-00815/vx-00815.mkv

Explain pros/cons of the naming examples.

Notes on the above examples:

Based on an actual real-world filename.
- DON’T! This is one of the worst cases ever!
- Big problems dealing with these files! (Access, conversion, migration, etc)
- Do: Stay with alphanumeric characters.
- Do: Avoid spaces.
- Don’t: Codec/format in preservation filenames. It will change! And then it’ll be wrong. If you fix the filename: All references to it will be invalid.
- Don’t: Titles will contain bad characters at some point.
UUID

Just an example, what a filename generated by a DAM/MAM may look like. It’s uniquely identifiable by a system, but a bit hard for humans. This is fine, as long as the data to correlate your files with your metadata is intact.
Example of naming persistency

The signature syntax is taken from the Austrian Mediathek. The identifier (Example: VX-00815) is short enough to be memorized by a human, and has a syntax that allows to identify the media type somewhat (V=VHS, VX=other video).

The path/filename structure also has a syntax rule that is used to split up the sub-directories, according to the item’s signature. This allows to avoid too many items/files in a single folder, but in a way that human (administrators) can still deal with the structure.

The catalogue/DAM knows the identifier and can then link to the files on the storage, according to these rules.

This also allows access to the files outside and independent of the catalogue/DAM, for any other reasons. Popular use cases are: automated format migrations, digitization ingest systems from different vendors/different media types.

It does have some drawbacks and doesn’t scale infinitely, but it’s a nice example that allows tool/OS/filesystem/vendor agnostic operations and has survived several migrations (tool/OS/filesystem/vendor) successfully.

Dos, Donts, Letgos

Do:
- Naming persistency, syntax rules
Don’t:
- Non-alphanumeric characters + spaces
- Codec / format names
- Title, Author, etc.
Let go:
- Human graspable names…

File-naming and structure

Relevant for:
- Tool/OS/Filesystem agnostic operation
- Migration
- Disaster recovery: find and assign files to their catalogue metadata.
Most important:
- Naming persistency: “Ensure the persistence of the originally-applied file name. Trying to rename files causes all sorts of administrative and technical problems and, if not handled careful, can lead to synchronization errors between a file and and the external database record that holds descriptive information about the file.”
- Naming syntax rules: No rule/system means that all files will have to be linked quasi manually with a catalogue/DAM. This has so many drawbacks and causes so many issues, I don’t even know where to begin.
- Avoid problematic characters (and spaces!). Seriously.
Nice to know? “Detox” “Detox is a utility designed to clean up filenames. It replaces difficult to work with characters, such as spaces, with standard equivalents. It will also clean up filenames with UTF-8 or Latin-1 (or CP-1252) characters in them.”
Let go: Beyond a certain number of assets, the names will have to be generated by rules, not human-readable anymore. This can feel very uncomfortable, because you lose control. You have to let go. Many DAMs auto-generate the filenames and folder structure. But this means: Your files are now owned by the DAM (vendor). Do you have plan what happens if you and the DAM go separate ways?
Possible consequences:
- lose MAM/DB = lose order = chaos

Files intact, but MAM gone?

Store identifying metadata

All in one?

Failsafe mechanisms

S.M.A.R.T.

Self-Monitoring, Analysis and Reporting Technology

“[…] is a monitoring system included in computer hard disk drives (HDDs), solid-state drives (SSDs),[1] and eMMC drives. Its primary function is to detect and report various indicators of drive reliability with the intent of anticipating imminent hardware failures.”

Source: Wikipedia: S.M.A.R.T.

S.M.A.R.T. Graph

RAID

Redundant Array of Inexpensive Disks
Example of a disk chassis

“[…] is a data storage virtualization technology that combines multiple physical disk drive components into one or more logical units for the purposes of data redundancy, performance improvement, or both.”

Source: Wikipedia: RAID

RAID

Different RAID levels:
- Stripe = Fast, but dangerous!
- Mirror = Perfect for OS system disks
- RAID5/6 = 1 or 2 disks fault tolerance (parity bits)
RAID is not a backup.

Hard- vs Software RAID?

Hardware:
- “easier” to setup and swap disks.
- Risk of vendor lock-in.
- Caution: Mainboard controllers are often “Pseudo-RAID”
Software:
- More complex to set up and swap disks.
- Linux kernel softraid: Very portable!
- Speed concerns with modern hardware possibly neglectible.

Linux mdadm RAID:

Easily portable/compatible/reconstructable
Well documented.
Production stable for many years.

Windows software RAID:

Surprising, but it seems to be quite undocumented. There seem to be no official data recovery tools (or instructions) provided by Microsoft.
I’d only use it to mirror devices.

Proprietary hardware/firmware RAID issue:

Often proprietary data layout, which does not guarantee that the disks can be read in a device/setup by another vendor - or different firmware/model.

If you don’t have matching spare parts on-site, your downtime is until you get one. If the unit was older than, let’s say 3 years, it’s not even sure if you get a compatible replacement anymore. This often leads to a forced upgrade of the whole hardware RAID.

Hardware RAID:

Requires hardware controller. Sometimes it’s an embedded external system.
“Easier” to configure and swap disks
No impact on the host CPU/RAM, since everything’s handled in the RAID hardware.
Completely transparent to the host. “Feels” like a physical disk.
Used to be necessary, before PCs got fast enough for Software RAID.
Host cannot access the individual drives directly (e.g. cannot access SMART readings)
Uses the whole disk.

Software RAID:

Uses CPU/RAM of the host machine.
Flexible configuration, mixed devices.
Can be made on partition level.
Host can access the individual drives directly (e.g. allows SMART monitoring, etc)
Linux kernel: open, documented and portable data layout = plug and play on any hardware.

More detailed comparisons:

Hardware raid or software raid? Our take on the great religious debate (2012)
Update: Hardware vs Software RAID: The great debate. (2013)
https://www.cyberciti.biz/tips/raid-hardware-vs-raid-software.html (a bit aged, 2009)

Sad, but true:

With > 4TB disks, RAID-6 may be insufficient…
Long rebuild times may “burn out” the left-over disks.
ZFS supports 3 disks parity, but … future?

The algorithm RAID-6 uses to provide data redundancy for rebuilding in case of harddisks failing, may have come to an end with HDD capacities beyond about 4TB per-disk.

The reason for that is, that in case of a disk failure, the RAID mechanism reads the parity data from all remaining disks to reconstruct the one that “has gone”. With larger disks, the time for this rebuild to happen is getting longer. During that period, all remaining disks are constantly “stressed” until the required data has been read and the new disk rewritten.

Considering that disk failures more likely occur in RAID arrays that have been running for quite some time, it can be assumed that the other disks may be stressed too hard during that rebuild, causing other disks to fail due to the recovery - resulting in a complete data loss.

Cluster & replication

Multiple copies on multiple servers
(“cluster nodes”)
Synchronized (replicated) automatically
Off-site replication possible
Replication is not a backup!
What if a node goes down?
How does “the cloud” do it?

Storage Debate!

Inside = The same stuff

Combining a mix of most (if not all) the above mentioned technologies.
Mostly unix-like operating systems (e.g. Linux).
And lots of carriers.
maintaining it = other people’s problems.

So what if cluster nodes fail?

Another node takes over.
Seamlessly. Transparently.
The faulty node is removed & replaced.
Life continues.

Erasure Coding

“protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in MinIO erasure code you can lose as many as half of drives and still the data remains safe.”

Source: MinIO Erasure Code Quickstart Guide

“Compared to data replication, erasure-coding approaches have better performance at reducing storage redundancy and data recovery bandwidth.”

Source: Reliability Assurance of Big Data in the Cloud (2015)

Erasure coding is a different (algorithmic) approach to provide data redundancy to counteract device/node failures.

Quote from the Minio website:

"Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. MinIO uses Reed-Solomon code to shard objects into variable data and parity blocks. For example, in a 12 drive setup, an object can be sharded to a variable number of data and parity blocks across all the drives - ranging from six data and six parity blocks to ten data and two parity blocks.

By default, MinIO shards the objects across N/2 data and N/2 parity drives. Though, you can use storage classes to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures.

In 12 drive example above, with MinIO server running in the default configuration, you can lose any of the six drives and still reconstruct the data reliably from the remaining drives."

More math at: Wikipedia: Erasure Code

Quote from Wikipedia:

"Example: In RS (10, 4) code, which is used in Facebook for their HDFS,[4] 10 MB of user data is divided into ten 1MB blocks. Then, four additional 1 MB parity blocks are created to provide redundancy. This can tolerate up to 4 concurrent failures. The storage overhead here is 14/10 = 1.4X.

In the case of a fully replicated system, the 10 MB of user data will have to be replicated 4 times to tolerate up to 4 concurrent failures. The storage overhead in that case will be 50/10 = 5X.

This gives an idea of the lower storage overhead of erasure-coded storage compared to full replication and thus the attraction in today’s storage systems."

Erasure Coding

More parity possible than RAID
Data can be spread across network cluster nodes.
Does not suffer from rebuild “burn out”.
More complex to set up & compute.
Not that widely known/supported/used yet.

Object storage

Files become “objects”.
There are no folders ^{(as we’re used to)}
There’s just an identifier.
Arbitrary metadata may be assigned to each “object”.
Scales infinitely (theoretically).
Clusters of block storages with a clever abstraction layer.

Object storage

Beyond classical, hierarchical filesystems

Object storage = The future?

Object based storage may very likely replace hierarchical filesystems, but things need to be rewritten/adapted to properly support it.

It’s not yet plug-compatible with existing programs.

Software Defined Storage (SDS)

Again: Onions, eh Layers.

The Cloud

No internet = no service.
Make sure your online bandwidth is sufficient.
Have an exit plan (+contract?) for migration.
What if their conditions change?
Where is your data actually stored?
Does it matter for you? ^{(e.g. legally)}
Again: Consider mixing.

Does it scale?

Does it scale?

What if you run out of diskspace?
Need to copy/move everything?
Or can you simply add “more space”?
What about your file/folder positions?
How to know if everything is still “there” and intact?

Data Integrity?

Fixity information & Hashcodes :)

Hashcodes: fixed size number that’s like a fingerprint for data.

CRC =
4294967295
MD5 =
d41d8cd98f00b204e9800998ecf8427e
SHA256 =
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
xxHash =
e4c191d091bd8853

Hashcodes are fixed-length numbers that are generated in a way that if even a single bit in the source data changes, that number will be completely different. They are often written in hexadecimal, therefore including the characters 0-9 and A-F.

If you have a hashcode for a set of data, it can be used to verify the bit-exact integrity of that data, by calculating the same-algorithm hashcode again and comparing them. If they’re identical, the data is intact. If two distinct sets of data have the same hash, it’s called a "hash collision. Hash algorithms are designed particularily to keep the chance for collision as low as possible.

Anyways: hashcodes are a must for safe data transfer and integrity checks.

Different hashcodes algorithms/implementations may have significantly different runtimes. When dealing with large quantities of data, this may matter.

MD5 is the most popular one around: Well known and widely supported by different applications/systems, etc.

Anyone transferring lots of uncompressed film? You may want to look at xxHash. It’s designed for speed.

In case someone has heard that MD5 is broken, fear not: For plain checking of file transfer or stored data integrity, MD5 is sufficient. It was cryptographically “broken” - which is relevant for security, but not for data integrity checking.

Important: Hashcodes are not sufficient for proving authenticity! In case you need to deal with important originals/documents and you need to make sure they’re originals and not forged, etc. please check out this:

Digital signatures are good, but could be signed with a date in the past (backdating).

If this is a concern, you may consider blockchain mechanisms: Blockchain cipher became popular for digital currency (like Bitcoin), but it can also be used to proof data authenticity and avoid backdating.

AFAIK there are currently no systems that implement this productively yet, but some have already researched into prototyping blockchain use for storage.

Do it early, do it continuously.

Create hashcodes early in the lifecycle.
Use them when transferring files.
Storage: Ongoing integrity checks matter.
Consider runtimes & interference with daily work.

Ongoing integrity checks on your storage: Know where your collection is at.

It also makes sense to verify your backup that way.

Consider runtimes:

In order to generate/verify a hashcode, the whole file must be read at least once.
The chosen hashcode algorithm and its implementation may have impact on processing speed.
Do some tests, calulations to find a sweet spot for how often you can do integrity checks.
Consider throttling throughput data to avoid interference with daily work.

Real world example:

Archival Institution had hashcodes (ingest system created them), but never used them. They told me: “We don’t need to verify hashcodes, we have no problems” So I checked. More than 1000 items had invalid hashcodes. Since they were never checked before, it is unknown when or what caused them - and also their impact on the material.

It’s good practice to document the result of integrity checks somewhere. It helps finding issues in your setup, or at least narrowing things down.

Hashcode manifest

As simple as that. Plain text.

Using (=validating) hashcodes

HashCheck: GUI tool showing data validation

Some tools

Our storage has integrity checks built in!

Good! 🥳
But it can only do so as soon as data has arrived.
Did you verify that the data got there intact? 🧐

Exercise

Make groups of 2-3 people each.
Download the fixity example zip:
http://download.av-rd.com/pub/exercises/fixity/hashcode-exercise_1a.zip
Extract the files.
Compare the manifests with the “reference” and try to spot issues (=differences).
Try to find out what caused these issues.
You may want to use a “diff” tool (e.g. MELD )

Backups

The 3-2-1 Backup Rule

Keep at least three copies of your data.
Store two backup copies on different devices or storage media.
Keep at least one backup copy offsite.

Backup integrity?

btw: Is your backup copy (still) intact…?
Did you check? ;)
Tape: Backup drives on shelf vs robot?
Handout: Backup checklist

Storage: Summary

Filenames matter.
There’s more than just one right solution…
Mixing is usually a good idea.
RAID6 might be at its limits :(
Consider LTFS and LTO generations
Embrace hashcodes!
Avoid black boxes.
Have a backup (plan)

- fin -

Questions? Comments?

Peter Bubestinger-Steindl

PB @ AV-RD.com

Storage Terms Collection

S.M.A.R.T. Self-Monitoring, Analysis and Reporting Technology
NAS Network Attached Storage
SAN Storage Area Network
RAID Redundant Array of Inexpensive Disks
Object Storage
JBOD Just a Bunch Of Disks
SDS Software Defined Storage

Managing and storing digital AV assets - Dos, Donts and Dunnos

Storage

Considerations

Physical carrier

HDD vs SSD

Good to know:

Tape & Drives

LTO Generations

The File System

Popular File Systems

LTFS: Linear Tape File System

File system: Disaster relevant?

Linking Metadata and Content

File-naming and structure

Dos, Donts, Letgos

Files intact, but MAM gone?

Store identifying metadata

All in one?

Failsafe mechanisms

S.M.A.R.T.

S.M.A.R.T. Graph

RAID

RAID

Hard- vs Software RAID?

Sad, but true:

Cluster & replication

Storage Debate!

Inside = The same stuff

So what if cluster nodes fail?

Erasure Coding

Erasure Coding

Object storage

Object storage

Object storage = The future?

Software Defined Storage (SDS)

Again: Onions, eh Layers.

The Cloud

Does it scale?

Does it scale?

Data Integrity?

Fixity information & Hashcodes :)

Do it early, do it continuously.

Hashcode manifest

Using (=validating) hashcodes

Some tools

Our storage has integrity checks built in!

Exercise

Backups

The 3-2-1 Backup Rule

Backup integrity?

Storage: Summary

- fin -

Questions? Comments?

Storage Terms Collection