Hello! :)

DLTP?

“Just store them damn files. Done. Next. Why all that fuzz? Pfff. Nerds.”

What would you like to know/discuss?

Things to think about

There is no evergreen format.
There is no one-size-fits-all.
Don’t panic. Don’t slack.
Less is more, but too little is no good.
Format obsolescence/support is a decision.
Convenience is a powerful drug.
We have no perception senses for digital.
Everything must sooner or later be migrated.
Exchange and share with others.

Know your…

users
staff
strength / limits
wishes and priorities
what to hold on to - and when to let go
policies ;)

And don’t let “Perfect be the enemy of good”.

Good questions?

What you are collecting (and what not)?
Why you are doing that?
Whom are you serving?
Which level of “long term” do you aim at?
Got enough/right resources?

Helps to decide & move

“Erstens lernt man seine Nutzergruppen kennen und kann seine Arbeit priorisieren. Beides erleichtert die Zielgruppenanalyse, d.h. die stetige Überprüfung, ob man als Einrichtung auf die Anforderungen seiner Nutzer eingestellt ist und ggf. Korrekturen vornehmen sollte.”

Source: Nestor mat19, p.106

Data Integrity?

Digibeta Dropouts

Audio Bit-Errors

Small Bit, Big Problem

JPEG header “signature” = FF D8 FF DB

0xFF = 0b11111111
0xBF = 0b10111111

What is “Fixity” information?

Metadata you can use to know if your bits are unchanged.
Stored in so called “manifests”.

Hashcodes

raw.txt

“This is a raw text file.”

MD5 = b3a243d2443037a783c8799fe2c4926a

Hashcodes

raw.txt

“This is a raw text file.⎕”

MD5 = 7096384353da7d8cb59b1395e63d1250

Hashcodes

raw.txt

“this is a raw text file.”

MD5 = a94a15d1b72bbfee7997bf237cf0347e

Hashcodes

raw-text.txt

“this is a raw text file.”

MD5 = a94a15d1b72bbfee7997bf237cf0347e

Different algorithms

CRC
MD5
SHA .. 1 .. 2 .. 256 .. SHA512?
WTF?

There are different hashing algorithms with different properties (pros/cons).

In a nutshell:

Shorter hash = faster(*)
but higher collision chance
= less secure
Longer hash = slower(*)
but lower collision chance
= more secure

For data integrity verification, short hashes are perfectly fine.

Since hashing algorithms are also used for security purposes (digital signatures), MD5 was said to be “broken”. This is only true for security/signature purposes. It is still perfectly valid for checking data integrity.

Hashcodes are fixed-length numbers that are generated in a way that if even a single bit in the source data changes, that number will be completely different. They are often written in hexadecimal, therefore including the characters 0-9 and A-F.

If you have a hashcode for a set of data, it can be used to verify the bit-exact integrity of that data, by calculating the same-algorithm hashcode again and comparing them. If they’re identical, the data is intact. If two distinct sets of data have the same hash, it’s called a "hash collision. Hash algorithms are designed particularily to keep the chance for collision as low as possible.

Anyways: hashcodes are a must for safe data transfer and integrity checks.

In case someone has heard that MD5 is broken, fear not: For plain checking of file transfer or stored data integrity, MD5 is sufficient. It was cryptographically “broken” - which is relevant for security, but not for data integrity checking.

Important: Hashcodes are not sufficient for proving authenticity! In case you need to deal with important originals/documents and you need to make sure they’re originals and not forged, etc. please check out this:

Digital signatures are good, but could be signed with a date in the past (backdating).

If this is a concern, you may consider blockchain mechanisms: Blockchain cipher became popular for digital currency (like Bitcoin), but it can also be used to proof data authenticity and avoid backdating.

AFAIK there are currently no systems that implement this productively yet, but some have already researched into prototyping blockchain use for storage.

Hashcode Examples

CRC =
4294967295
MD5 =
d41d8cd98f00b204e9800998ecf8427e
SHA256 =
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
xxHash =
e4c191d091bd8853

Algorithms in order of complexity/size: > CRC —— MD5 —— SHA256 —— SHA512

Speed matters when it comes to calculating hashes for several hundreds of TB or PB of data.

Execution speed depends on the actual implementation of the algorithm. Even if a simpler algorithm may be faster in theory, it may not make a difference if the implementation isn’t speed-optimized. However, speeding up hashing becomes more and more interesting e.g. for transfer of digital cinema files, because validating these data amounts may currently be a bottleneck.

Different hashcodes algorithms/implementations may have significantly different runtimes. When dealing with large quantities of data, this may matter.

MD5 is the most popular one around: Well known and widely supported by different applications/systems, etc.

Anyone transferring lots of uncompressed film? You may want to look at xxHash. It’s designed for speed.

When?

Creating?
- Generate fixity information as early as possible in a file’s lifecycle.
Exchanging?
- Use fixity information to safely transfer any data safely.
Checking?
- Find sweet spot for continuous checks.

Tools

HashCheck Shell extension: code.kliu.org/hashcheck (Windows)
md5sum (cross-platform, CLI)
LoC “BagIt” bags (manifest specification)
Bagger, et al (cross-platform, GUI)
AVPreserve “Fixity” (cross-platform, GUI)
hashdeep (cross-platform, CLI)

File naming & structure

Make choices.

Establish file and folder naming conventions, identifiers, and storage locations which works/scales for your collection size/needs.

Larger systems: You have to let go.
The thing with “tricky” characters…

Tricky characters

What to look out for (=avoid) when naming files?

non-alphanumeric / non-ASCII
Spaces!
< > " / \ ? % : | * :
(total) string length

Why?

Why? Computer systems really really have problems with it. Even up to date. The filename was intended and designed as a handle where machine and human meet. Not a notepad ;)

Information that goes beyond the usage as a human+machine filename handle, but causes problems is IMO better put somewhere where special characters are kept and treated properly - like a database, or inside the file as embedded metadata.

Modifiying filenames to make them work more smoothly for computers can be a very emotional thing: Invisible Defaults and Perceived Limitations: Processing the Juan Gelman Files (by Elvia Arroyo-Ramirez, 2016)

I find it good that she highlights that lanuguage and expression is a very personal thing and represents cultural codings - and that it’s our duty as archivists to safeguard what is handed over to us as original as we can.

She also stumbles over incompatibilites between computer (operating) systems.

When reading the article I felt that she was putting blame on the Linux system for not treating certain encodings well, when in fact it’s the other way around: Windows introduced its own codepages, and also they didn’t configure their Linux system correctly so it would read these characters properly.

I want to add though, that actually it’s open source systems that support the biggest number of languages and encodings that exist worldwide: Especially the ones that are not as “commercially interesting” for other systems to really put effort it.

Keep and guard the materials handed over to you as best as you can, yet be aware that computers that use filesystems have a hard time dealing with certain characters in filenames. Preserve the text of the original filenames, but maybe not necessarily in the filename, but rather as metadata somewhere where it doesn’t cause machines to freak out.

“Stabilizing” Filenames

A filename is not a catalogue. Replacing (or removing) potentially “problematic” characters from file- and foldernames makes sense.

…but document it!

Personal/small collections?

Sortable timestamp
20190618_103918-short_description.jpg (so alphabetical = chronological)
Consistent naming per folder? IMG_0801.JPG, SL740465.jpg, DSCN7717.JPG, …
Folders = groups, collections
Hierarchical folders:
2021/09-Event/person_A
(=YYYY / MM-event / record_source)

Medium/large collections?

You’ll have to let go. At least somewhat.

Not really human readable
(UUID = 6b172fc3-3675-450e-b698-d5dbd47e23fa)
Structure, name, location defined by algorithms
Reference in database

Without that DB, you’re lost. Backup!

Born digital vs Digitized

Workflow differences?
Format normalization?
In the end they’re all files.

Digital: Film vs video

Resolution?
Framerate?
Color information?
Interlaced vs progressive?
Size?
Different tools and data formats?
2k (2048 x 1080) vs Full HD (1920 x 1080)?

It’s difficult, but it’s easy.

Somewhat. ;)

Digital Video

1 GBE Network
50/100 Mbps I/O throughput (broadcast quality)
650 MB/s lossless (SD)
IMX/PCM/MXF, FFV1/PCM/MKV
Insane variations of formats
preservation/[mezzanine]/access copy
4k too now :(

Digital Film

10 GBE Network
Up to 3.82 Gbps (4k, 12bpc RGB, 24fps)
High data throughput demand
Folder-handling: Image sequence + audio + sidecar
Limited, but island-supported variations of formats
Multiple copies: overscan, color-graded, etc.

Digital: Film vs video

Film:

Images + Audio + folders = preserves well.
but handles harder.
Professional bubble = less crazy variations.

Video:

Format variations/dialects should be streamlined.
AV streams may be full of surprises ;)
Broader user base = crazy variations!

Common:

Validate technical properties (e.g. MediaConch)
Deal with large number of files.
Add/use fixity information.
Use open formats.
ffmpeg (conversion/access)

Metadata

Lack of.
Too much. (Example: onion-layered standards overkill)
Plain wrong (because noone checked)
Use standards. (if reasonable/possible)

Metadata formats

TXT vs XML vs PDF vs DOC(x)?

As plain text as possible.
Machine readable/parseable. Structured.
DOC(x) and PDF don’t parse well.
As simple as possible, as complicated as necesary.

Storage

“Just storing files” is not preservation.
Storage market focus may deviate from your use case.
There’s more than just one right solution…
Mixing is usually a good idea.

Storage debate

The Cloud?
In-house?
Outsourced?

Media Types: Overview

Classic “Spinning Platters” harddrive LTO: Linear Tape Open cartridge Optical Disks SSD: Solid State Disk (no cover) SD cards

The File System?

Raw storage = unstructured chain of bits
File system (FS) = file/folder structure
Different FS = different data formats:
- FAT{16,32} (Microsoft)
- NTFS (Microsoft)
- EXT{2,3,4} (Linux *)
- ZFS (Sun *)
- HFS+ (Apple)
- LTFS (IBM *)

(*) Open formats

FAT: * FAT introduced in 1977. * Widespread, well-known and documented * Default for SD cards and embedded devices * FAT32: 32bit limitation (max 4GB filesize, limited max. device size) * No access permissions (=everyone can read/write everything) * Very simple and straightforward filesystem * Non-free / license fees by Microsoft.

NTFS: * Introduced in 1993. * max volume size (different implementations): 256 TiB (pre 2016), now 8 PiB (2019 or later) * Non-free / license fees by Microsoft.

EXT: * ext2 introduced in 1993. * ext4 standard for Linux-based OS. * ext4 and NTFS supported by digital cinema standard. (Because the projectors probably run Linux ;)) * max volume size: 1 EiB (1024 PiB) * No licensing fees

ZFS: * Introduced in 2005 * More than “just a filesystem”: combines RAID (redundancy, handle harddisk failure) Logical Volume Management (LVM) and filesystem in one. * Integrated integrity checks * (recently) well known among IT administrators of unixoid systems. * Requires slightly more experience to handle (because of complexity. * Very interesting filesystem (also for preservation). Too much to describe here. You may want to read more on Wikipedia :)

HFS+: * HFS Introduced in 1985. * max volume size: 8 EB * Successor of “HFS” * Officially only supported on Apple OS

LTFS: * LTFS first prototype (Linux and Mac OS X) during 2008/2009 * Only for LTO tapes. * See separate page/slide on LTFS for details.

LTFS: Linear Tape File System

Open specification = vendor neutral
Better for preservation, but may not support “convenience” features.
All implementations must:
- Correctly read media that was compliant with any prior version.
- Write media that is compliant with the version they claim compliance with.

The Linear Tape File System (LTFS) is a good solution to avoid vendor lock-in by tape filesystems. Please make sure that you can read the data written onto the tapes under the following conditions:

Different drive
From different vendor
With (FOSS) software (and/or from different vendor)

Otherwise, you won’t really know if you can access your data as technology- and vendor-neutral as desired.

By default, vendors may write data in their own way on LTO tapes. This allows them to implement some non-standard features that often provide more “comfort” by offering additional features. Using such vendor-specific features, instead of LTFS has the downside of reducing the possibilites to read that data under different conditions in the future.

Therefore it is advised to use LTFS for long term preservation use cases.

Decisions & planning

Multi-tier strategy:
online, nearline, offline
fast=expensive, slow=cheaper
Size demands:
Calculate filesize and expected quantity.
Does it scale?
How well/easily/affordable?
Consider all layers:
distributed, filesystem, drive, physical carrier

Storage checklist

What’s the desired level of long-term?
Which carrier type(s) are you storing on?
Which use cases does it need to address?
Which functions are in place?
(access control, monitoring, integrity checks, etc)
Which automated actions are in place/desired?
How can it be scaled if you run out of space?

Errors? Backup!

Production Backup

The 3-2-1 Backup Rule

Keep at least three copies of your data.
Store two backup copies on different devices or storage media.
Keep at least one backup copy offsite.

The Story of ToyStory2

https://www.youtube.com/watch?v=8dhp_20j0Ys

Backup checklist

How is the backup/restore done?
Who takes care of the backup?
Who can do a restore?
Is the restore being tested before needed?
When/why is a backup restored?
How can integrity of backup be verified?
How long is the expected “restore downtime”?

Migration

How long is “long term”?

Eternal Migration

There is no final carrier.
There is no evergreen format.

Therefore fact: Any data must sooner or later be migrated.

Oh, btw: this also applies to everything in your workflow/toolchains.

Migration

Prefer modular (LEGO) over monolithic (Playmobil).
Schedule it before it’s “necessary”.
Consider impact on daily operations:
downtime, performance, staff, …

LTO Generations

New: About every 2-3 years.
Shelf vs robot
Got drives?
Got the right interfaces/cables?
Got time?

Policies

First of all: Don’t be intimidated. If it feels too much, it’s already a good start to write it down for yourself.

Anything is better than nothing.

Digital Preservation Policy

“A guideline that describes the essential setting, principles, structures and objectives of a digital archive”

Technical Properties Policy

Define which technical properties you accept/support:

Data Formats codecs, container, metadata, …
Resolution:
image, fps, depth, samplerate, …
etc.

Written? Published?

Writing it down:
- Highlights fuzzy assumptions
- and “unknown-unknowns”
Publishing:
- Profit from insights into other’s decisions.
- Reference for argumenting (decisions, resource needs).
- Extra community-karma points!

- FIN -

Questions & Comments welcome! :)

Peter Bubestinger-Steindl
p.bubestinger@av-rd.com