Data Integrity?

What is “Fixity” information?

Hashcodes

Filename: plain.txt

“This is a raw text file.”

MD5 = b3a243d2443037a783c8799fe2c4926a

Hashcodes

Filename: plain.txt

“This is a raw text file.⎕”

MD5 = 7096384353da7d8cb59b1395e63d1250
REF = b3a243d2443037a783c8799fe2c4926a

Hashcodes

Filename: plain.txt

“this is a raw text file.”

MD5 = a94a15d1b72bbfee7997bf237cf0347e
REF = b3a243d2443037a783c8799fe2c4926a

Hashcodes

Filename: plain-text.txt

“this is a raw text file.”

MD5 = a94a15d1b72bbfee7997bf237cf0347e
REF = a94a15d1b72bbfee7997bf237cf0347e

Different algorithms

CRC
MD5
SHA .. 1 .. 2 .. 256 .. SHA512
XXHASH

There are different hashing algorithms. In a nutshell:

Shorter hash = faster
but higher collision chance
= less secure
Longer hash = slower
but lower collision chance
= more secure

For data integrity verification, short hashes are perfectly fine though.

Since hashing algorithms are also used for security purposes (digital signatures), MD5 was said to be “broken”. This is only true for security/signature purposes. It is still perfectly valid for checking data integrity.

“xxHash” is relatively new, and is the only “Non-cryptographic hash function” in the above list, designed for speed and not security. It is becoming more common for fixity checks in A/V production, but yet it’s still rather the exception.

Hashcodes are fixed-length numbers that are generated in a way that if even a single bit in the source data changes, that number will be completely different. They are often written in hexadecimal, therefore including the characters 0-9 and A-F.

If you have a hashcode for a set of data, it can be used to verify the bit-exact integrity of that data, by calculating the same-algorithm hashcode again and comparing them. If they’re identical, the data is intact. If two distinct sets of data have the same hash, it’s called a “hash collision. Hash algorithms are designed particularily to keep the chance for collision as low as possible.

Anyways: hashcodes are a must for safe data transfer and integrity checks.

In case someone has heard that MD5 is broken, fear not: For plain checking of file transfer or stored data integrity, MD5 is sufficient. It was cryptographically “broken” - which is relevant for security, but not for data integrity checking.

Important: Hashcodes are not sufficient for proving authenticity! In case you need to deal with important originals/documents and you need to make sure they’re originals and not forged, etc. please check out this:

Digital signatures are good, but could be signed with a date in the past (backdating).

If this is a concern, you may consider blockchain mechanisms: Blockchain cipher became popular for digital currency (like Bitcoin), but it can also be used to proof data authenticity and avoid backdating.

AFAIK there are currently no systems that implement this productively yet, but some have already researched into prototyping blockchain use for storage.

Hashcode Examples

CRC =
4294967295
MD5 =
d41d8cd98f00b204e9800998ecf8427e
SHA256 =
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
xxHash =
e4c191d091bd8853

Algorithms in order of complexity/size: > CRC —— MD5 —— SHA256 —— SHA512

Speed matters when it comes to calculating hashes for several hundreds of TB or PB of data.

Execution speed depends on the actual implementation of the algorithm. Even if a simpler algorithm may be faster in theory, it may not make a difference if the implementation isn’t speed-optimized. However, speeding up hashing becomes more and more interesting e.g. for transfer of digital cinema files, because validating these data amounts may currently be a bottleneck.

Different hashcodes algorithms/implementations may have significantly different runtimes. When dealing with large quantities of data, this may matter.

MD5 is (still) the most popular one around: Well known and widely supported by different applications/systems, etc.

Anyone transferring lots of uncompressed film? You may want to look at xxHash. It’s designed for speed.

When?

Generate fixity information as early as possible in a file’s lifecycle.

Different levels

Fixity information can be gathered/documented on different levels:

Filesystem
Whole files
Inside files: Content

Level 1: Filesystem Listing

Linux / MacOS: $ ls -la > dirlist.txt
Windows: C:\> dir /s /a > dirlist.txt

screenshot of files/folders in a file manager (Thunar)

For preservation, it’s good practice to document these file/folder properties too. Especially when receiving personal collections. They can be used to tell more about a file than you’d think:

date/time context: Which year? epoch? How old was the author then? Where (in his life) was the author when creating this?
Ownership / access rights : How private/public was this? For what reason?
Flags: Why was it hidden? Oh! It was meant to be run as program. Interesting…

By default, date/time format may not be sufficient/suitable for preservation and/or exchange. Therefore make sure that date/time are displayed in a format tht is exactly interpretable.

GNU/Linux systems offer the ability to format it in ISO8601, which is great:

$ ls -la --time-style=full-iso

Nice trick: If you want to transfer files from A-to-B, and want to make sure that its timestamps are preserved, you can pack it in a ZIP file: If the application allows it, disable compression (or use TAR).

This wraps a layer around the files, so that their timestamps are now stored as tech-MD inside the archive file.

Level 2: Per File

(The Classic)

Linux / MacOS: md5sum *.* > MD5SUMS.md5

Level 3: Content

“Content hashing is still hardly known - yet incredibly powerful.”

One hash per data stream.
One hash per frame/group of samples

Level 3: Content - Streams

$ ffmpeg -i input_file -map 0 \
-f streamhash -hash md5 -hide_banner - -v quiet

Output:

0,v,MD5=3f874757d9c1a2bc8adacb070f1a2e60
1,a,MD5=484a92455b87cc48d6d9cad5dd93435c
2,a,MD5=fdb680635a4cc3dd8419c96387760031

Level 3: Content - Image / Samples

ffmpeg -loglevel quiet -i VIDEOFILE -an -f framemd5 VIDEOFILE.framemd5

Generates One hash per frame:

Some Tools

HashCheck

GUI to handle hashcodes (Windows only).
Website: code.kliu.org/hashcheck

LoC BagIt “Bags”

“Bags have built-in inventory checking, to help ensure that content transferred intact.”

Bagger

A GUI for handling BagIt bags.

Bagger

Hashcode use: When?

Ingest into preservation environment
Periodically in storage/backup
During transfers or access
Deduplification

Data Integrity - Introduction

Data Integrity?

What is “Fixity” information?

Hashcodes

Hashcodes

Hashcodes

Hashcodes

Different algorithms

Hashcode Examples

When?

Different levels

Level 1: Filesystem Listing

Level 2: Per File

(The Classic)

Level 3: Content

Level 3: Content - Streams

Level 3: Content - Image / Samples

Some Tools

HashCheck

LoC BagIt “Bags”

Bagger

Bagger

Hashcode use: When?

Comments?

Questions?