Data Integrity?

What is "Fixity" information?

Hashcodes

raw.txt

"This is a raw text file."

MD5 = b3a243d2443037a783c8799fe2c4926a

Hashcodes

raw.txt

"This is a raw text file.⎕"

MD5 = 7096384353da7d8cb59b1395e63d1250

Hashcodes

raw.txt

"this is a raw text file."

MD5 = a94a15d1b72bbfee7997bf237cf0347e

Hashcodes

raw-text.txt

"this is a raw text file."

MD5 = a94a15d1b72bbfee7997bf237cf0347e

Different algorithms

CRC
MD5
SHA .. 1 .. 2 .. 256 .. SHA512
XXHASH
WTF?

There are different hashing algorithms. In a nutshell:

Shorter hash = faster
but higher collision chance
= less secure
Longer hash = slower
but lower collision chance
= more secure

For data integrity verification, short hashes are perfectly fine.

Since hashing algorithms are also used for security purposes (digital signatures), MD5 was said to be "broken". This is only true for security/signature purposes. It is still perfectly valid for checking data integrity.

"xxHash" is relatively new, and is the only "Non-cryptographic hash function" in the above list, designed for speed and not security. It is becoming more common for fixity checks in A/V production, but yet it's still rather the exception.

Hashcodes are fixed-length numbers that are generated in a way that if even a single bit in the source data changes, that number will be completely different. They are often written in hexadecimal, therefore including the characters 0-9 and A-F.

If you have a hashcode for a set of data, it can be used to verify the bit-exact integrity of that data, by calculating the same-algorithm hashcode again and comparing them. If they're identical, the data is intact. If two distinct sets of data have the same hash, it's called a "hash collision. Hash algorithms are designed particularily to keep the chance for collision as low as possible.

Anyways: hashcodes are a must for safe data transfer and integrity checks.

In case someone has heard that MD5 is broken, fear not: For plain checking of file transfer or stored data integrity, MD5 is sufficient. It was cryptographically "broken" - which is relevant for security, but not for data integrity checking.

Important: Hashcodes are not sufficient for proving authenticity! In case you need to deal with important originals/documents and you need to make sure they're originals and not forged, etc. please check out this:

Digital signatures are good, but could be signed with a date in the past (backdating).

If this is a concern, you may consider blockchain mechanisms: Blockchain cipher became popular for digital currency (like Bitcoin), but it can also be used to proof data authenticity and avoid backdating.

AFAIK there are currently no systems that implement this productively yet, but some have already researched into prototyping blockchain use for storage.

Hashcode Examples

CRC =
4294967295
MD5 =
d41d8cd98f00b204e9800998ecf8427e
SHA256 =
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
xxHash =
e4c191d091bd8853

Algorithms in order of complexity/size: > CRC ------ MD5 ------ SHA256 ------ SHA512

Speed matters when it comes to calculating hashes for several hundreds of TB or PB of data.

Execution speed depends on the actual implementation of the algorithm. Even if a simpler algorithm may be faster in theory, it may not make a difference if the implementation isn't speed-optimized. However, speeding up hashing becomes more and more interesting e.g. for transfer of digital cinema files, because validating these data amounts may currently be a bottleneck.

Different hashcodes algorithms/implementations may have significantly different runtimes. When dealing with large quantities of data, this may matter.

MD5 is the most popular one around: Well known and widely supported by different applications/systems, etc.

Anyone transferring lots of uncompressed film? You may want to look at xxHash. It's designed for speed.

When?

Generate fixity information as early as possible in a file's lifecycle.

Different levels

Filesystem
File (=data)
Content (=payload)

Level 1

Filesystem Listing

Linux / MacOS
$ ls -la > dirlist.txt

Windows:
C:\> dir /s /a > dirlist.txt

For preservation, it's good practice to document these file/folder properties too. Especially when receiving personal collections. They can be used to tell more about a file than you'd think:

date/time context: Which year? epoch? How old was the author then? Where (in his life) was the author when creating this?
Ownership / access rights : How private/public was this? For what reason?
Flags: Why was it hidden? Oh! It was meant to be run as program. Interesting...

By default, date/time format may not be sufficient/suitable for preservation and/or exchange. Therefore make sure that date/time are displayed in a format tht is exactly interpretable.

GNU/Linux systems offer the ability to format it in ISO8601, which is great:

$ ls -la --time-style=full-iso

Nice trick: If you want to transfer files from A-to-B, and want to make sure that its timestamps are preserved, you can pack it in a ZIP file: If the application allows it, disable compression (or use TAR).

This wraps a layer around the files, so that their timestamps are now stored as tech-MD inside the archive file.

Level 2

Per File

Linux / MacOS
$ md5sum *.* > MD5SUMS.md5

Level 3

Inside: Content Hash

Some Tools

HashCheck

GUI to handle hashcodes (Windows only).
Website: code.kliu.org/hashcheck

LoC BagIt "Bags"

"Bags have built-in inventory checking, to help ensure that content transferred intact."

Bagger

A GUI for handling BagIt bags.

Bagger

Hashcode use: When?

Ingest into preservation environment
Periodically in storage/backup
During transfers or access
Deduplification

Data Integrity

Data Integrity?

What is "Fixity" information?

Hashcodes

Hashcodes

Hashcodes

Hashcodes

Different algorithms

Hashcode Examples

When?

Different levels

Level 1

Filesystem Listing

Level 2

Per File

Level 3

Inside: Content Hash

Some Tools

HashCheck

LoC BagIt "Bags"

Bagger

Bagger

Hashcode use: When?

Comments?

Questions?