Why bother? - Let’s just have:

Best quality
Smallest filesize
Preserve original properties
Fast and easy to open/use/edit
Lasts forever
+cherries 🍒 & ice cream 🍦on top!
…

Tempting…

Hey, it’s a standard!
Hey, everyone’s using it!
Hey, the “big ones” are using it!
Hey, it’s from a major company!
Hey, it can do everything!
Hey, it’s so easy to use!
Hey, it’s gratis!

Digital AV formats…

Which do you know?
Which do you use?
Which would you like to know more about?

What are your wishes, needs, expectations of a format? 🤔️

Digital Video Trinity

What’s a Container / Wrapper?

"A container format […] is a file format that allows multiple data streams to be embedded into a single file.

Usually along with metadata for identifying and further detailing those streams."

Source: Wikipedia: Container format (computing)

What’s a Container?

Think of a regular paper folder…

It’s a wrapper around content.
Contains Metadata.
Structures the content streams.

What’s a Codec?

“A codec is a device or computer program which encodes or decodes a data stream or signal.”

Source: Wikipedia: Codec

What’s a Codec?

Think of a human language…

It’s coded information.
There may be dialects.
Different people may
“speak / understand” differently.

Format Naming

Triplet notation greatly helps reducing confusion:

H.264 / AAC in MP4
FFV1 / PCM in MKV (Matroska)
ProRes / PCM in MOV
DPX / WAV (PCM) in a folder
etc

See The MPEG-4 confusion 😜️

“Rebranded” Format Names

Some (professional) video formats are actually a profile-set for existing (standards) formats.

This is good! 😇️

Popular examples:

Name	Formats
Digital Cinema Package (DCP) =	Motion-JPEG2000 (lossy), PCM in MXF + XML
AVC-Intra =	H.264/MPEG-4 AVC
MPEG IMX =	H.262/MPEG-2
HDCAM SR =	MPEG-4/Part2 Simple Studio Profile (SStP)
Broadcast Wave Format (BWF) =	linear PCM in WAV
WebM =	Matroska (MKV) container

Let’s look inside! :)

VLC / MediaInfo

Website: videolan.org/vlc

Website: mediaarea.net/MediaInfo

Characteristics / Properties

	File 1	File 2	File 3
Container	MOV	MOV	MOV
Videocodec	UYVY	H.264	XviD
Resolution	720 x 576px	1920 x 1080	640 x 480
FPS	25	24	30000/1001
-
Audiocodec	PCM	AAC	MP3
Samplerate	48 kHz	48 kHz	44.1 kHz
Channels	Stereo	Surround 5.1	Mono

Bitrate = Data per Time

= How many bytes your (lossy) encoder may spend on the quality of the material.

Higher bitrate	Better Codec
Better quality, but larger files	Better quality at same size or: Same quality at smaller size

Significant properties

Knowing and deciding which properties to safeguard and which are allowed to change.

See:
LoC FADGI: DRAFT Significant Properties for Digital Video
Nestor (DE): Leitfaden DLTP AV Medien

Significant properties

Depend on media type (and use case).

Video	Audio	Metadata
resolution framerate aspect ratio colorspace subsampling …	“resolution” (= samplerate, bit-depth) channels channel layout …	language title author rights information …

“Different strokes for different folks” 😉

Digitization: As-original, as-untouched as possible. Records in realtime?
(Plus: has headroom for optional restoration/improvements.)
Preservation: Stand the tests of time.
(Highest ‘original’ quality)
Production: For daily work. High quality.
(Optional, if preservation format can be used for this)
Access For quick and easy access.
(Quality not necessarily best/high, but very convenient to play)

For audio: we’re lucky.

PCM/WAV is used from digitization to preservation - and if bandwidth ain’t narrow, even for access.

Why? Because it became “small enough”.

“Different strokes for different folks” 😉

Digitization: (V210, FFV1, MPEG-4 / PCM) uncompressed/lossless or very-high quality lossy.
Preservation: (V210, FFV1, MPEG-2, MPEG-4 / PCM) uncompressed/lossless or (very)high-quality lossy, open & documented, error-robust.
Production: (MPEG-2, MPEG-4, ProRes / PCM) high-quality/high-bitrate (lossy).
Access: (H.264 / AAC in MP4) Most often lossy-compressed currently-popular format combination.

Format choice = A balance of …

Size vs Quality
Features
Performance
Sustainability
plus: time, budget, staff
and of course: convenience

Good starting point for assessing practical usefulness.

Risks to format longevity

Data errors
Obsolescence
Vendor lock-in
Interoperability/complexity issues
Quality degradation

Countermeasures?

Data errors: Files are corrupt. These errors may include filenames or filesystem tech-MD.
- Obsolescence: Not supported anymore by accessible tools. This is neither God-given, nor irreversible, unless it’s a black-box format. Format obsolescense/support is a human-made decision. Documentation/schematics are an important game changer.
- Vendor lock-in: For long-term preservation, vendor- and technology-neutrality is a must: They will come and go, and with lock-in situations Eternal Migration is hindered or even impossible. Format normalization helps here.
- Interoperability issues: If a format is read and written differently by different applications, it might “morph”. This morphed version might work fine with the tools used in a certain environment, but might be completely broken if read/written by another tool that misunderstands the “dialect”.
  
  A format offering more features, can be considered more “complex”. Whereas a format offering less features is considered “simpler”. Simpler (and more popular) formats are usually less susceptible to interoperability issues, than complex (and/or less popular ones)
- Quality degradation: Avoid unncessary copy/encoding generation losses, or property conversion. This may include, avoiding additional lossy encoding steps, resizing the image, resampling audio, etc.

Data errors: Error resilience?

Bitstream checksums:
Ability to know if the content is intact.
Error concealment: Optional choice of decoder to “mask” decoding issues. (decoder specific)
Make backup copies! 😇

Open vs Closed

Theory vs Practice

“Implementation overrules paper specs. Always.”

If the implementation is open: you can use, study, share and improve if necessary.

If it’s closed/proprietary then there’s nothing you can do about it: Black box. Vendor lock-in. With digital, this means that it’s quite likely that this as-is binary may not be able to keep running/functioning when its environment changes.

You’ve probably seen how fast even good and stable programs “age” until they stop working, once their native Windows version is “too long ago…”?

Even with official standards, their implementation is what creates the actual encoding/decoding. Regardless of what’s written in any paper manual (=theory): The code implementation (=practice) is what counts.

Therefore, it makes sense to demand an openly accessible reference implementation, with its source code under a license that allows to make use of it as you see fit.

The Eternal Replayer

+ =

Format Complexity

Format Complexity: Less is More

Good rule = “Minimalistic Data Format”:

As simple as possible
As complicated as necessary

Simpler = more stable, easier to use, keep alive, reconstruct or fix.

Your use cases/priorities?

Who will want/need to work with these files?
Under which conditions?
For how long?
Digitization vs Production vs Preservation vs Access?
Which properties are significant to you?

Yagni Kiss Moscow?

YAGNI / KISS / MoSCoW

It may look cryptic, but is actually quite simple and useful:

It is an example for finding the “minimalistic data format” that suits your needs.

YAGNI = “You Ain’t Gonna Need It”:
It’s supposed to prevent you from selecting a format that is possibly too bloated or unncessarily complex, or less-well supported.
KISS = “Keep It Simple, Stupid”: “The KISS principle states that most systems work best if they are kept simple rather than made complicated; therefore, simplicity should be a key goal in design, and unnecessary complexity should be avoided.”
MoSCoW = “Must, Should, Could, Won’t”: “The Moscow method is a prioritization technique used in management, business analysis, project management, and software development to reach a common understanding with stakeholders on the importance they place on the delivery of each requirement”

The example shows 4 different container formats for AV. Each one has different number and kinds of supported features, represented by the size of its circle.

Larger circle = more features.
Smaller circle = less features.
Overlapping circles = common features.

Exercise: Your Format Policy

Must	Should	Could	Won’t
______	______	______	______
______	______	______	______
______	______	______	______
______	______	______	______

Choose a use-case and try to phrase your “wishes”.

Examples of published Policies

Guidelines for the Digital Film Collection
(Austrian Filmmuseum, 2018)
Digital Preservation & Access Strategy
(Irish Film Archive, 2016)
Digital Preservation: Policy, Standards and Procedures
(Netherlands Institute for Sound and Vision, 2016)

The above links point to preservation policies published by other AV archival institutions. They also contain information about file format choices - in the context of AV preservation.

As you can see, the length and level of detail of such policies can vary greatly. Often it is great to simply have a rather short written form that at least provides insights in “how” and “why” someone has chosen one (format) option over another.

It is also good practice to write such policies in a way that they can be interpreted and applied properly even if the actual technical options/conditions change. For example, it’s okay to say “we chose format XY” - but without writing down “why”, this choice may seem incomprehensible and outdated once “format XY” is superseded by another format, or something else in the tech-envirionment or the conditions has changed.

Another example: It was common to use “MP3” for some (longer) recordings in the audio archiving domain, instead of WAV/PCM. Now this may seem odd, but at that time 10 MB/Minute WAV/PCM (CD-Resolution) was heavy lifting in terms of digital computer storage.

But back then, a “large” HDD RAID was 80 GB. Consisting of several 20 GB harddisks. Now, this would fit on a USB-Stick.

“Nothing ever doesn’t change, but nothing changes much.”

All the time.

Checklist for a Preservation Format

Sustainability:

Disclosure?
Open reference implementation/libs?
Adoption/popularity?
Complexity?
Independence vs external contingencies?
Artificial restrictions?
Self descriptive?

Quality and functionality:

Preserve “original”?
Image/sound quality?
Interoperability?
Editing?
Support for (additional/expected) properties?
Performance & data size?

Source: meemoo.be

Preservation Checklist: translated! ⭐

Sustainability:

Documentation openly accessible?
Open reference implementation?
How likely is it to be supported in tools/devices for which userbase?
Which features are implemented/tested/stable?
Which choice/requirements do I have to handle it beyond “shelf life”?
Is it legal/possible to handle it in future/different situations?
Can it contain proper metadata?

Quality and functionality:

Preserve significant properties?
Sufficient image/sound quality and robustness to multi-generation copies?
Interoperability / ease of usage & access?
Direct use for editing?
How many different formats will I need (pile up)?
Handle performance / data size requirements?

Original source: meemoo.be

Documentation openly accessible? Without documentation (or access to it), a file format is the equivalent of a secret code/language. You can imagine how well “secret code” satisfies preservation requirements?
Open reference implementation? Any code that implements a file format can be called an “implementation” of that format. A “reference implementation” is an application that is able to read/write that format in a proper way, used to check the validity of other implementations that read/write the same format. With proprietary (=closed source) formats it is not uncommon that there is no real “offical” reference implementation - but merely a black-box binary provided by the vendor, not necessarily freely accessible and possibly subject to change at their will. Without an open reference implementation (=including source code), interoperability issues are more likely to happen.
How likely is it to be supported in tools/devices & for which userbase?

Adoption of a format plays a major role in how easy (=cheaper) vs hard (=more expensive) it is to use a format. It may be necessary/useful to engage with vendors of developers and ask for supporting a certain format in tools you would like to use.

The userbase is important, as it has a great impact on where a format is supported, and for which use-cases it is real-world tested. If a format is common in a userbase that covers your use cases, it’s good. If a format was designed/intended for a userbase further away from your use cases, you may find that you get less support (or understanding) for your needs.

Which features are implemented/tested/stable?

Developing, testing and supporting a file format is a tricky task. Therefore it is common that the “most popular/important” features are the ones most likely to be tested and working reliably. Other functions that may also be defined (even in a standardized format), may not receive the same attention by its supporters, often leading to (interoperability-)issues with not-so-popular functions. This is especially important for us, since certain features required for preservation are often less important for the main intended use-case of a format.

A popular example is “lossless”: both, JPEG2000 as well as H.264 can produce lossy as well as lossless encodings. However in practice, the lossless feature is often neither tested (and sometimes not even supported) in many of their implementations.

Which choice/requirements do I have to handle it beyond “shelf life”? For digital formats, this can be reduced to: If you have an Open Source version that is able to encode/decode a format, you can most likely continue using it under future (=unknown), possibly changing conditions.
Is it legal/possible to handle it in future/different situations? Patents, proprietary licensing and copy protection mechanisms are merely artificial restrictions that must be considered when choosing a file format for preservation. Especially since access to the material should not be hindered by them.
Can it contain proper metadata? It is good practice to include at least rudimentary metadata that allows to “identify” a file if found in the wild. Typical fields are: title, unique-identifier, institution name, etc.

Depending on your collection and users, different metadata may be useful or required to create a proper AIP.

Quality and functionality:

Preserve significant properties? Can the format maintain certain, relevant properties of the source as close to the original as possible?
Sufficient image/sound quality and robustness to multi-generation copies? Can the format provide sufficient quality, and how much impact would repeated encoding (lossy) have on it?
Interoperability / ease of usage & access? Can the format easily be used with different tools, under different operating systems, environments, devices, etc?
Direct use for editing? Can the format be used directly for non-linear editing or extracting parts, or does it require pre-transcoding in order to do so?
How many different formats will I need (pile up)? Can the format handle and depict the majority of my collection, or would I need different formats for other source material? It is intended to have a small number of different file formats in an archive, as maintaining workflows/tools becomes more difficult (=expensive) with more formats. However, it is also not advisable to simply choose a “can do anything and everything” format, as mentioned above: More features = harder to implement, support, maintain, etc.
Handle performance / data size requirements? Can the required (daily) workflows be supported in due time, or are there bottlenecks, budget constraints, etc?

Links

- Fin -

Questions?

Comments?

Peter Bubestinger-Steindl

Peter @ ArkThis.com

Understanding and Selecting a File Format for AV

Why bother? - Let’s just have:

Tempting…

Digital AV formats…

Digital Video Trinity

What’s a Container / Wrapper?

What’s a Container?

What’s a Codec?

What’s a Codec?

Format Naming

“Rebranded” Format Names

Popular examples:

Let’s look inside! :)

VLC / MediaInfo

Characteristics / Properties

Bitrate = Data per Time

Significant properties

Significant properties

“Different strokes for different folks” 😉

For audio: we’re lucky.

“Different strokes for different folks” 😉

Format choice = A balance of …

Risks to format longevity

Data errors: Error resilience?

Open vs Closed

Theory vs Practice

The Eternal Replayer

Format Complexity

Format Complexity: Less is More

Your use cases/priorities?

Yagni Kiss Moscow?

Exercise: Your Format Policy

Examples of published Policies

Checklist for a Preservation Format

Preservation Checklist: translated! ⭐

Links

- Fin -

Questions?

Comments?