Why bother? - Let’s just have:

Best quality
Preserve original properties
Lowest size
Fast and easy to open/use
Lasts forever
+cherries 🍒 & ice cream 🍦on top!
…

Digital Video Trinity

Format Naming

Triplet notation greatly helps reducing confusion:

H.264 / AAC in MP4
FFV1 / PCM in MKV (Matroska)
ProRes / PCM in MOV
DPX / WAV (PCM) in a folder
etc

What’s a Codec?

“A codec is a device or computer program which encodes or decodes a data stream or signal.”

Source: Wikipedia: Codec

What’s a Codec?

Think of a human language…

It’s coded information.
There may be dialects.
Different people may
“speak / understand” differently.

What’s a Container?

“A container format (informally, sometimes called a wrapper) […] is a file format that allows multiple data streams to be embedded into a single file, usually along with metadata for identifying and further detailing those streams.”

Source: Wikipedia: Container format (computing)

What’s a Container?

Think of a regular paper folder…

It’s a wrapper around content.
Contains Metadata.
Structures the content streams.

Let’s look inside! :)

Data Structure (Hex Editor)

(Documented?) Data Structure

VLC / MediaInfo

Website: videolan.org/vlc

Website: mediaarea.net/MediaInfo

Characteristics / Properties

	File 1	File 2	File 3
Container	MOV	MOV	MOV
Videocodec	UYVY	H.264	XviD
Resolution	720 x 576px	1920 x 1080	640 x 480
FPS	25	24	30000/1001
-
Audiocodec	PCM	AAC	MP3
Samplerate	48 kHz	48 kHz	44.1 kHz
Channels	Stereo	Surround 5.1	Mono

Format choice = A balance of …

Sustainability:

Disclosure?
Open reference implementation/libs?
Adoption/popularity?
Complexity?
Independence vs external contingencies?
Artificial restrictions?
Self descriptive?

Quality and functionality:

Preserve “original”?
Image/sound quality?
Interoperability?
Editing?
Support for (additional/expected) properties?
Performance & data size?

⭐Translates to…

Sustainability:

Documentation openly accessible?
Open reference implementation?
How likely is it to be supported in tools/devices for which userbase?
Which features are implemented/tested/stable?
Which choice/requirements do I have to handle it beyond “shelf life”?
Is it legal/possible to handle it in future/different situations?
Can it contain proper metadata?

Quality and functionality:

Preserve significant properties?
Sufficient image/sound quality and robustness to multi-generation copies?
Interoperability / ease of usage & access?
Direct use for editing?
How many different formats will I need (pile up)?
Handle performance / data size requirements?

Documentation openly accessible? Without documentation (or access to it), a file format is the equivalent of a secret code/language. You can imagine how well “secret code” satisfies preservation requirements?
Open reference implementation? Any code that implements a file format can be called an “implementation” of that format. A “reference implementation” is an application that is able to read/write that format in a proper way, used to check the validity of other implementations that read/write the same format. With proprietary (=closed source) formats it is not uncommon that there is no real “offical” reference implementation - but merely a black-box binary provided by the vendor, not necessarily freely accessible and possibly subject to change at their will. Without an open reference implementation (=including source code), interoperability issues are more likely to happen.
How likely is it to be supported in tools/devices & for which userbase?

Adoption of a format plays a major role in how easy (=cheaper) vs hard (=more expensive) it is to use a format. It may be necessary/useful to engage with vendors of developers and ask for supporting a certain format in tools you would like to use.

The userbase is important, as it has a great impact on where a format is supported, and for which use-cases it is real-world tested. If a format is common in a userbase that covers your use cases, it’s good. If a format was designed/intended for a userbase further away from your use cases, you may find that you get less support (or understanding) for your needs.

Which features are implemented/tested/stable?

Developing, testing and supporting a file format is a tricky task. Therefore it is common that the “most popular/important” features are the ones most likely to be tested and working reliably. Other functions that may also be defined (even in a standardized format), may not receive the same attention by its supporters, often leading to (interoperability-)issues with not-so-popular functions. This is especially important for us, since certain features required for preservation are often less important for the main intended use-case of a format.

A popular example is “lossless”: both, JPEG2000 as well as H.264 can produce lossy as well as lossless encodings. However in practice, the lossless feature is often neither tested (and sometimes not even supported) in many of their implementations.

Which choice/requirements do I have to handle it beyond “shelf life”? For digital formats, this can be reduced to: If you have an Open Source version that is able to encode/decode a format, you can most likely continue using it under future (=unknown), possibly changing conditions.
Is it legal/possible to handle it in future/different situations? Patents, proprietary licensing and copy protection mechanisms are merely artificial restrictions that must be considered when choosing a file format for preservation. Especially since access to the material should not be hindered by them.
Can it contain proper metadata? It is good practice to include at least rudimentary metadata that allows to “identify” a file if found in the wild. Typical fields are: title, unique-identifier, institution name, etc.

Depending on your collection and users, different metadata may be useful or required to create a proper AIP.

Quality and functionality:

Preserve significant properties? Can the format maintain certain, relevant properties of the source as close to the original as possible?
Sufficient image/sound quality and robustness to multi-generation copies? Can the format provide sufficient quality, and how much impact would repeated encoding (lossy) have on it?
Interoperability / ease of usage & access? Can the format easily be used with different tools, under different operating systems, environments, devices, etc?
Direct use for editing? Can the format be used directly for non-linear editing or extracting parts, or does it require pre-transcoding in order to do so?
How many different formats will I need (pile up)? Can the format handle and depict the majority of my collection, or would I need different formats for other source material? It is intended to have a small number of different file formats in an archive, as maintaining workflows/tools becomes more difficult (=expensive) with more formats. However, it is also not advisable to simply choose a “can do anything and everything” format, as mentioned above: More features = harder to implement, support, maintain, etc.
Handle performance / data size requirements? Can the required (daily) workflows be supported in due time, or are there bottlenecks, budget constraints, etc?

Your format(s)…?

Your use cases/focus?

Who will want/need to work with these files?
Under which conditions?
For how long?
Digitization vs Production vs Preservation vs Access?
Which properties are significant to you?

“Different strokes for different folks” 😉

Digitization: As-original, as-untouched as possible. Headroom for optional restoration/improvements.
Preservation: Stand the test of time.
Highest original quality.
Mezzanine: For daily work. High quality.
Optional, if preservation format can be used for this.
Access For quick and easy access.
Quality not necessarily best/high.

Significant properties

Knowing and deciding which properties to safeguard and which are allowed to change.

See:
LoC FADGI: DRAFT Significant Properties for Digital Video
Nestor (DE): Leitfaden DLTP AV Medien

Significant properties

Depend on media type (and use case).

Video	Audio
resolution framerate aspect ratio colorspace subsampling …	“resolution” (= samplerate, bit-depth) channels channel layout

Quality

Avoid generation loss. ^{(if possible)}
Avoid resizing. ^(=rescaling)
Don’t invent more bits. ^{(e.g. DV as v210)}
Preserve colorspace / bit-depth, etc.
More headroom for lower quality. ^{(e.g. 24bit/96kHz for Shellack)}
Select high enough bitrate. ^(lossy)
(Or proper “Constant Rate Factor (CRF)”)

Bitrate = Data per Time

Mbps / 8 = MB / second
MB/s * 60 = MB / minute
MB/min * 60 = MB / hour

btw: Constant (CBR) vs Variable (VBR) bitrate?

If you know the bitrate, you can calculate the size of your files, by multiplying the bitrate by the runtime duration (time).

A fixed bitrate however is only available for:

constant bitrate (lossy) compression.
uncompressed.

For lossy compression, bitrate is an encoding parameter, but for uncompressed we’ll show later in this session how to calculate the size.

For lossless or variable bitrate, the exact size cannot be pre-calculated - only estimated. The actual size can still vary, since the size of these kind of encodings greatly depends on the content being encoded. Rule of thumb: Less motion = smaller size and vice versa. Noise make files larger.

There’s also the choice with some formats between constant bitrate and variable bitrate. Constant bitrate is simpler, more robust, straight to calculate/estimate. Simpler=preserves better. Variable bitrate is another way to compress space, but makes decoding/interpreting the data more complicated and may lead to different behavior in different tools/setups.

Size

Bitrate = Size vs Quality
(bitrate as parameter exists only for lossy encoding)
Uncompressed > lossless > lossy

Performance

Often a tradeoff between:

Processing power (CPU/RAM)
(format/algorithm complexity)

I/O bandwidth (disk/network)
(data size)

Format Examples

	Video	Audio	in Container
Preservation:	V210 FFV1, J2K High-bitrate lossy …	PCM FLAC	MOV MKV MXF
Mezzanine:	ProRes H.264 DVCPRO50 …
Access:	H.264 VP9 DVD BluRay …	MP3 AAC Opus …	MP4 ^(M4V) MKV ^(WebM) MPG

Lossy, Lossless, Uncompressed?

How it affects quality and preservation.

Lossy

“Lossy compression” literally means: some parts of the content are “lost” (=thrown away) in order to gain more compression (=smaller files). This process is irreversible and accumulates (=generation loss).

Of course the above image is exaggerated, but it shows pretty well what lossy compression is and what artefacts a typical MPEG-like compression algorithm produces.

(btw: This is a snapshot image of the highest-quality version of the video on the original website (around 2009)).

Due to the vast data sizes of AV, lossy encoding is the current de-facto default in most environments that handle digital AV (except film). High quality lossy codecs may sometimes be called “visually lossless” (e.g. ProRes). Which only means that they’re pretty good at hiding their artefacts from your eyes/ears. High-bitrate lossy formats are standard in broadcast and production.

For audio this is different, merely because the data size is so “trivial” for modern computer systems that the studio standard is uncompressed (PCM) in WAV. Suggesting lossy encoding to audio engineers as production source format would probably get you kicked out of their studio ;)

Generation Loss

Here’s an example where one can see the difference between each encoded version in popular broadcast codecs.

Since it’s almost certain that there’s not “evergreen format”, any format will sooner or later have to be migrated to another one. The longer an item is preserved, the more often it will encour such a format change (transcoding). Therefore, generation loss should be considered.

Also: Usage of archive material: What happens if you hand out your material to an editor, say on DVD or a mezzanine format? The editor then edits and exports to another format, the broadcaster or cinema then to another? …and in the end this work is then also converted to an access format.

And theeeen, years later that access copy is used as a source for a documentary or online edit.

How many generation losses accumulate here? How could they be reduced?

Generally: please try to avoid unnecessary generation losses. Why not? :)

Lossless

“It’s like ZIP for film!”

No generation loss
Way larger than lossy
Smaller than uncompressed

Uncompressed

No generation loss
Dead simple (=preserves well)
The largest possible version
Uncompressed != Uncompressed?
There’s more than just 1 “uncompressed”
(Ex: RGB, BGR, UYVY, YUY2, V210, etc)

Uncompressed - Think of it as:

4px RGB Image: (8bpc = 24bit/pixel)

RGB RGB RGB RGB     (4*3 = 12 byte)

4px YUV Image: (8bpc, 4:2:2, 16bit/pixel)

UYVY UYVY           (2*4 = 8 byte)

2 samples audio: (2ch, 16bits)

LL RR LL RR         (4*2 = 8 byte)

Paper analogy

Risks to format longevity

Data errors
Obsolescence
Vendor lock-in
Format complexity
Interoperability issues

Countermeasures?

Data errors: Files are corrupt. These errors may include filenames or filesystem tech-MD.
- Obsolescence: Not supported anymore by accessible tools. This is neither God-given, nor irreversible, unless it’s a black-box format. Format obsolescense/support is a human-made decision. Documentation/schematics are an important game changer.
- Vendor lock-in: For long-term preservation, vendor- and technology-neutrality is a must: They will come and go, and with lock-in situations Eternal Migration is hindered or even impossible. Format normalization helps here.
- Interoperability issues: If a format is read and written differently by different applications, it might “morph”. This morphed version might work fine with the tools used in a certain environment, but might be completely broken if read/written by another tool that misunderstands the “dialect”.

Data errors: Error resilience?

Bitstream checksums:
Ability to know if the content is intact.
Error concealment: Optional choice of decoder to “mask” decoding issues. (decoder specific)
Make backup copies! 😇

Obsolescence / Vendor Lock-in

Open vs Closed

Theory vs Practice

“Implementation overrules paper specs. Always.”

If the implementation is open: you can use, study, share and improve if necessary.

If it’s closed/proprietary then there’s nothing you can do about it: Black box. Vendor lock-in. With digital, this means that it’s quite likely that this as-is binary may not be able to keep running/functioning when its environment changes.

You’ve probably seen how fast even good and stable programs “age” until they stop working, once their native Windows version is “too long ago…”?

Even with official standards, their implementation is what creates the actual encoding/decoding. Regardless of what’s written in any paper manual (=theory): The code implementation (=practice) is what counts.

Therefore, it makes sense to demand an openly accessible reference implementation, with its source code under a license that allows to make use of it as you see fit.

The Eternal Replayer

+ =

Format Complexity

Format Complexity: Less is More

Good rule = “Minimalistic Data Format”:

As simple as possible
As complicated as necessary

Simpler = more stable, easier to use, keep alive, reconstruct or fix.

Yagni kiss Moscow?

YAGNI / KISS / MoSCoW

It may look cryptic, but is actually quite simple and useful:

It is an example for finding the “minimalistic data format” that suits your needs.

YAGNI = “You Ain’t Gonna Need It”:
It’s supposed to prevent you from selecting a format that is possibly too bloated or unncessarily complex, or less-well supported.
KISS = “Keep It Simple, Stupid”: “The KISS principle states that most systems work best if they are kept simple rather than made complicated; therefore, simplicity should be a key goal in design, and unnecessary complexity should be avoided.”
MoSCoW = “Must, Should, Could, Won’t”: “The Moscow method is a prioritization technique used in management, business analysis, project management, and software development to reach a common understanding with stakeholders on the importance they place on the delivery of each requirement”

The example shows 4 different container formats for AV. Each one has different number and kinds of supported features, represented by the size of its circle.

Larger circle = more features.
Smaller circle = less features.
Overlapping circles = common features.

Now, according to the MoSCoW method, write down which features you:

must have
should have
could have
won’t have (this time)

And check which format provides them, then draw a dot in the corresponding circle area.

For example: Only MXF may be able to provide support for broadcast-specific metadata/functions, therefore that feature will only have a dot in the MXF circle. Whereas, all (except) AVI can store aspect ratio - so that dot would go into overlap of all - except AVI. The feature of “extremely simple, well-documented and stable/unmodified for ages” would likely to be a dot in either AVI or MKV.

Use this to find out which format fits your needs, while being “as simple as possible and only as complicated as necessary”.

Summary: Preservation Format

Can be used to generate all other versions.
Depicts the “original” source as accurately as possible.
No artifical restrictions for using it.
Now and under unknown future (=unknown) conditions.
Well documented, no secrets, FOSS implementation exists.
Bit error resilience would be nice.
Consider GOP=1 (=Intraframe only).
Audio format: Normalize to uncompressed PCM/WAV.
For video container formats, consider using MKV or MOV.
MXF only if really necessary because:
As simple as possible, as complicated as necessary.

summarizes as: “preserves well”

Best practices for ingest/digitization

Capture analog video without adding generation loss.
Uncompressed ^(v210) or lossless ^{(FFV1, J2K)}.
Or fallback option: high-quality lossy. At the highest quality (bitrate) you can store and manage well over time.
Capture digital tape as “natively” as possible. ^{(MiniDV, DAT, DigiBeta, etc.)}
Store already-digital files “as original” as possible.
Transcode only if codec does not satisfy “sustainability” checklist. Rewrap/rewrite container. Always. Even if identical.

Tempting…

Hey, it’s a standard!
Hey, everyone’s using it!
Hey, the “big ones” are using it!
Hey, it’s from a major company!
Hey, it can do everything!
Hey, it’s so easy to use!
Hey, it’s gratis!

Rather…

ask, ask, ask.
get documentation.
get sample files.
try handling/opening them outside their “usual” bubble.
with at least 1 open implementation.

Ask users with similar/identical use cases than yours how they’re doing with that format.
- Get file format data layout specifications and other related documentation that you can get your hands on. We’ve already profited from the ones who did this for past “data formats”: analogue ones or even ancient writings on clay stone tablets, etc.
- Get sample files and try using them as-if in your desired workflow. Ask manufacturers or service-providers to provide you samples, or show you how your samples behave in their application/environment.
- Try sample files “outside their usual bubble”: This means that sometimes certain format behave well within expected environments (e.g. ProRes on Apple stuff, certain MXF-flavors in broadcast tools, etc) - but when deviating from thes tested grounds, things may look quite differently.
- If you have at least 1 open implementation (=binary plus source code under a copyleft license), you have everything you need to get out of that format under any future conditions. Without artificial restrictions.

Comments?

Questions?

p.bubestinger@av-rd.com

Understanding and Selecting a File Format

Why bother? - Let’s just have:

Digital Video Trinity

Format Naming

What’s a Codec?

What’s a Codec?

What’s a Container?

What’s a Container?

Let’s look inside! :)

Data Structure (Hex Editor)

(Documented?) Data Structure

VLC / MediaInfo

Characteristics / Properties

Format choice = A balance of …

⭐Translates to…

Your format(s)…?

Your use cases/focus?

“Different strokes for different folks” 😉

Significant properties

Significant properties

Quality

Bitrate = Data per Time

Size

Performance

Format Examples

Lossy, Lossless, Uncompressed?

Lossy

Generation Loss

Lossless

Uncompressed

Uncompressed - Think of it as:

Paper analogy

Risks to format longevity

Data errors: Error resilience?

Obsolescence / Vendor Lock-in

Open vs Closed

Theory vs Practice

The Eternal Replayer

Format Complexity

Format Complexity: Less is More

Yagni kiss Moscow?

Summary: Preservation Format

Best practices for ingest/digitization

Tempting…

Rather…

Comments?

Questions?

Links