File formats:
Making choices / Best practices

Peter Bubestinger-Steindl

2022-04-26

Summary: Preservation Format

Can be used to generate all other versions.
Depicts the “original” source as accurately as possible.
No artifical restrictions for using it.
Now and under unknown future (=unknown) conditions.
Well documented, no secrets, FOSS implementation exists.
Bit error resilience would be nice.
Consider GOP=1 (=Intraframe only).
Audio format: Normalize to uncompressed PCM/WAV.
For video container formats, consider using MKV or MOV.
MXF only if really necessary because:
As simple as possible, as complicated as necessary.

summarizes as: “preserves well”

Best practices for ingest/digitization

Capture analog video without adding generation loss.
Uncompressed ^(v210) or lossless ^{(FFV1, J2K)}.
Or fallback option: high-quality lossy. At the highest quality (bitrate) you can store and manage well over time.
Capture digital tape as “natively” as possible. ^{(MiniDV, DAT, DigiBeta, etc.)}
Store already-digital files “as original” as possible.
Transcode only if codec does not satisfy “sustainability” checklist. Rewrap/rewrite container. Always. Even if identical.

Tempting…

Hey, it’s a standard!
Hey, everyone’s using it!
Hey, the “big ones” are using it!
Hey, it’s from a major company!
Hey, it can do everything!
Hey, it’s so easy to use!
Hey, it’s gratis!

Rather…

ask, ask, ask.
get documentation.
get sample files.
try handling/opening them outside their “usual” bubble.
with at least 1 open implementation.
Before you commit to a format.
Try transcoding it “losslessly” to uncompressed.

Ask users with similar/identical use cases than yours how they’re doing with that format.
- Get file format data layout specifications and other related documentation that you can get your hands on. We’ve already profited from the ones who did this for past “data formats”: analogue ones or even ancient writings on clay stone tablets, etc.
- Get sample files and try using them as-if in your desired workflow. Ask manufacturers or service-providers to provide you samples, or show you how your samples behave in their application/environment.
- Try sample files “outside their usual bubble”: This means that sometimes certain format behave well within expected environments (e.g. ProRes on Apple stuff, certain MXF-flavors in broadcast tools, etc) - but when deviating from thes tested grounds, things may look quite differently.
- If you have at least 1 open implementation (=binary plus source code under a copyleft license), you have everything you need to get out of that format under any future conditions. Without artificial restrictions.
- If you can transcode the format to an uncompressed form, without losing any information, you know that you have means of getting out of that format once you may decide to switch to another one (in the future).

Wrapping up with our checklist:

Sustainability:

Disclosure?
Open reference implementation/libs?
Adoption/popularity?
Complexity?
Independence vs external contingencies?
Artificial restrictions?
Self descriptive?

Quality and functionality:

Preserve “original”?
Image/sound quality?
Interoperability?
Editing?
Support for (additional/expected) properties?
Performance & data size?

⭐Which translates to:

Sustainability:

Documentation openly accessible?
Open reference implementation?
How likely is it to be supported in tools/devices for which userbase?
Which features are implemented/tested/stable?
Which choice/requirements do I have to handle it beyond “shelf life”?
Is it legal/possible to handle it in future/different situations?
Can it contain proper metadata?

Quality and functionality:

Preserve significant properties?
Sufficient image/sound quality and robustness to multi-generation copies?
Interoperability / ease of usage & access?
Direct use for editing?
How many different formats will I need (pile up)?
Handle performance / data size requirements?

Documentation openly accessible? Without documentation (or access to it), a file format is the equivalent of a secret code/language. You can imagine how well “secret code” satisfies preservation requirements?
Open reference implementation? Any code that implements a file format can be called an “implementation” of that format. A “reference implementation” is an application that is able to read/write that format in a proper way, used to check the validity of other implementations that read/write the same format. With proprietary (=closed source) formats it is not uncommon that there is no real “offical” reference implementation - but merely a black-box binary provided by the vendor, not necessarily freely accessible and possibly subject to change at their will. Without an open reference implementation (=including source code), interoperability issues are more likely to happen.
How likely is it to be supported in tools/devices & for which userbase?

Adoption of a format plays a major role in how easy (=cheaper) vs hard (=more expensive) it is to use a format. It may be necessary/useful to engage with vendors of developers and ask for supporting a certain format in tools you would like to use.

The userbase is important, as it has a great impact on where a format is supported, and for which use-cases it is real-world tested. If a format is common in a userbase that covers your use cases, it’s good. If a format was designed/intended for a userbase further away from your use cases, you may find that you get less support (or understanding) for your needs.

Which features are implemented/tested/stable?

Developing, testing and supporting a file format is a tricky task. Therefore it is common that the “most popular/important” features are the ones most likely to be tested and working reliably. Other functions that may also be defined (even in a standardized format), may not receive the same attention by its supporters, often leading to (interoperability-)issues with not-so-popular functions. This is especially important for us, since certain features required for preservation are often less important for the main intended use-case of a format.

A popular example is “lossless”: both, JPEG2000 as well as H.264 can produce lossy as well as lossless encodings. However in practice, the lossless feature is often neither tested (and sometimes not even supported) in many of their implementations.

Which choice/requirements do I have to handle it beyond “shelf life”? For digital formats, this can be reduced to: If you have an Open Source version that is able to encode/decode a format, you can most likely continue using it under future (=unknown), possibly changing conditions.
Is it legal/possible to handle it in future/different situations? Patents, proprietary licensing and copy protection mechanisms are merely artificial restrictions that must be considered when choosing a file format for preservation. Especially since access to the material should not be hindered by them.
Can it contain proper metadata? It is good practice to include at least rudimentary metadata that allows to “identify” a file if found in the wild. Typical fields are: title, unique-identifier, institution name, etc.

Depending on your collection and users, different metadata may be useful or required to create a proper AIP.

Quality and functionality:

Preserve significant properties? Can the format maintain certain, relevant properties of the source as close to the original as possible?
Sufficient image/sound quality and robustness to multi-generation copies? Can the format provide sufficient quality, and how much impact would repeated encoding (lossy) have on it?
Interoperability / ease of usage & access? Can the format easily be used with different tools, under different operating systems, environments, devices, etc?
Direct use for editing? Can the format be used directly for non-linear editing or extracting parts, or does it require pre-transcoding in order to do so?
How many different formats will I need (pile up)? Can the format handle and depict the majority of my collection, or would I need different formats for other source material? It is intended to have a small number of different file formats in an archive, as maintaining workflows/tools becomes more difficult (=expensive) with more formats. However, it is also not advisable to simply choose a “can do anything and everything” format, as mentioned above: More features = harder to implement, support, maintain, etc.
Handle performance / data size requirements? Can the required (daily) workflows be supported in due time, or are there bottlenecks, budget constraints, etc?

Comments?

Questions?

p.bubestinger@av-rd.com

File formats: Making choices / Best practices

Summary: Preservation Format

Best practices for ingest/digitization

Tempting…

Rather…

Wrapping up with our checklist:

⭐Which translates to:

Comments?

Questions?

Links

File formats:
Making choices / Best practices