FFV1 im AV Archivbereich
(nestor Webinar)

Peter Bubestinger-Steindl
(p.bubestinger@av-rd.com)

2021-05-20

FFV1: Was bringt’s? Was kann’s? Was kostet’s? Und wie geht’s weiter?

FFV1: Was bringt’s?

It saves disk space (*)

  • No generation loss, bit-proof encoding.
  • Master for generating different format (access) copies.
  • It “just works™”

    At highest quality.

    +Significant Properties = safe. long-term. open.

How it all began…?

Format consensus and interoperability
Format consensus and interoperability

2009, AT, Vienna

Hermann Lewetz at the Austrian Mediathek wanted to apply the same principles to video preservation as we all agree on the standards for audio:
Lossless or uncompressed, if possible. Lossy is to be avoided.

See: IASA-TC 03

Uncompressed, J2K, MXF: Okay. But tricky.

Options?

Lossless codec comparison graph (MSU, 2007)
Lossless codec comparison graph (MSU, 2007)

The format consensus was… in 2014!

And that’s 5 years in the future (2014)
And that’s 5 years in the future (2014)

FFmpeg & FFV1: Getting involved.

Plus: ©onflicts of Inte®est℠™

How dare you…

“…keeping us from wanting to do affordable, easy, open and interoperable lossless? For long-term reasons.”

FFV1: A short history about almost everything

  • 2003: FFV1 created in Open Source project “FFmpeg”
  • 2006: Bitstream frozen (version 1)
  • 2009: Picked up for preservation
  • 2010: Funding improvements (Mediathek)
  • 2012: Added 14bit RGB, Multithreading, SliceCRC
  • 2013: Official release of “FFV1.3”
  • 2014: PREFORMA Project
  • 2016: IETF Standardization in progress
  • 202+: Number of FFV1 users growing. Fast.

FFV1: 2021?

Practically the current defacto (and soon IETF) standard lossless preservation format.

And it’s Free and Open Source, so accessible and usable without artifical restrictions.

FFV1: Was kann’s?

Page 1

  • Absolutely completely reversible lossless compression.
  • Always lossless. Nothing to misconfigure.
  • Similar compression as J2K (but faster).
  • Not the fastest lossless codec, but best speed/size/support ratio.

Fast enough for SD editing in realtime

Smooth FFV1 playback in Virtualbox.
Smooth FFV1 playback in Virtualbox.

Quality (original = lossless)

Quality (x264 8Mbps)

Quality (x264 2Mbps)

Quality (x264 850k)

Quality (x264 200k)

FFV1: Was kann’s?

Page 2

  • Preserves almost all common “pix_fmts” as-is.
  • Open since day 1.
  • YUV + RGB up to 16bpc (*)
  • Widely supported by now (thx to FFmpeg libs).
  • Easy to migrate to other/future formats.
  • Error detection built-in (Slices)

Slices

Example what 9 slices would be
Example what 9 slices would be

Error detection/concealment

What is MKV?

“The Matroska Multimedia Container is a free, open-standard container format, a file format that can hold an unlimited number of video, audio, picture, or subtitle tracks in one file.”

Source: Wikipedia: Matroska

What is MKV?

  • It’s a container (like MOV, AVI, MXF)
  • Initial release: December 2002
  • IETF standardization in progress.
  • Also Open since day 1.
  • And btw: the base for Google’s “WebM” format

MKV is awesome.

  • Can contain/keep almost any source material (*)
  • Which is good for preservation
  • Like: any codecs, VBR, VFR, color infos, etc etc
  • Well documented and well supported (*)
  • It’s tech-design allows adaptability to future things.

FFV1 & MKV = ❤️

  • FFV1 goes well into: MKV, but also AVI/MOV.
  • All 3 combinations are preservation-approved.
  • With MKV being the most vendor-neutral and open/accessible option.
  • Not only stable, but future-proof.
  • Theoretically no format obsolescence issues.

Users: Moving to FFV1/MKV

 

British Film Institute: 2019 vs 2020

FFV1: Performance

FFV1: Performance

Dude, it’s huge!

PAL SD (720x576px)

Uncompressed: ca. 1.37 GB/m
(< 4 min. on a DVD)

  • FFV1 VHS: ca. 350 MB/min
  • FFV1 DigiBeta : ca. 700 MB/min

FFV1: Size estimations

Film

  • Varying between 1/3 and 2/3.
  • Average = 1/2 of uncompressed.
  • Better with less bits (=less noise).

Professional FFV1 Fun Facts!

  • Noise compresses bad (no pattern).
  • Synthetic (CGI) compresses best: ~1/9th (!)
  • Lossy is always smaller.

Thanks Alex for the Data!

Frisch aus dem Archiv der sozialen Demokratie Quelle: Archiv der sozialen Demokratie (2021-05)

Hardware & Speed.

How to use it?

Virtualdub2 & FFV1
Virtualdub2 & FFV1

Most people however, prefer this:

$ ffmpeg -c:v ffv1 -c:a pcm_s16le -map 0 preserved_yo.mkv

(FFmpeg is king. But that’s another story.)

When to use it?

  • Analogue captures.
  • Normalizing (=converting) born-digital files with proprietary (non-whitelist) codecs.
  • Scientific video research data.
  • Personal originals.
  • More backups than uncompressed.
  • Network transfer.

When not to use it?

  • Born digital already (=lossy encoded) in “preservable” format.
    (Refer to the format whitelist approach)
  • If filesize simply is too much for your setup/budget to handle.
  • …or simply overkill for a given use case.
  • Or you prefer lossy artefacts.

The Whitelist approach

A format normalization decision for preservation:

  1. Does an open, FOSS licensed implementation for decoding (to uncompressed) exist?
  2. Is the format a documented and/or an open standard?
  3. Is the license/patent situation of the format (and codec implementations) resolved for unlimited access/usage?
  4. Can you put that in your AIP without future restrictions for access? (maybe has to run then on Unknown Future Alien Technology 3000™?)

Was kostet’s?

  • Computation time (CPU).
  • FOSS = Free to use, study, share & improve.
  • Support- and license costs: Optional.
    (Think of it like Roads and Bridges)

Rawcooked

“It’s like ZIP for film! Awesome.”®

  • 1-stop-shop for image sequences (🎞️) to FFV1/PCM in MKV. 📼 💾
  • And can completely bit-hashcode-proof restore the original file sequence and structure. 🥳
  • Has a Free + Paid license model. 🍒
  • Yes, it’s OpenSource. ✅

https://mediaarea.net/RAWcooked

Und wie geht’s weiter?

  • Finalize standardization.
  • Better support for easier color handling.
  • Continue improving tool/support landscape.

Und wie geht’s weiter?

Thanks to everyone who contributed!

An Original Drawing (Ireland, dated around 2014 AD)
An Original Drawing (Ireland, dated around 2014 AD)

Fersch.

Frasch’n?

(ffv1@av-rd.com)