How to display raw data?

Ideas…?

Data as Text? 😒️

Hex editing! 🤩️

What is “Hex”?

“Hex” is short for “Hexadecimal” (which literally means “16”), and is a
base-16 numeral system.

btw: To avoid confusion, hex numbers are usually prefixed with “0x”:
0xFF, 0x10, 0x12345, etc.

Comparison to Decimal

	Dec	Hex
	0	0
	1	1
	2	2
	…	…
	8	8
	9	9
Base =	10	A
	11	B
	12	C
	13	D
	14	E
	15	F
	16	10	= Base (16)
	…	…
	255	FF

Hex 💕️ Byte

Why is hexadecimal (base-16) better than decimal (base-10) for digital data?

1 pair = 1 byte

Hex 💕️ Byte

From “00” to “FF” = 0..255
= The value range of 1 byte.
Each hex digit represents 4 bits.
1 Byte = 8 Bit = 256 numbers [0..255]
4 Bit = 16 numbers [0..15]
[0..15] (dec) = [0..F] (hex)

Digital data is nothing else than a finite stream of bytes. When working with AV files, it’s common that a certain data value requires reading multiple bytes in a row.

For example: * 16-bit PCM audio sample = 2 bytes * 24 bit RGB pixel color = 3 bytes

Sometimes (but more if you’re doing embedded systems stuff, like microcontrollers and hardware) you may also need to know which bit at which position is set/unset in a byte.

As strange as it may sound, but breaking a byte in half, visually, makes sense if you need to work with the actual bits:

Each hex-digit represents a Nibble - that’s 4 bit.

4 bit = 16 different combinations:

  0. 0000 (0x0)
  1. 0001 (0x1)
  2. 0010 (0x2)
  3. 0011 (0x3)
  
  4. 0100 (0x4)
  5. 0101 (0x5)
  6. 0110 (0x6)
  7. 0111 (0x7)
  
  8. 1000 (0x8)
  9. 1001 (0x9)
  10. 1010 (0xA)
  11. 1012 (0xB)
  
  12. 1100 (0xC)
  13. 1101 (0xD)
  14. 1110 (0xE)
  15. 1111 (0xF)

Less combinations than the letters in the western alphabet, so you’ll remember at least some of those bit-patterns. Working with data on this level automatically involves practicing it, anyways.

For starters, I’d like to introduce you to a few you’ll instantly remember forever:

0x0 = 0000 = no bit set
0xF = 1111 = every bit set

Combined to a byte:

0x00 = 0000 0000
0xFF = 1111 1111

Voila. You’ve already mastered 4 important patterns! Up for just 2 more?

0x10 = 0001 0000
0x11 = 0001 0001

You get the idea.

btw, you also know if a hex-digit is the left or the right half (upper/lower) of the byte.

Character encoding

Text as Data?

Let’s Hack in Hex!

Hex editor: Introduction

Data offset (in byte)
Data view (1 hex-pair = 1 byte)
Text view

Hex editor: Introduction

Data offset: (in byte)
Shows “where in the file/stream” you are.
The last number equals the filesize in total.
Data view: (1 hex-pair = 1 byte)
This is it: These are your bytes.
Each shown as a number: [00..FF]
Text view:
This is each byte interpreted as text, applying a certain character set.
Non-printable (or non-alphanumeric) characters are often represented by a placeholder. Cursor position highlights the corresponding hex-value, too.

“Magic bytes”

.PNG
RIFF
PK..
JFIF
AIFF
.Eß£
%PDF-
8BPS
…

Source: List of File Signatures (Wikipedia)

Exercise

Identify the file types in the given set, using a Hexeditor and the
“Magic Byte” list on Wikipedia

Unix “file” command (1973!)

See Wikipedia: File (command)

Initially released in 1973 as part of the Unix operating system, and then re-impelemted 1986 in an OpenSource licensed version, the “file” command can be found on any unix-like machine (like Linux, MacOS, etc) out-of-the-box.

Here’s what it does in which order to figure out what kind of data a file is: Wikipedia

if the file cannot be read, or its Unix file type is undetermined, the file program will indicate that the file was processed but its type was undetermined.
file must be able to determine the types directory, FIFO, socket, block special file, and character special file.
zero-length files are identified as such.
an initial part of file is considered and file is to use position-sensitive tests. (<- scan for “magic bytes” in the header, or other filetype specific patterns)
the entire file is considered and file is to use context-sensitive tests.
the file is identified as a data file.

Exercise / Puzzle

The file “whatami”:

Identify what it is?
Find out what is wrong with it?

MIME Type

“Multipurpose Internet Mail Extensions (MIME) is an Internet standard that extends the format of email messages to support text in character sets other than ASCII, as well attachments of audio, video, images, and application programs.” – Wikipedia: Media Type

MIME Type Examples

application/zip
application/pdf
text/html
text/xml
text/csv
text/plain
image/png
image/jpeg
image/gif
audio/aac
audio/mpeg
video/DV
video/H264
video/mp4

Complete List (IANA), 2019-10-16

Remember our “no suffix” file set?

So where’s my audio/video/image?

Data Structure

Header? Payload?

“header refers to supplemental data placed at the beginning of a block of data being stored or transmitted. In data transmission, the data following the header is sometimes called the payload or body.” – Wikipedia: Header (computing)

Examples

Viewing and Interpreting Binary Data

How to display raw data?

Data as Text? 😒️

Hex editing! 🤩️

What is “Hex”?

Comparison to Decimal

Hex 💕️ Byte

1 pair = 1 byte

Hex 💕️ Byte

Character encoding

Text as Data?

Let’s Hack in Hex!

Hex editor: Introduction

Hex editor: Introduction

“Magic bytes”

Exercise

Unix “file” command (1973!)

Exercise / Puzzle

MIME Type

MIME Type Examples

Remember our “no suffix” file set?

So where’s my audio/video/image?

Data Structure

Header? Payload?

Examples

Comments?

Questions?