Recommendations for designing magic numbers of binary file formats

conaclos · 2025-03-17T10:21:48 1742206908

   SHOULD include a zero byte

I guess it is expected to be at the end of the magic number to act as a null-termibated string?

   MUST include a byte sequence that is invalid UTF-8

I guess it is to differentiate a text file from a specific format?

   MUST include at least one byte with the high bit set

Any reason?

robinhouston · 2025-03-17T11:00:58 1742209258

I think the idea of all these is to make the file not be recognised as text (which doesn't allow nulls), ASCII (which doesn't use the high bit), UTF-8 (which doesn't allow invalid UTF-8 sequences).

Basically so that no valid file in this binary format will be incorrectly misidentified as a text file.

wjholden · 2025-03-17T10:25:18 1742207118

The author explains their reasoning in the next post: https://hackers.town/@zwol/114155807716413069

rcxdude · 2025-03-17T11:02:16 1742209336

Well, the 3rd point follows from the second: all sequences without the high bit set are valid ASCII, and all valid ASCII sequences are valid UTF-8.

hgomersall · 2025-03-17T10:22:13 1742206933

As anyone able to break down why those requirements are desirable?

rickdeckard · 2025-03-17T11:25:27 1742210727

From the top of my head, most are to make it as clear as possible that the file is binary and NOT text:

> MUST be the very first N bytes in the file

For every system to be able to parse it without loading the entire file

> MUST be at least four bytes long, eight is better

To reduce risk of two different binary files on the same system having the same magic number

> MUST include at least one byte with the high bit set

To avoid wrongful identification as an ASCII file (ASCII doesn't use the high bit)

> MUST include a byte sequence that is invalid UTF-8

To avoid wrongful identification as UTF-8 text file

> SHOULD include a zero byte

To avoid wrongful identification as ANY text file

weinzierl · 2025-03-17T09:24:07 1742203447

Why is ELF a good example?

    7F 45 4C 46

- MUST be the very first N bytes in the file -> check

- MUST be at least four bytes long, eight is better -> check, but only four

- MUST include at least one byte with the high bit set -> nope

- MUST include a byte sequence that is invalid UTF-8 -> nope

- SHOULD include a zero byte -> nope

So, just 1.5 out of 5. Not good.

By the way, does anyone know the reason it starts with DEL (7F) specifically?

0xFF0123 · 2025-03-17T09:59:50 1742205590

It's (7F) ELF

indigoabstract · 2025-03-17T10:48:02 1742208482

Hmm, I would expect that to be 31F, if it stood for "ELF" in correct Hexspeak.

（评论） (comments)

（评论）
(comments)