Thank you for emphasizing this. Many junior devs have been bitten by not being told early enough the difference between encryption (requires a secret to be reversed), hashing (cannot be reversed) and encoding (can always be trivially reversed).

Also good to know that while the output looks random, it follows the same entropy as the input. Ie: don't base64 encode your password to make it stronger.

If however your password is engineered to be easier to remember, for example by using dictionaries or some kind of scheme that has a lower entropy, then the base64 encoding step adds a single bit of strength to your password. Meaning anyone who is brute forcing your password using a smart password cracker, has to configure that cracker to also consider base64 encoding as a scheme, basically forcing it to perform a single extra operation for every try.

Anyway, useless information, you shouldn't be using password schemes like that. The horse shoe battery staple type password style should be quite sufficient I think.

I wonder if its better to make an encoder that uses words and the output looks like "horse shoe battery staple" except you don't release your dictionary list of potential words output by the encoder, but then you guarantee that you can always re-create a password if you lose it, assuming you don't lose the dictionary file.

It depends... Is it the sum or the average that counts?

[1] - https://github.com/qntm/base2048

[2] - https://security.stackexchange.com/questions/85663/is-it-a-g...

[3] - https://security.stackexchange.com/questions/4943/will-using...

A base64-encoded string will be ~30% longer than the original string.

Hashing has many purposes besides security, and for that reason there are many hash libraries. If you plan on using hashes for something related to security or cryptography, you need to use a hash designed for that purpose! Yes, CRC hashing is really fast, and that's great, but it's not great when you use it for user passwords.

https://en.wikipedia.org/wiki/Cryptographic_hash_function

https://en.wikipedia.org/wiki/Password-hashing_function

I still see newly released projects that choose md5. Like, sure, for the intended use case, probably nobody will construct a collision, but why even allow the possibility?

Generally speaking, hashing user-provided data as-is is only a guarantee of integrity, not of authenticity (see HMAC), nor secrecy.

If they want to encrypt something just tell them to use ROT13 twice.

    $ function iter {
    N="$1"
    CMD="$2"
    STATE=$(cat)
    for i in $(seq 1 $N); do
       STATE=$(echo -n $STATE | $CMD)
    done
    cat 

[1] https://twitter.com/p4bl0/status/298900842076045312

[2] https://news.ycombinator.com/item?id=5181256

[3] https://www.reddit.com/r/compsci/comments/18234a/the_base64_...

[4] https://web.archive.org/web/20130315082932/http://fmota.eu/b...

  $ echo -n "abcde" |base64

(however, the parent's use of "echo" would be fine as it's not using a variable and so won't be interpreting a dash as an extra option etc)

I just tested it out with:

  sh /bin/echo -n "test"
  /bin/echo: 3: Syntax error: "(" unexpected

printf is always the better choice.

https://datatracker.ietf.org/doc/html/rfc4648#section-5

If you want to eliminate ambiguity for human readers, you can drop to Base58 but in almost all cases, if you are BaseXX-encoding something, it’s long enough that copy-pasting is the norm, so it doesn’t usually matter.

https://en.wikipedia.org/wiki/Base62

Base32 is sufficient in most cases and can avoid some incidental swear words.

If you want density go for Z85, which is a 4 -> 5 byte chunked encoding and therefore much more efficient on a pipelined CPU.

https://rfc.zeromq.org/spec/32/

No, typically the extra characters used are “-“ and “_”. That’s what the table in the IETF link shows.

This isn't a major issue, which means there's no easy answer and it generally comes down to preference if this is a requirement or not.

  UXTerm*charClass: 33:48,36-47:48,58-59:48,61:48,63-64:48,95:48,126:48

FWIW, browser only selects either `foo` or `bar_baz`, but not the whole `foo-bar_baz`.

Well, you're in luck: tilde and dot aren't part of base64url

This isn't a major issue, which means there's no easy answer and it generally comes down to preference if this is a requirement or not.

Since a base64 string with padding is always guaranteed to be a multiple of four characters long, if you get a string that is not a multiple of four in length, you can figure out how much padding it should have had, which tells you how to handle the last three bytes of decoding.

Which makes it a little confusing why base64 needs == padding in the first place.

The base64 under "useful alphabets" is the "natural", iterative divide by radix, base. There's the RFC's "bucket" conversion base under extras.

- when do you use =, when do you use == and do you always add = / == or are there cases where you dont add = / == ?

- how to precisely handle leftover bits. for example the string "5byte". and is there anything to consider when decoding?

For context: since a base64 character represents 6 bits, every block of three data bytes corresponds to a block of four base64 encoded characters. (83 == 24 == 64)

That means it's often convenient to process base64 data 4 characters at a time. (in the same way that it's often convenient to process hexadecimal data 2 characters at a time)

1) You use = to pad the encoded string to a multiple of 4 characters, adding zero, one, or two as needed to hit the next multiple-of-4.

So, "543210" becomes "543210==", "6543210" becomes "6543210=", and "76543210" doesn't need padding.

(You'll never need three = for padding, since one byte of data already needs at least two base64 characters)

2) Leftover bits should just be set to zero; the decoder can see that there's not enough bits for a full byte and discard them.

3) In almost all modern cases, the padding isn't necessary, it's just convention.

The Wikipedia article is pretty exhaustive: https://en.wikipedia.org/wiki/Base64

Padding chars at the end (of stream / file / string) can be inferred from the length already processed, and thus are not strictly necessary.

Note how padding is treated is quite subtle, and has resulted in interesting variations in handling as discussed at: https://eprint.iacr.org/2022/361.pdf

So you're essentially encoding in groups of 24 bits at a time. Once the data ends, you pad out the remainder of the 24 bits with = instead of A because A represents 000000 as data.

For the record, I had to read the whole thing twice to understand that too.

This is because if you’ve only got one of the three bytes you’re going to need, your data looks like this:

   XXXXXXXX

   XXXXXX XX????

   XXXXXX XX0000 000000 000000

The ‘=‘ isn’t just telling you those last 12 bits are zeroes - they’re telling you to ignore the last four bits of the previous digit too.

Similarly with two bytes remaining:

   XXXXXXXX YYYYYYYY

   XXXXXX XXYYYY YYYY??

   XXXXXX XXYYYY YYYY00 000000

Funny side effect of this:

Some base64 decoders will accept a digit right before the padding that isn’t either a multiple of four (with one byte of padding) or of 16 (with two).

They will decode the digit as normal, then discard the lower bits.

That means it’s possible in some decoders for dissimilar base64 strings to decode to the same binary value.

Which can occasionally be a security concern, when base64 strings are checked for equality, rather than their decoded values.

https://github.com/Rezmason/excel_97_egg/blob/main/glsl/base...

I got it down to about thirteen lines of GLSL:

https://github.com/Rezmason/excel_97_egg/blob/main/glsl/base...

I use it for Cursed Mode of my side project, which renders the WebGL framebuffer to a Base64-encoded, 640x480 pixel, indexed color BMP, about 15 times per second:

https://rezmason.github.io/excel_97_egg/?cursed=1

If the length of your input data isn't exactly a multiple of 3 bytes, then encoding it will use either 2 or 3 base64 characters to encode the final 1 or 2 bytes. Since each base64 character is 6 bits, this means you'll be using either 12 or 18 bits to represent 8 or 16 bits. Which means you have an extra 4 or 2 bits which don't encode anything.

In the RFC, encoders are required to set those bits to 0, but decoders only "MAY" choose to reject input which does not have those set to 0. In practice, nothing rejects those by default, and as far as I know only Ruby, Rust, and Go allow you to fail on such inputs - Python has a "validate" option, but it doesn't validate those bits.

The other major difference is in handling of whitespace and other non-base64 characters. A surprising number of implementations, including Python, allow arbitrary characters in the input, and silently ignore them. That's a problem if you get the alphabet wrong - for example, in Python `base64.standard_b64decode(base64.urlsafe_b64encode(b'\xFF\xFE\xFD\xFC'))` will silently give you the wrong output, rather than an error. Ouch!

Another fun fact is that Ruby's base64 encoder will put linebreaks every 60 characters, which is a wild choice because no standard encoding requires lines that short except PEM, but PEM requires _exactly_ 64 characters per line.

I have a writeup of some of the differences among programming languages and some JavaScript libraries here [1], because I'm working on getting a better base64 added to JS [2].

[1] https://gist.github.com/bakkot/16cae276209da91b652c2cb3f612a...

[2] https://github.com/tc39/proposal-arraybuffer-base64

People have been making the argument for over a decade that base64 is incumbent and so people stick with it due to interoperability. But base85 represents a 20% compression gain for basically free from a computational perspective. Isn't that worth switching over as a widely used standard?

An alternative would be the encoding specified by RFC 1924, which uses a different, noncontiguous set of characters. It still has the drawback that dividing by 85 is a bit slower than dividing by 64 (which is just a bit shift).

Last but not least, Base64 has the benefit of being easily recognizable by a human. Due to its relatively restricted character set, it doesn’t look like just line noise, but also doesn’t look like more intelligible syntax, or like hex, etc. It sits in a middle sweet spot.

CPUs were slower relative to memory in those days, so it may have made a difference. Many more programs were CPU bound, and people wrote assembly more often

Also base64 is just easier to code

For example, in JavaScript, that involves making sure it's a well-formed string. I did a write-up of that here: https://web.dev/articles/base64-encoding.

Arithmetic coding will produce an encoding from binary to 50-ary symbols. But then again, there are much simpler ways to do that if exact base conversion is not a requirement (e.g. choose a good enough approximate solution for 2^n = 50^m and do the conversion in n-bit chunks).

You can though? You represent the encoder state in fractions of 50. In practice, it means the encoder state will not use all possible binary strings. But you don't need to trade-off in approximation size.

I am not 100% certain, but pretty convinced you can work with much smaller encoder states of only a few bytes, achieving the same compression performance of chunk approximations requiring kilobytes in size of memory.

usenet exists since 1979/80 [1] and base64 was first described in 1987 [2].

1: https://en.wikipedia.org/wiki/Usenet 2: https://base64.guru/learn/what-is-base64

https://en.wikipedia.org/wiki/BREACH

As these are probably not compressible that means there really isn’t a whole lot lost compared to the optimal solution.

Just to nail the point home - couple of latest snapshots have a bug where they dont clear previous entries in Tabs_ https://www.reddit.com/r/vivaldibrowser/comments/17aqe76/tim... After ~23 restarts Tabs_ file grew to 144MB, still being rewritten every time you close a Tab :o, and grows by 6MB every restart.

How wasteful is Vivaldi/Chrome https://www.reddit.com/r/vivaldibrowser/comments/xu6o3k/bad_... :

\AppData\Local\Vivaldi\User Data\Default\Network\TransportSecurity ~700KB file is being regularly deleted and recreated

https://imgur.com/JAhCV3C

This 700KB file is being rewritten 2-4 times _per minute_ while only updating ~10 entries inside. Here is an example of whats inside TransportSecurity

https://chromium.googlesource.com/chromium/src.git/+/080fbdc...

Those entries arent even important for permanent storage ... and the only data changing is expiry. WTF is going on?

\AppData\Local\Vivaldi\User Data\Default\Preferences again ~700KB, also only useless stuff changes between rewrites, like:

    "last_visited"
    "language_model_counters"
    "predictionmodelfetcher"
    ""expiration"
    "last_modified"
    "visits"
    "lastEngagementTime"

\AppData\Local\Vivaldi\User Data\Default\Sessions\Tabs_ multi megabyte file (~8MB for ~300 tabs) force written with every closing of a tab. Why is it so big? After all its just a list of opened tabs, right? The answer is pretty terrifying. Its full of base64 encoded jpeg thumbnails for _every single tab preview_.

    "thumbnail":"data:image/jpeg;base64

\AppData\Local\Vivaldi\User Data\Default\Sessions\Session_ same as Tabs_, ~same contents including useless base64 encoded jpegs inside, rewritten together with Tabs_.

\AppData\Local\Vivaldi\User Data\Local State 12KB, at this point its almost insignificant that we are constantly rewriting this file. This one keeps such "important" garbage as

    "browser_last_live_timestamp"
    "session_id_generator_last_value"
    "reloads_external"

I sort of understand the logic behind the decision that led to making Tabs_ and Session_ files slam users SSD on every tab interaction - someone was very concerned with Vivaldi constantly losing Tabs and Sessions on browser Crash. But the way this got implemented is backwards and not exactly achieves intended goal.

1 Saves to \AppData\Local\Vivaldi\User Data\Default\Sessions happen ONLY on Tab close, not on Tab Open. You can still open 10 Tabs, have browser crash and lose those tabs.

2 Why would you store base64 encoded jpeg thumbnails when you have image cache storage in the browser?

3 Why flat file rewrites instead of leveraging SQLite, IndexedDB or LevelDB? All available natively in Chrome codebase. Two first journaled, third still claiming crash resistance.

Why am I making this post? 'I mean it's one banana, Michael. What could it cost, $10?' https://www.youtube.com/watch?v=Nl_Qyk9DSUw. Lets count together an average daily session with Vivaldi being open for 6 hours and someone opening and closing ~100 Tabs. TransportSecurity 360 minutes x 0.7 x 2 = 500MB. Preferences 360 minutes x 0.7 x 2 = 500MB. 100 Tab closes 100 x 8 x 2 = 1.6 GB. 2.6GB per day. ~ 1TB/year of system SSD endurance burned doing useless writes. A reminder "A typical TBW figure for a 250 GB SSD lies between 60 and 150 terabytes written"

Edit: Addendum for people assuming its all Chromium fault. Chrome 106.0.5249.91 released 2 days ago. Preferences and TransportSecurity are indeed being written ~1/minute :/, but Tabs_ and Session_ are NOT _on every single tab close_ like in Vivaldi. 15 minute Chrome session resulted in 3xSession_ and 2xTabs_ writes. Chrome also doesnt appear to be storing base64 encoded thumbnails in Tabs_/Session_ files. Looks like the issue is caused by Vivaldis own hack after multiple complaints about crashes resulting in lost data.

Additionally even if you dont care about SSD wear there is also issue of additional power draw for mobile users. Heavy IO is not cheap.

Edit2: Found another one:

Vivaldi creates 9 files in \AppData\Local\Vivaldi\User Data\Default\JumpListIconsVivaldiSpeedDials and immediately deletes them.

What is Jump List anyway? MS says "A jump list is a system-provided menu that appears when the user right-clicks a program in the taskbar or on the Start menu." Does Vivaldi support Jump List in the first place? Chrome does so probably yes.

How To Get Back The Jump List Of Google Chrome In Taskbar https://www.youtube.com/watch?v=WG1tv-kceF4

Sure enough after enabling "show recently opened" Vivaldi does populate it with Speed Dial items. Why is Vivaldi refreshing JumpListIconsVivaldiSpeedDials so often? It tries to Regenerate favicons for jump list items:

- Even when Jump List is disabled.

- Even if NOTHING changed on the Speed Dial.

- Even if Speed Dial "show Favicons" Setting is _Disabled_.

- Despite ALL Vivaldi Jump list Speed Dial entries using default Vivaldi icon and NEVER using favicons.

- EVERY time Session_ file is written, and those are written on _every tab close_ https://www.reddit.com/r/vivaldibrowser/comments/xu6o3k/bad_....

- and then DELETES all the generated data anyway making whole operation a huge waste of CPU and IO resources.

I understand what you mean, of course text can be represented and treated as binary, and the inverse often as well although it isn't necessarily true. Even in Windows-1252, where the upper 127 characters are in use, there are control characters such as null, delete, and EOT which I'd be impressed if a random chat program preserves them across the wire.

I also don't read an implication that ASCII couldn't be converted to b64

Well, there is binary, and there is text. Sure, all text - like "strawman" ;) - is binary somehow, but not all binary data is text, nor can be even interpreted as such, even if you tried really hard ... like all those poor hex editors.

I don't think it implies that at all.

Text isn't binary. Text can be encoded in binary, and there are different ways to do it. ASCII, UTF-8/16/32, latin-1, Shift-JIS, Windows-1252, etc. Many can't encode all text characters, especially languages that don't use the Latin alphabet.

The fact that you have to ensure you're using the correct encoding when processing text from a binary stream is proof enough that text isn't binary. Python before 3.x allowed you to treat binary and text as equal, and it often caused problems.