man ascii

It can be a bit confusing, but the gist is that you have 2 chars being show in each line, I would prefer a view where you see the same char with shift and/or ctrl flags, but you can only ask so much

Ended up learning ipfw through the firewall manpage on FreeBSD, and using my skills to setup and manage an IPFW at work.

It's amazing how much you get done with no TV and no internet. Also played a lot of nethack.

Why the hell did I never try this? Maybe because typing ascii table into my favorite search engine and clicking one of the first links was fast enough

`man` is basically manual documentation on anything on your system, not only commands. Most commands do have a manpage for them, but it is not a requirement. The argument of the command is just the file name for the document

You mean 0D and 0A, or 13 and 10, but that mix of base really stood out to me in an otherwise good article. I'm one of numerous others who have memorised most of the base ASCII table, and quite a few of the symbols as well as extended ASCII (CP437), mainly because it comes in handy for reading programs without needing a disassembler. Those who do a lot of web development may find the sequence 3A 2F 2F familiar too, as well as 3Ds and 3Fs.

I can see the rationale for <=> being in that order, but [\] and {|} are less obvious, as well as why their position is 1 column to the left of <=>.

  switch (my_char | 'A' ^ 'a') {
  case 'A' | 'a': /* ... */ break;
  /* ... */
  }

But what’s more, ever wondered whence the control (Ctrl) key presses like Ctrl-H to backspace, or Ctrl-M for carriage return? Well, inspecting the ASCII chart it becomes evident: the Ctrl key simply masks bit 6 (0x40), turning a letter into its respective control character!

It seems to work for codepoints up to U+00FF, for instance:

    - Å (U+00C5) vs å (U+00E5)

    - Ă (U+0102) vs ă (U+0103)

ISO Latin-1 was the character set on many Unix systems, Amiga OS, MS-Windows (as "Windows-1252" with extra chars), and was for many years the default character set on the web.

I'm an emacs user, and when I use a readline-based REPL I use ctrl-M a lot. I thought it was inherited from the emacs keybindings, like many other shortcuts from GNU readline

1. normalization

2. backwards running text (hey, why not add spiral running text?)

3. fonts

4. invisible characters

5. multiple code points with the same glyph

6. glyphs defined by multiple code points (gee, I thought Unicode was to get away with that mess from code pages!)

7. made up languages (Elvish? Come on!)

8. you vote for my made-up emoticon, and I'll vote for yours!

1, 2, 4, 5, 6, and, unfortunately, 8, all fall under "ability to encode written text from all human languages". And that includes historical. Some of the issues (5 & 6) are due semantic difference even if the resulting glyph looks the same. Unfortunately you can't expect programmers to understand pesky little thing like languages having different writing, so you end up with normalisation to handle the fact that one system sent "a + ogonek accent" and another (properly) sent "a with ogonek" (these print the same but are semantically different!), and now you need to figure out normalisation in order to be able to compare strings.

7. just like 8 are down to proposal of specific new forms of writing to add to Unicode. Elvish had one since 1997 but only now got a tentative "we will talk about it". Klingon, which is IIRC more complete language including native speakers (...weird things happened sometimes) does not have outside of private use area.

Emojis were added because they were used with incompatible encodings first, even before unicode happened, and without including something like SIXEL into unicode they were unrepresentable (and with SIXEL would lose semantic information)

I know what its original mission was, which was a character set.

It's been mangled beyond recognition - by including semantic information which is in the purview of context, and presentation information (italics, fonts) which is in the purview of markup languages and layout information (backwards text) which is also in the purview of markup.

> you can't expect programmers to understand pesky little thing like languages having different writing,

But you're requiring programmers to understand all the complicated normalization rules? Normalization is a totally unnecessary feature. Just use the normalized code points. Done.

> these print the same but are semantically different!

Think about what this means. How ever did people manage to read and understand printed books? The semantic meaning comes from the context, not the glyph. For example, I can use `a` to mean the `ahh` sound, or the `ayy` sound, or mean a variable in algebra. How can I know which? The context.

It is totally impossible to add every meaning a glyph has.

> would lose semantic information

Unicode is supposed to be a character set. That's it. Characters do not have semantic information without context.

Oh, and here's some drawkcab text I wrote without any help from Unicode at all.

I had to add some code into the D compiler to reject Unicode text direction "characters" because they can be used to invisibly insert malware into ordinary code.

Adding toy "languages" should be for people having fun, not Unicode.

How can these possibly be semantically different? Isn’t the point of combining characters to create semantic characters that are the combination of those parts?

"Ą" in polish is not "A" with some accent. And the idea behind unicode was to preserve human written text, including keeping track of things like "this is letter A1 with an accent, but this is letter A2 that looks visually similar to A1 with accent but is different semantically". Of course then worries about code page size resulted in the stupidity of Han unification, so Unicode is a bit broken.

Especially because the codepoint is actually called "Combining Ogonek".

And for anyone writing in Cyrillic, it's actually more accurate to use the combining form, even as its own letter, because the only precomposed form technically uses a latin A.

But my main point is that I do not think there is supposed to be any semantic difference in Unicode based on whether you use precomposed or decomposed code points.

There are writing systems where combining accents are used to represent just variation on a letter. Use of combining characters for "Ą" (and "Ć" and "Ł" and many other so-called "polish letters") is, at best, a historical artefact of trying to write them in deficient encodings.

There is no semantic difference, just an encoding one, the end result looks the same and means the same thing (well, to a point, it still depends on the context - like what language you mean - but within the same context it's the same thing and there are even Unicode rules to treat it the same like in search etc.)

And precomposed is just the same historical deficiency - you could've just as well designed a more compact encoding with no precomposed letters, only combinations

Is Unicode kind of messy? Sure, but that's just natural consequences of writing systems being messy. Every point you made was for a sensible reason that is in a scope of Unicode mission (representing all text in all writing systems).

Except it doesn't actually work. 'a' has a zillion different semantic meanings, all dependent on context. There is no crisis with somebody reading a book and misunderstanding which particular semantic meaning it has, because it is inferred from the context.

Semantic meaning always comes from context, and Unicode cannot fix that. People can use the mathematical code point for 'a' instead of the text 'a' and the semantic Unicode meaning is meaningless because the reader will see, like the letter 'a' in because, that it is a text meaning.

The only thing you get with multiple code points for 'a' is you can send out multiple identical appearing texts, but are different Unicode, so you can determine who leaked the memo.

Unicode's extremely limited markup ability helps nobody.

That being said, Emoji are a drop in a bucket when it comes to the number of encoded code points. Nicely enough, by encoding emoji outside the BMP, you can now use characters from astral planes in a lot more places without software breaking.

Unfortunately there is plently of precendent for this ramshacklism. Like ACK/NAK - those are protocol signals, not characters! ENQ? What even is Shift In/Shift Out (SI/SO)? Then the database characters toward the end there FS, RS, GS, US.

> backwards running text (hey, why not add spiral running text?)

You jest, but you do have cursor positioning ANSI sequences which are designed to let text draw anywhere on your screen. And make it blink! You also don't find it weird to have a destructive "clear-screen" sequence?

> glyphs defined by multiple code points

I wonder when they started putting the slash across the 0 to differentiate from the O.

> you vote for my made-up emoticon, and I'll vote for yours!

I mean you do have the private Unicode range where you can actually do that. But before that, SIXEL graphics.

They are not part of the ASCII standard.

> I wonder when they started putting the slash across the 0 to differentiate from the O

That's an issue for fonts, not character sets.

I also half hate emoji as it pollutes human writable text with bitmaps that are difficult to reproduce by hand on paper with a writing instrument - it's not text. I say half hate as it allows us a standard set of icons to use that can be easily rendered in-line with text or on their own.

According to Wikipedia¹, American typewriters were pretty consistent with keyboard layout until the IBM Selectric electric typewriter. Apparently "small" characters (like apostrophe, double-quote, underscore, and hyphen) should be typed with less pressure to avoid damaging the platen, and IBM decided the Selectric could be simpler if those symbols were grouped on dedicated keys instead of sharing keys with "high pressure" symbols, so they shuffled the symbols around a bit, resulting in a layout that would look very familiar to a modern PC user.

Because IBM electric typewriters were so widely used (at least in English speaking countries), any computer company that wanted to sell to businesses wanted a Selectric-style layout, including the IBM PC.

Meanwhile, in other countries where typewriters in general weren't so popular or useful, the earliest computers had ASCII-style punctuation layout for simplicity, and later computers didn't have any pressing need to change, so they stuck with it. Japanese keyboards, for example, are still ASCII-style to this day.

¹: https://en.wikipedia.org/wiki/IBM_Selectric#Keyboard_layout

> The first printing character is space; it’s an invisible character, but it’s still one that has meaning to humans, so it’s not a control character (this sounds obvious today, but it was actually the source of some semantic argument when the ASCII standard was first being discussed).

Hmm.. Interesting that space is considered a printing character while horizontal tab and newline are control characters. They're all invisible and move the cursor, but I guess it makes sense. Space is uniquely very specific in how the cursor is moved one character space, so it's like an invisible character. Newline can either imply movement straight down, or down and to the left, depending on a configuration or platform (e.g. DOS vs UNIX line endings). Horizontal tab can also move you a configurable amount rightwards, and perhaps it might've been thought a bit differently, given there's also a vertical tab, which I've got no idea on how it was used. Maybe it's the newline-equivalent for tables, e.g. "id\tcolor\v1\tred\v2\tblue\v" or something like that.

Interesting also that BS is a control char while DEL is a printing(?) char. I guess that's because BS implies just movement leftwards over the text, while DEL is all ones like running a black sharpie through text. Guess that's what makes it printing. Wonder if there were DEL keys on typewriters that just stamped a black square, and on keypunchers that just punched 7 holes, so people would press "backspace" to go back then "delete" to overwrite.

I've used ASCII a lot, but even after so many years, I'm getting moments where it's like "oh this piece isn't just here, it needs to be here for a deep reason". It's like a jigsaw puzzle.

a figure caption in this page says 'This is a historical throwback to paper tape, where the keyboard would punch some permutation of seven holes to represent the ones and zeros of each character. You can’t delete holes once they’ve been punched, so the only way to mark a character as invalid was to rewind the tape and punch out all the holes in that position: i.e. all 1s.' which is mostly correct, except that it wasn't a historical throwback; paper tape was perhaps the most important medium for ascii not just in 01963 and 01967 but probably in 01973, maybe even in 01977. teletype owners today are still using paper tape that was manufactured during the vietnam war, where it was used in unprecedented volume for routing teletype messages by hand

the dominant early pc operating system, cp/m (if it's not overly grandiose to call it an 'operating system') had system calls for reading and writing the console, the disk, and the paper tape punch and reader. when i hooked up a modem to my cp/m system to call bbses, i hooked it up as the punch and reader

So that's why \177 (DEL) is the loneliest control character. Wow. Thank you!

The "control codes" were to "control" the printhead. So "carriage return" meant move the "print carriage" back to the left margin. "New line" meant move the paper platen one line height of rotation to move the paper to the next line. In that context, "back space" was "move print head one space left" (rather more like a "reverse space"). The article does mention that there was some debate about whether space should be considered "printable", but if you consider a mechanical printer, as the head is moving to the right and banging out characters onto the paper, the spaces between words do, sort of, look like "printables" (of a sort, a "print nothing" character as it were).

Tab's being control characters then make a bit more sense, in that they cause the printhead to jump some fixed distance to the right.

The article stated why DEL is where it is (all ones) -- so that for punched paper tape, one could get a punch-out of every position, which was then interpreted as "nothing here" by the tape reading machine.

As for typewriters, no, none had a "black box" blot out key. Correction (for typewriters without built in correction tape) was one of: retype the page, apply an eraser (and hopefully not damage the paper surface too much) then retype character and continue, or apply correction fluid (white-out) and retype character and continue.

For those typewriters with built in correction tape options (at least some IBM Selectric models, possibly more) the typewriter would retype the character using the "white-out" ribbon, then retype the replacement character using the normal "typewriting" ribbon.

Isn't that incorrect? Tab doesn't jump a "fixed distance to the right," it jumps a variable distance to the next tab-stop to the right.

I saw an analogous use of backspace on some OS I ran into 30 years ago cruising around either Tymnet or TELENET. (I wish I could remember the OS...)

The password prompt assumed local echo. After entering a password the host would send a series of backspaces and various patterns of characters (####, **, etc) to overprint the locally-echoed (and printed) characters.

there was also a solution for cheaper typewriters: small sheets of "white-out" paper (known under the genericized brand name "Tipp-Ex" here in Germany) that you could hold between the ink ribbon and the paper to "overwrite" a typo.

[0]: https://en.m.wikipedia.org/wiki/ISO_2047

When a display terminal has nondestructive backspace (backspace character doesn't erase), it can be software emulated with BS-SPACE-BS.

At your Linux terminal, you can do "stty echoe" (echo erase) to turn this on (affecting the echoing of backspace characters that are input, not all backspace characters).

Dial-up BBSes had this as a configurable setting also.

Instead, we crudely use commas and tabs as delimiters instead of something like RS (#30).

Values were separated with char(254), subvalues were separated with char(253), and the third level were char(252) separated.

It was... unique, but worked. And to be fair, PICK originated in the 60's, so this method probably evolved in parallel to the ASCII table!

The only thing you'd need is editors to support some way of entering and displaying the RS, and CTRL+^ is a bit of a kludge as it ends up CTRL+SHIFT+6.

Of course, if a record itself can contain RS for subrecords, things become more complicated. I guess you could use `\^`.

What do you do if you have a record that includes a record separator character? Given that you have this problem anyway, why do you want a character dedicated to achieving the same thing that a comma achieves?

Of course, the fact that record separators aren't on keyboards is probably why CSVs use commas.

It was probably possible to type an RS with Ctrl+Shift+. and the others with similar combos.

Bash will insert the control character (rather than interpret it) if you prefix it with Ctrl-V.

It is still valid to disallow the ASCII control characters, one just has to make sure that it is done comprehensively, in all places users may input them. But that's not created by using ASCII control characters, that's a consequence of the "ban the control characters entirely" approach regardless of what the control characters are.

It's neat when you can get away with it, but I generally prefer to define a robust encoding scheme instead. A minimal one like "replace backslash with double-backslash, replace control characters with backslashed characters" and "replace backslash sequences with their control characters, including backslash-backslash as a single backslash" can be inserted almost anywhere in just a few lines of string replace (or stream processing if you need the speed). The only tricky bit is you need to make sure you get the order correct or you corrupt data, and while I've done this enough to have it almost memorized now I do recall feeling like the correct order is backwards from what I naturally wanted the first few times. But it is simple and robust if you get it right.

(A trivial evening project for some; not for all of us)

No, they'll be right exactly as often, 0% of the time.

But their mistake will show up less frequently, causing more problems when it does.

As soon as it's possible for some of your data to come from someone else's dataset, you're guaranteed to have to accommodate record separators within your data as well as within the metadata. You're better off using a system that plans for this inevitability than one that pretends it can't happen at all.

> But their mistake will show up less frequently, causing more problems when it does.

Enough people use CSVs (and have limited, small-scale use-cases) that I'd be willing to bet "less frequently" means never for at least 1% of people who use CSVs.

I don't know whether the chance of no problems is worth the increased difficulty of problems that do occur - considering that balance feels a bit silly because if you're aware there could be a problem in a context where you could choose between commas and unit separators, you could just add validation or escaping.

As soon as you have validation or escaping, having a record separator character loses its entire purpose. The existence of the character is predicated on the idea that you don't have to do that, and that idea is false.

That's why the character is never used. It's a conceptual mistake that was accidentally enshrined in a series of encoding standards that had enough free space to accommodate it.

I disagree with this - the data needs to be stored somehow, and while other characters (like comma) can be used, having a dedicated character can help - for example if the data might legitimately contain commas or newlines but not unit separators or record separators, then escaping isn't needed if you use unit/record separators (although validation is still necessary).

TSV is widely used, but lacks a way to escape the tab and new line characterss. RS-V is the same, but allows including tabs and new lines in records.

This comes up every time. Options:

1. You disallow it. And you might as well disallow all the control codes except the carriage return, line feed, and other “spacing” characters. Because what are they doing in the data proper? They are in-band signals.

2. You use the Escape character to escape them

3. Weirdest option: if you really want to nest in a limited way you can still use the group and file separator characters

The order used by ASCII is sometimes called "ASCIIbetical", which I think is wonderful.

https://en.wiktionary.org/wiki/ASCIIbetical

Such a waste and no extensibility kills and claim to elegance of some shifted binary numbers, that'd the wrong end to focus your optimization efforts on

  man ascii

  % (uname; cd /usr/ports; ls -d */ascii)
  FreeBSD
  zsh: no matches found: */ascii
  % which ascii
  ascii not found
  %

So you will either get command not found, or man page not found.

The man page comes preinstalled on most modern non-embedded POSIX systems. The command does not.

It also makes clear why ESC can be entered as `^[` or ENTER (technically CR) as `^M` on some terminals (still works in my xterm), because the effect of the control key is to unset bits 6 and 7 in the original set-up.

Of course you can color in the fields too, if you want.

The charts that simply show you the assignments in hex and octal obscure the elegance of the design.

https://en.wikipedia.org/wiki/EBCDIC

On the 4th floor of my building the computer systems lab has a glass front that has what looks like a punch card etched in frosted glass but if you look closer it was made by sticking stickers on the glass.

I made a "punchcard decoder" on a 4x6 card to help people decode the message on the wall

https://mastodon.social/@UP8/112836035703067309

The EBCDIC code was designed to be compatible with this encoding which has all sorts of weird features, for instance the "/" right between "R" and "Z"; letters don't form a consecutive block so testing to see if a char is a letter is more complex than in ASCII.

I am thinking of redoing that card to put the alphabet in order. A column in a punched card has between 0 to 3 punches, 0 is a space, 1 is a letter or a symbol in the first column, if one of the rows at the top is punched you combine that with the number of the other punched row on the left 3x9 grid. If three holes are punched one of them is an 8 (unless you've got one of the extended charsets) and you have one of the symbols in the right 3x6. Note the ¬ and ¢ which are not in ASCII but are in latin-1.

Except people who want to use Japanese and not have it render weirdly, something that was easy in any internationalised software that used the traditional codepage system, but is practically impossible in Unicode-based software.

But that only causes confusion because you need to provide external information which way to interpret them, just like a code page

No amount of extra characters was going to address what Unicode did.

ASCII was not a mistake at all. Adopting it unified what was surely going to be a real mess.

At the time it made sense, and the control functions were needed. Still are.

Control characters were needed for terminals. They never made sense for text. Mixing the two matters is the problem.

Oh yeah, lets not forget that at the time you had other nationalistic countries/territories/people with their own superior technology all vying for the top position, all well trying to out do each other. Then you also had manipulative monopolies/trade embargo's and wars.

It isn't perfect but people aren't perfect.

CR and LF aren't dedicated, and have precise cursor movement meanings, rather than being a logical line ender.

There was a proposal in the 80s to reassigning the -otherwise useless- VT (vertical tab) character for the purpose. Unfortunately unfruitful.

The filesystems I used had 5 kinds of file: random-access, sequential, ISAM, Fortran-carriage-control and carriage-return-carriage-control. The only people who used the latter were the eggheads that used that new-fangled C programming language brought over from Bell Lab's experimental Unix system.

You're probably just looking for the record separator (036). If you are storing multiple text records and a block of memory, that would be the ideal ASCII code to separate them.

Fortunately unfruitful, since if it had gained adoption, there'd be a mix of three different line endings (and combinations thereof) in widespread use, instead of two.

Favorite fact is that 127 is the DEL because for hole punching it removes all the info. I love those little nuggets of history

bob bemer more or less invented ascii. he was also an ibm guy before mackenzie's crowd pushed him out of ibm for promoting it. he wrote a much better book about the history of ascii which is also freely available online, really more a pamphlet than a book, called "a story of ascii": https://archive.org/details/ascii-bemer/page/n1/mode/2up

tom jennings, who invented fido, also wrote a history of ascii, called 'an annotated history of some character codes or ascii: american standard code for information infiltration'; it's no longer online at his own site, but for the time being the archive has preserved it: https://web.archive.org/web/20100414012008/http://wps.com/pr...

jennings's history is animated by a palpable rage at mackenzie's self-serving account of the history of ascii, partly because bemer hadn't really told his own story publicly. so jennings goes so far as to write punchcard codes (and mackenzie) out of ascii's history entirely, deriving it purely from teletypewriter codes—from which it does undeniably draw many features, but after all, bemer was a punchcard guy, and ascii's many excellent virtues for collation show it

as dwheeler points out, the accomplished informatics archivist eric fischer has also written an excellent history of the evolution of ascii. though, unlike bemer, fischer wasn't actually at the standardization meetings that created ascii, he is more careful and digs deeper than either bemer or jennings, so it might be better to read him first: https://archive.org/details/enf-ascii/

it would be a mistake to credit ascii entirely to bemer; aside from the relatively minor changes in 01967 (including making lowercase official), the draft was extensively revised by the standards committees in the years leading up to 01963, including dramatic improvements in the control-character set

for the historical relationship between ascii character codes and keyboard layouts, see https://en.wikipedia.org/wiki/Bit-paired_keyboard

    Typewriter: !@#$%^&*()
    Apple:      !"#$%&'()
    Digits:     1234567890

>A bit-paired keyboard is a keyboard where the layout of shifted keys corresponds to columns in the ASCII (1963) table, archetypally the Teletype Model 33 (1963) keyboard. This was later contrasted with a typewriter-paired keyboard, where the layout of shifted keys corresponds to electric typewriter layouts, notably the IBM Selectric (1961). The difference is most visible in the digits row (top row): compared with mechanical typewriters, bit-paired keyboards remove the _ character from 6 and shift the remaining &() from 7890 to 6789, while typewriter-paired keyboards replace 3 characters: ⇧ Shift+2 from " to @ ⇧ Shift+6 from _ to ^ and ⇧ Shift+8 from ' to . An important subtlety is that ASCII was based on mechanical typewriters, but electric typewriters became popular during the same period that ASCII was adopted, and made their own changes to layout.[1] Thus differences between bit-paired and (electric) typewriter-paired keyboards are due to the differences of both of these from earlier mechanical typewriters.

>[...] Bit-paired keyboard layouts survive today only in the standard Japanese keyboard layout, which has all shifted values of digits in the bit-paired layout.

>[...] For this reason, among others (such as ease of collation), the ASCII standard strove to organize the code points so that shifting could be implemented by simply toggling a bit. This is most conspicuous in uppercase and lowercase characters: uppercase characters are in columns 4 (100) and 5 (101), while the corresponding lowercase characters are in columns 6 (110) and 7 (111), requiring only toggling the 6th bit (2nd high bit) to switch case; as there are only 26 letters, the remaining 6 points in each column were occupied by symbols or, in one case, a control character (DEL, in 127).

>[...] In the US, bit-paired keyboards continued to be used into the 1970s, including on electronic keyboards like the HP 2640 terminal (1975) and the first model Apple II computer (1977).

In base-2 machines, the letters are mixed with punctuation, which is pretty horrible design which makes simple things complex, and does not actually bring anything new to the table.

In BCD machines it is slightly better, except letters aren't contiguous either - row 0 is bad, but it's the extra space between R and S which is really ugly. And it's unusable with BCD operations anyway, as high nibble values are used extensively.

Naive sorting simply does not work... lowercase before uppercase, punctuation in the middle of the alphabet, numbers after letters.

I see no elegance there, it's like the worst example of legacy code.

And the whole punch card -> 8-bit is pretty illogical, just like the cards themselves. How come no punches in zone don't correspond to 0 high bits?

(and don't get me started on punch card.. it started with "let's do 1 hole per column for digits" - OK, makes sense; then "let's do 2 hole/column for uppercase" - I guess OK but why did you put extra char in the middle... but then it's 4 holes/column for superscripts? 3-6 holes/col for punctuation? If someone were to design punch cards today but using same requirements, they could easily come up with a much more logical schema)

OK a bit over the top ... the designers of EBCSDIC had a rather tight set of constraints to deal with, none of which included: "be inclusive". Again, if I really had to be charitable (I looked after a System/36, back in the day), the hardware was rather shit too, sorry ... constrained. Yes constrained. Why should six inch fans fire up reliably after a few years of use and not need a poke after an IPL? No real dust snags and I carefully sprayed some WD40 on the one that I could get at. I have modern Dells and HPs in horrid environments that do better with shitty plastic fans.

EBCDIC is not elegant at all unless excluding non English characters in an encoding system is your idea of elegant.

According to this: https://en.wikipedia.org/wiki/EBCDIC it expended loads of effort with dealing with control eg: "SM/SW" instead of language.

ASCII and EBCDIC and that basically say: fuck you foreigners!

We now have hardware that is apparently capable of messianic feats. Let's do the entirety of humanity some justice and really do something elegant. It won't involve EBCDIC.