一个幽灵正在 Unicode 中游荡
A spectre is haunting Unicode

原始链接: https://www.dampfkraft.com/ghost-characters.html

1978年,日本通商产业省发布了 JIS X 0208 编码标准。其中包含了一些“幽灵文字”——即没有已知发音、含义或明确来源的符号。这些异常字符在长达几十年的时间里一直是个谜,直到1997年的一项调查才试图揭开它们的根源。 研究人员发现,这些字符大多是编目错误的结果。例如,有些字符是在剪贴地名拼件并进行影印时产生的;纸张拼接处的痕迹被误认为是多余的笔画,从而创造出了一个不存在的新字符。虽然大多数幽灵文字最终都被追溯到这类平凡的文书失误,但其中一个名为“彁”的字至今仍无法解释,它可能是某个现有字符的误读版本。 尽管这些幽灵文字源于意外,但它们被永久地固化在 JIS 标准中,并随后被整合进 Unicode。因此,这些幽灵符号如今存在于全球现代数字系统中,成为了字符标准化早期阶段人类失误所留下的永久遗产。

```Hacker News最新 | 过往 | 评论 | 提问 | 展示 | 招聘 | 提交登录一个幽灵正在缠绕 Unicode (dampfkraft.com)14 点,由 sensanaty 发布于 2 小时前 | 隐藏 | 过往 | 收藏 | 1 条评论帮助 joshdavham 9 分钟前 [–] 作者 Paul McCann (polm) 是我最喜欢的程序员之一!过去十年里,他在日语自然语言处理领域做了出色的工作,这对我的语言学习项目帮助很大。他维护着一个 Python 版的 mecab(日语分词器)封装库 [0],写过一本面向英语母语者的日语自然语言处理书籍 [1],还曾在 Spacy 工作过 [2]。[0] https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ [2] https://spacy.io/回复 指南 | 常见问题 | 列表 | API | 安全 | 法律 | 申请加入 YC | 联系 搜索: ```
相关文章

原文

In 1978 Japan's Ministry of Economy, Trade and Industry established the encoding that would later be known as JIS X 0208, which still serves as an important reference for all Japanese encodings. However, after the JIS standard was released people noticed something strange - several of the added characters had no obvious sources, and nobody could tell what they meant or how they should be pronounced. Nobody was sure where they came from. These are what came to be known as the ghost characters (幽霊文字).

Be careful what you write. via the NDL

For a long time the ghost characters remained an unexplained and mostly forgotten curiosity, but in 1997 an investigation was launched to discover where they had come from. While all characters in the JIS standard were supposed to have a record of their sources, even when it existed it wasn't very specific, typically just listing the document it was sourced from.

You'd think that listing the source would make tracking down the origins of the characters easy, but it's important to clarify what counts as a "source" - one of the more common sources for the ghost characters was the "Overview of National Administrative Districts" (国土行政区画総覧), a comprehensive list of place names in Japan. You might, as I initially did, imagine this to be a kind of atlas, an oversize book with at most a few hundred pages. It turns out the latest edition is a seven volume set with each volume having roughly nine hundred pages. Imagine tracking down a single character without a page reference.

Despite the difficulty, the investigation into the ghost characters was successful in discovering their origins - mostly. By interviewing the catalogers involved in the creation of the standard, the investigators established that some characters were inadvertently invented as mistakes in the cataloging process. For example, 妛 was an error introduced while trying to record "山 over 女". "山 over 女" occurs in the name of a particular place and was thus suitable for inclusion in the JIS standard, but because they couldn't print it as one character yet, 山 and 女 were printed separately, cut out, and pasted onto a sheet of paper, and then copied. When reading the copy, the line where the two little pieces of paper met looked like a stroke and was added to the character by mistake. The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me.

The core ghost characters: 妛挧暃椦槞蟐袮閠駲墸壥彁

In the end only one character had neither a clear source nor any historical precedent: 彁. The most likely explanation is that it was created as a misreading of the 彊 character, but no specific incident was uncovered.

Following the general adoption of the JIS standards these characters all made their way into Unicode, which has its own separate set of ghost characters introduced during CJK unification.

To sum up - in 1978 a series of small mistakes created some characters out of nothing. The errors went undiscovered just long enough to be set in stone, and now these ghosts are, at least in potential, a part of every computer on the planet, lurking in the dark corners of character tables.

At this rate they'll presumably be with humanity forever. Ψ

References / related links:

联系我们 contact @ memedata.com