放弃 Scientific Linux 是个错误
Abandoning Scientific Linux Was a Mistake

原始链接: https://blog.melashri.net/posts/scientific-linux-mistake/

CERN 最近决定将其 2200 个加速器系统迁移至 Debian,这为基础设施管理带来了一个深刻的教训:将标准化误认为是对厂商的依赖是极其危险的。 十多年前,CERN 和费米实验室(Fermilab)退役了由社区维护、与 RHEL 兼容的发行版“Scientific Linux”。当时,这一决定看似理性,是为了避免重复 CentOS 已完成的工作而采取的成本节约措施。回过头来看,作者认为这是一个错误。Scientific Linux 不仅仅提供了软件,它还提供了“期权价值”——一种受机构自主控制的能力,当上游供应商(如红帽)改变政策时,这是一种可靠的退出策略。 由于拆解了内部专业知识并向商业化生态系统靠拢,这些机构失去了独立转向的能力。随后,由 CentOS 的转型以及硬件支持的变动所带来的动荡,迫使它们进行了昂贵的迁移。此次转向 Debian 凸显出机构的独立性值得为其付出代价。作者最终指出,虽然标准化对于效率至关重要,但研究机构必须保护其基础设施的自治能力。从长远来看,机构的韧性远比通过将关键系统外包给单一供应商所节省的短期工程工时更有价值。

这篇 Hacker News 讨论围绕着一篇名为《放弃 Scientific Linux 是个错误》的博文展开。社区的反应主要呈批评态度,集中在两个方面: 1. **技术批评:** 评论者认为作者对 Scientific Linux 的赞美过于模糊。一些用户分享了关于该系统软件包过时的负面经历(例如旧版本的 OpenSSL),这阻碍了开发工作。另一些人则质疑作者为何不直接转向 Rocky Linux、AlmaLinux 或 Debian 等现代替代方案。 2. **人工智能相关担忧:** 很大一部分讨论对文章的写作风格表达了不满。许多用户指出了其中的“大模型腔调”,认为这些文风特征暗示文章是由人工智能生成的。批评者将此类内容贴上“低质量投入”和“垃圾内容”的标签,并指出平台上此类 AI 生成内容日益泛滥,已成为越来越令人困扰的问题。 总的来说,该讨论串反映出一个持怀疑态度的受众群体,他们更看重原创的高质量人类写作和切实的技术解决方案,而非他们眼中泛泛而谈的机器生成内容。
相关文章

原文

A recent CERN announcement caught my attention. By the end of 2026, more than 2,200 industrial computers and embedded systems around CERN's accelerator complex are expected to be running Debian 13.

CERN is not abandoning the Red Hat ecosystem. AlmaLinux and RHEL remain the main supported Linux distributions across much of the organization; Debian support is currently limited to accelerator front-end systems.

Still, the move took me back to a decision made more than a decade ago: CERN and Fermilab's gradual abandonment of Scientific Linux. With hindsight, I think it was a mistake.

Not because Scientific Linux was technically superior to CentOS, or because maintaining another Linux distribution was free. And not because CERN or Fermilab could somehow have controlled what Red Hat or IBM later chose to do.

The mistake was treating Scientific Linux mainly as duplicated engineering work. It was infrastructure.

What Scientific Linux was solving

Scientific Linux grew out of a practical problem in High Energy Physics. Large experiments span laboratories, universities, computing centers, and countries. If one site builds against one version of glibc, another uses something slightly different, and a third runs an entirely different packaging environment, things become painful very quickly.

The scientific community also needed an unusual combination: a Linux distribution that would remain stable for years, work with enterprise software, be freely redistributable, and run across institutions without requiring a commercial license for every machine.

Red Hat Enterprise Linux provided the stability and long lifecycle, and Red Hat published the source needed to rebuild it. Scientific Linux turned that source into a community resource. Fermilab announced the distribution in 2003, and CERN joined soon afterward. It eventually spread far beyond those two laboratories. Universities, research institutions, experiments, companies, and even systems aboard the International Space Station used it or distributions derived from it.

Scientific Linux was never simply "RHEL with a different wallpaper." It gave the scientific community an institutionally independent implementation of the Enterprise Linux platform.

Then CentOS looked like the obvious answer

When Red Hat and CentOS joined forces in 2014, moving to CentOS looked perfectly reasonable. CentOS already offered what many Scientific Linux users wanted: a freely available Enterprise Linux rebuild backed by a much larger general-purpose community.

CERN began moving its next major release from Scientific Linux CERN to CERN CentOS 7. Scientific Linux 5 and 6 remained supported, but the future platform at CERN would be based on CentOS. The rationale was compelling. Why should CERN and Fermilab spend scarce engineering time rebuilding a Linux distribution when CentOS was already doing essentially the same work?

Why maintain an HEP-specific distribution when the scientific community could converge on a larger common platform? If Red Hat itself was supporting the CentOS project, that seemed to make the platform more sustainable, not less. In 2019, Fermilab took the argument to its logical conclusion or its literal "end." There would be no Scientific Linux 8. Fermilab would deploy CentOS 8 instead and work with CERN and other laboratories to improve CentOS for high-energy physics.

Taken on its own, this was not an irrational decision. But it removed something that was hard to see on a spreadsheet: a credible exit.

A distribution has option value

This is what I think we underestimated.

The cost of maintaining Scientific Linux was visible. People had to rebuild packages, test updates, maintain repositories, produce releases, handle security updates, and support users.

The benefit of an independent distribution was harder to quantify. As long as CentOS behaved exactly as CERN and Fermilab expected, Scientific Linux looked redundant. That is how redundancy always looks when nothing has failed yet.

Scientific Linux gave the scientific community its own implementation of a RHEL-compatible computing environment. More importantly, it kept alive the people, processes, infrastructure, and institutional knowledge needed to maintain one.

That capability had option value. You might not need to exercise the option this year, or even this decade. It becomes valuable when the assumptions underneath your primary platform change. Those assumptions changed surprisingly quickly.

[!NOTE]

CERN uses a similar argument to justify the FCC: the knowledge required to build large machines such as accelerators and detectors is valuable and must be preserved. If Europe does not build the next machine after the LHC, that expertise may be lost, the next large machine may be built in China, and Europe may lose its technological leadership in the field. The same argument applies to software infrastructure.

The CentOS assumption did not last

In December 2020, Red Hat changed the role of CentOS.

CentOS Linux, the traditional downstream rebuild of released RHEL versions was discontinued in favor of CentOS Stream, which sits ahead of RHEL rather than behind it. This was more than a change in release cadence.

Organizations had standardized on CentOS because they wanted a freely distributable approximation of the current RHEL release. The product they had standardized on effectively ceased to exist.

CERN and Fermilab immediately had to reconsider their Linux strategy. CERN noted that the shorter CentOS Stream lifecycle was incompatible with some scientific use cases. The laboratories evaluated Stream, RHEL licensing arrangements, and the emerging Enterprise Linux rebuilds. By 2022, CERN and Fermilab were recommending AlmaLinux as the standard distribution for experiments.

The circularity is hard to miss.

Scientific Linux had been retired partly because maintaining another RHEL rebuild seemed unnecessary while CentOS existed. A few years later, the community needed an independent RHEL-compatible distribution again, so CERN and Fermilab adopted another independently governed RHEL-compatible distribution.

AlmaLinux is a good project. This is not a criticism of it. The requirement never disappeared; only our implementation of it did.

Then the ground moved again

In 2023, Red Hat changed how RHEL-related sources were publicly distributed. CentOS Stream became the sole public repository for that source material, replacing the previous git.centos.org publication model.

This did not make RHEL "closed source," despite how the change was sometimes described. That characterization would be inaccurate. But it made the dependency structure of the Enterprise Linux rebuild ecosystem much more obvious.

By then, the lesson should have been familiar: technical compatibility with an upstream product is not the same as independence from the organization that controls it.

Scientific Linux had strategic value that was never properly accounted for. Its existence meant CERN, Fermilab, and other institutions were not merely consumers of an ecosystem. Together, they could reproduce a critical part of that ecosystem themselves.

Once discarded, that capability is much harder to recreate than to keep alive.

And now CERN is moving part of the accelerator complex to Debian

The latest chapter makes this history particularly interesting. CERN's accelerator controls group first tried to remain within the Red Hat ecosystem. According to the presentation at MiniDebConf Winterthur 2026, the original plan involved CentOS Stream, with Debian prepared as a fallback because, in the presenters' words, they could not afford new surprises.

Then came a more physical problem. RHEL 9 raised its x86-64 baseline to x86-64-v2. RHEL 10 moved to x86-64-v3.

Replacing old hardware may be inconvenient but manageable for an ordinary server fleet. Accelerator controls are not an ordinary server fleet.

These computers interface with custom electronics, specialized boards, legacy buses, real-time systems, and equipment installed throughout a vast accelerator complex. The hardware can remain operational for decades because replacing one computer may mean redesigning the electronics connected to it.

CERN's 2023 risk analysis estimated that staying in the Red Hat ecosystem could cost about 5.4 million CHF. It could also require redesigning around eleven boards, hiring more engineers and technicians, reorganizing racks, recabling systems, and recommissioning equipment.

The solution was refreshingly simple: do not replace millions of francs' worth of functioning hardware to satisfy an operating system's CPU baseline. Replace the operating system.

CERN chose Debian. By the end of 2026, more than 2,200 industrial computers and embedded systems around the accelerator complex are expected to run Debian 13.

The irony is difficult to miss

What interests me most is not that CERN chose Debian, but why Debian was valuable.

The accelerator team specifically praised its continued support for older and less common architectures and its ability, as a community-led distribution, to compete with Red Hat.

CERN is also sponsoring Freexian to strengthen Debian's long-term-support ecosystem. There is an important principle here: when an institution depends on infrastructure supplied by a community project, contributing resources to keep that project independent and healthy may cost less than consolidating everything around one vendor ecosystem in pursuit of short-term efficiency.

That sounds remarkably close to what Scientific Linux used to provide.

Scientific Linux was maintained by a surprisingly small group

This is another reason I question whether ending Scientific Linux produced the savings we assumed. The distribution was not maintained by hundreds of engineers.

Its project history lists only a handful of major developers in its later years, with Fermilab providing the main sponsorship, build infrastructure, bandwidth, and website. That does not make the work trivial. Release engineering, security updates, testing, package rebuilds, infrastructure, and user support all take real time.

But those costs should be compared with the value they produced. Scientific Linux provided a stable computing platform to an enormous scientific ecosystem for roughly two decades. Preserving that capability should not have been evaluated solely in terms of the engineer-hours saved by adopting CentOS.

The relevant question was: what would it cost to lose the ability to operate independently if the surrounding ecosystem changed?

The past decade gives us at least a partial answer.

Did Scientific Linux keep Red Hat honest?

There is a tempting, stronger argument: that Scientific Linux and other independent rebuilds constrained Red Hat, and that removing one of the major alternatives helped concentrate power around CentOS and RHEL.

I think there is something to it, but I would not present it as historical fact. We cannot know the counterfactual.

Scientific Linux probably would not have prevented Red Hat from changing CentOS. IBM completed its acquisition of Red Hat in July 2019, after Fermilab had already announced that there would be no Scientific Linux 8. It would therefore be misleading to reduce this to a story about IBM killing something CERN and Fermilab should have anticipated.

What I do believe is that independent alternatives change incentives. A vendor behaves differently when major customers and institutions have a credible exit than when leaving requires them to rebuild years of infrastructure.

This is not specific to Red Hat. It is basic dependency management. Competition matters even when nobody switches; the possibility of switching matters too. Scientific Linux provided that possibility.

And because major scientific institutions maintained it rather than a commercial Linux vendor, its incentives were unusually well aligned with long-lived scientific infrastructure.

The real mistake was confusing standardization with dependency

Standardization is good.

Particle physics could not operate efficiently if every university and laboratory invented its own incompatible computing platform. CERN and Fermilab were right to want common interfaces, compatible binaries, common packaging, and predictable operating environments. But standardization and monoculture are not the same thing.

The strategy went wrong when we assumed that standardization required abandoning an independent implementation of the standard. Scientific Linux could have remained boring.

In fact, boring was exactly what we needed from it. It did not need to compete with Fedora on innovation or Ubuntu on desktop adoption. It needed to remain a reproducible, institutionally controlled Enterprise Linux platform: somewhere the scientific community could go if the commercial ecosystem moved in a direction incompatible with scientific computing.

That is valuable infrastructure even when almost nobody notices it. Especially then.

The lesson is larger than Scientific Linux

I do not think the answer today is to resurrect Scientific Linux. The ecosystem has moved on. AlmaLinux exists. Rocky Linux exists. Debian is proving increasingly useful at CERN. Containers and modern software distribution have reduced the importance of the host operating system for many workloads.

The lesson I take from Scientific Linux is about how research institutions value infrastructure. We are good at calculating the immediate cost of maintaining something ourselves. We are much worse at calculating the long-term cost of losing the ability to maintain it.

When deciding whether to retire institutional open-source infrastructure, the calculation should include more than maintenance hours. It should account for governance, concentration risk, the cost of migration if an upstream project changes direction, the institutional knowledge being discarded, and the value of a credible alternative that you hope never to need.

The CERN accelerator team's final advice in its Debian presentation is probably the best summary:

"Distribution portability is a very good thing."

It took more than a decade, several Enterprise Linux strategy changes, a new community rebuild, and now the migration of thousands of accelerator computers to arrive there. Scientific Linux was already teaching us that lesson twenty years ago.

We just stopped listening.

联系我们 contact @ memedata.com