Against dependency cooldowns as a response to supply chain attacks
Dependency cooldowns are suddenly in vogue. They have quite quickly become widely recommended and it looks like they're going to get fast-tracked into the basket of "industry standard best practices".
The hope is that by just waiting N days after release before adopting a new
version - instead of adopting it immediately - any surreptitiously inserted
hacks will have been discovered by someone else and the bad release "yanked"
(or removed). And this does look, on the surface, like an effective approach:
most supply-chain attacks are indeed detected within a few days.
But while dependency cooldowns are individually modestly - and only modestly - beneficial for those observing them, they place substantial costs onto everyone else. And they don't address the core issue: publishing and distribution are different things and it's not clear why they have to be coupled together.
Dependency cooldowns - the weakness of individual action
Frankly, dependency cooldowns work by free-riding on the pain and suffering of others. Fundamental in the dependency cooldown plan is the hope that other people - those who weren't smart enough to configure a cooldown - serve as unpaid, inadvertent beta testers for newly released packages. If there's a problem, those poor saps get hacked, everyone notices that they got hacked, and the problematic package/executable is yanked before the dependency cooldowners' thresholds are reached.
Even if this worked for individuals - I think it's impossible to sustain it as a sensible or moral system for the entire ecosystem to observe.
Another issue is that dependency cooldowns require a lot of different people to do work. Python has multiple package managers at this point (how many now? 8?). All must implement dependency cooldowns. And every project ever created has to configure the cooldown - which often isn't particularly easy or clear given that package managers often choose completely different ways to do it.
But in fact, even the cooldowners suffer. It's extremely easy to accidentally
circumvent the cooldown that you have sagely configured in your project file.
In Python, a single, personal, pip install litellm outside your project
config would have recently gotten you
hacked. So the cooldown
approach is not actually complete, nor particularly safe.
At some point, probably through either LLM use or old-fashioned copypasta - for that is how the majority of project configurations are created - a "responsible" cooldown then becomes the de facto default. And, to mangle Greenspun, any sufficiently widespread dependency cooldown becomes an ad-hoc, informally specified, hole-ridden, slow implementation of an upload queue.
Upload queues - the many upsides of central action
The obvious alternative is that instead of everyone configuring a cooldown - over and over, again and again - in different package managers and projects that instead we just do it once, a single time, in the central dependency server. An "upload queue". Make new packages wait some period of time after they are published, before they are distributed.
(I use "publication" here to mean sending the release (tarball, whl, gem, whatever) to the central index (npm, pypi, rubygems). Conversely, "Distribution" is when the central index starts serving the release to the public.)
During the time after publication but prior to distribution, you can run internal lint tools, make the package available to external automated security scanners (whose brand names are to be displayed prominently if they find anything), display a public diff of the changes in the built package and even make the queued releases available for intentional, explicitly volunteering beta testers to try out.
Upload queues have precedent. The Debian project uses an upload queue. Packages are uploaded to the repository and then face a minimum wait of 2-10 days prior to making it into the "testing" distribution. An upload queue separates package publication and package distribution.
Publishing is making a package and posting it to the repository. Distribution is when the package is made available to the public. There is no special reason why these two activities need to happen at the same time - it's just a historical accident of how language-specific package indexes grew up.
Upload queues achieve the same goal as dependency cooldowns, but without any of the problems. Upload queues resolve the free-rider problem: no longer are the configurationally-challenged used as free guinea pigs for the cooldowners. Package managers need not implement anything. Projects do not need to add yet another config option. Security linters do not need to flag it. And even when accidentally installing a one-off package on a developer laptop without the configuration, you remain protected.
Removing the element of surprise
Upload queues have another substantial benefit. In the majority of supply chain attacks it is not just the inserted hacks that were unauthorised - the whole release was unauthorised. Making published packages sit around for a few days dramatically reduces the power of release credentials. That's important because making something less powerful is a good second to securing it better.
Making published releases sit around for a few days prior to distribution also removes the entirely unnecessary element of surprise when a new release appears. It gives users advance notice that a new release is coming and foreknowledge of exactly when it will be available.
And it's not just users who need advance knowledge. The upload queue period is also a good time to notify maintainers, to make sure that they are all indeed aware of the forthcoming release. "Notification: Release 2.4.1 has entered the upload queue" would be just the wake-up call required to avert a supply chain attack getting rolled out in many cases.
This applies more than double for AI
People rarely say it so explicitly but, LLMs mean that markdown is now an executable file format. Whether that is exciting or terrifying (or both!) probably comes down to personal taste. But it's a simple fact; and after all that's how Agent Skills work: you download some markdown file, and now your LLM has a new 3rd party dependency.
The first big supply-chain attack on LLMs is doubtless just a matter of time - someone inserting their "Disregard that!" into some popular markdown file.
I've been thinking about this problem for a side project of mine which is a public memory system for AI agents called "Soapstones". It's a place for them (meaning AI agents) to record how to do things; like searching for HN posts, getting the current weather, looking up exchange rates, etc. Soapstones is effectively a package manager for markdown files. And, because it is, the supply chain attack issue exists.
In fact it applies double. Not only might people poison the markdown files with "Disregard that!"'s but LLMs could foolishly upload secret information to the system - such as their API keys.
And the solution here, as well, is an upload queue. In fact, a double upload queue. Moderators review each upload for supply-chain attacks. And the owners of the agents review each upload to make sure they approve of it too.
I think an upload queue is the only reasonable option here - I can't just ask LLMs to "be careful downloading recently uploaded stuff" - that would be madness.
Is funding a problem? Not obviously
One likely retort is "who will pay for this?". Firstly, it's not clear that huge funding is required. The Debian project has maintained an upload queue for decades and has a security team who expedite exceptions for security reasons.
Secondly, it's not all that clear that every important package index is actually strapped for cash. NPM, Inc was a venture-funded startup and is now a wholly owned subsidiary of Microsoft. The Python Software Foundation, who run PyPI, already have a laundry list of corporate sponsors many of whom would benefit from an upload queue. And the PSF even recently took in $1.5m from Anthropic for, among other things: supply-chain security.
But another option is to provide expedited security reviews for commercial projects as a paid service.
Commercial entities are often in a hurry to get a new version out ASAP - to fix some (non-security) bug that is affecting a customer, to make a big announcement or just to bring forward some important server-side deprecation. Simply charge them for expedited review as a paid service.
Expedited review would not be an opt-out from process. It wouldn't mean the release gets distributed instantly as soon as the company hits "publish" on their side - all the automated stuff should still run to completion. Rather, a manual check simply just means wall-clock time can be reduced substantially for a specific release.
Package indexes already need security response teams: yanking releases, maintaining embargoes, dealing with typosquatting and coordinating 0days. Charging commercial projects for expedited review is a good way to cross-fund that. Corporate urgency subsidises the security apparatus that serves the whole ecosystem.
Individually rational, collectively bonkers
Dependency cooldowns are one of those things that serve you somewhat well when you do it yourself. And we all do it to some extent. For some things, I don't want to be the first to upgrade: the family TV set-top box is mission critical infra in my household.
But it's qualitatively different to take personal, subjective responses to each upgrade situation and fossilise them into the community best practices. I don't want my security to depend on someone else getting hacked first.
Contact/etc
Notes
Debian stable effectively is an upload queue - the whole point is that it's made up of older releases that have already been subject to a QA process. The upload queue for language-oriented package managers need not be so comprehensive - but I do think there is something to take from the Debian example.
One thing that is under-discussed is how good the automated scanners are getting. One of the key elements of that is that they examine built artefacts rather than just the upstream source code - which allows them to notice cases where they differ. Giving scanners more time to run (whether by cooldowns or by upload queues) appears to be one of the low hanging fruit.
And another is how dangerous Github actions appears to be. A significant majority of supply-chain attacks appear to rest on exploits to GHA, especially for open source projects where members of the public are able to open PRs.
If you are interested in Soapstones there is a Discord server you can join to talk about it.