This, to me, is the most important question. There is no way Andres Freund just happened to find the _only_ backdoored popular open source project out there. There must be like a dozen of these things in the wild?

There's multiple things like this in this backdoor that seems like they've been super sneaky (using a compile check to disable Landlock is genius) but then half-assed the last step.

And that's with the failure party being just accidentally borking it so the test always fails or always succeeds when it shouldn't. You can see why it's a juicy target for malicious actions.

    bad-3-corrupt_lzma2.xz has three Streams in it. The first and third
    streams are valid xz Streams. The middle Stream has a correct Stream
    Header, Block Header, Index and Stream Footer. Only the LZMA2 data
    is corrupt. This file should decompress if --single-stream is used.

    $ cat tests/files/bad-3-corrupt_lzma2.xz | xz -d --single-stream
    ####Hello####

And lastly, they act as a marker that can be used by a later regex to locate the file _without_ referencing it by name directly nor using the actual Hello and World strings.

    $ gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null`
    $ echo $gl_am_configmake
    ./tests/files/bad-3-corrupt_lzma2.xz

I don't know how common this practice is in the linux distro builds. Obviously if it's common, it would take a lot of work to clean up to make this possible, even if it's even possible in the first place. It seems like something that would be doable with bazel, but I'm not sure about other build systems.

i live in a different part of the dev world, but could this be written to be less obtuse so it’s more obvious what’s happening?

i get that a maintainer can still get malicious code in without the same rigor as an unaffiliated contributor, but surely there’s a better way than piles of “concise” (inadvertently obfuscated?) code?

It's a very old smell, basically.

The central problem is that back in the 80s and 90s, there were scads of different Unix-like systems, each with their own warts and missing features. And software authors wanted to minimize their build dependencies.

So many communities standardized on automating builds using shell scripts, which worked everywhere. But shell scripts were a pain to write, so people would generate shell scripts using tools like the M4 macro preprocessor.

And this is why many projects have a giant mass of opaque shell scripts, just in case someone wants to run the code on AIX or some broken ancient Unix.

If you wanted to get rid of these impenetrable thickets of shell, you could:

1. Sharply limit the number of platforms you support.

2. You could standardize on much cleaner build tools.

3. You could build more key infrastructure in languages which don't require shell to build portably.

But this would be a massive undertaking, and a ton of key C libraries are maintained by one or two unpaid volunteers. And dropping support for "Obscurnix-1997" tends to be a fairly controversial decision.

So much of our key infrastructure remains surrounded by a morass of mysterious machine-generated shell scripts.

but this also shows what's really missing from these old projects, infrastructure, QA, CI, modern tools, etc.

and adding these requires humans in the loops, and every change needs to be checked, verified, etc.

and it's a hard task for a loose community of volunteers.

even the super fancy Rust community kind of shrugged and let crev die silently

https://github.com/crev-dev/cargo-crev

This is how a non negligible number of your tools are built.

At a glance, I don't think so. At least, not the fact that the bash looks like a convoluted mess. Sometimes that's how a tight bash script looks (whether it SHOULD be tight or not is a different argument).

For me, the thing that looks suspicious is the repeated tr calls. If I see that, I assume someone is trying to be clever, where 'clever' here is a pejorative. If I were a maintainer and someone checked that in, I'd ask them to walk me through what it was doing, because there's almost always a better solution to avoid chaining things like that.

The real problem here is that there wasn't another maintainer to look at this code being brought in. A critical piece of the stack relied on a single person who, in this case, was malicious.

You get a giant .sh file that displays a license, asks you to accept, then upon acceptance, cats itself, pipes through head/tail, into cpio to extract the actual assets.

This is in general a problem with traditional permission models. Pure capabilities would never leave binaries writable by anything other than the compiler/linker, shrinking the attack surface.

Nobody in Java word is linking against test resources in prod, and a sanely designed build system should in fact make this extremely difficult. That shit went away in the maven/gradle days - which is for a reason, ant is basically makefiles for Java, gradle/maven are a build system not a pile of scripts. And that transition happened 20 years ago!

If you can’t even prevent a test resource being linked into a final build you are not serious. I don’t care about legacy whatever, that’s an obvious baseline metric for security culture / build engineering.

Maybe not “prevent” but like, tooling should absolutely make it blindingly obvious that you’re violating best-practices by disabling scoping rules or including unusual source/resource roots, etc.

C has never moved past the 1970 mindset of build being a pile of scripts with a superstructure bolted on. Just like Ant. Even the attempts to fix C’s build are just better ways to programmatically generate better bash scripts to keep you going off the rails.

edit: also this was heavily obfuscated in some binary files that were marked as test files ("good" and "bad" xz compressed test file). No way to spot this if you don't know what you're looking for.

Code reviews are more about education and de-siloing.

What other controls would you use to prevent them contributing malicious commits, besides closely reading your co-contributor's commits, and disallowing noisy commits that you don't fully comprehend and vouch for?

We assume that it'd be unethical to surveil the contributor well enough to detect the change in alliance. That would violate their privacy.

Is it reasonable to say, "game over, I lose" in that context? To that end, we might argue that an embedded mole will always think of ways to fool our review, so this kind of compromise is fatal.

But let's assume it's not game over. You have an advanced persistent threat, and you've got a chance to defeat them. What, besides reviewing the code, do you do?

I would expect most people to at least ask for more clarification on random changes to `head` offsets, honestly - or any other diff there.

If they had access to just merge whatever with no oversight, I guess the blame is more on people using this in other projects without vetting their basic security of projects they fully, implicitly trust, though. As bad as pulling in "left-pad" in your password hashing lib at that point.

The "random binaries in the repo" part is also egregious, but more understandable. Still not something that should have gotten past another pair of eyes, IMHO.

this sort of vetting you're talking about is gonna turn up nothing. Most vetting is at the source code level anyway, not in the tests, nor the build files tbh. It's almost like a checkbox "cover your ass" type work that a hired consultant would do.

Unless you're someone in gov't/military, in which case yes, you'd vet the code deeply. But that costs an arm and a leg. Would a datacenter/hosting company running ssh servers do that?

> xz -dc $top_srcdir/tests/files/$p | eval $i | LC_ALL=C sed "s/\(.\)/\1\n/g" | LC_ALL=C awk 'BEGIN{FS="\n";RS="\n";ORS="";m=256;for(i=0;i

You should probably expect the potential for abuse?

We’re moving towards complexity that is outpacing human ability for any one person to understand, explain, and thus check an entire object.

And for what? Build efficiency? Making a “trick” thing? When was the project ever going to go back and make things simpler? (Never)

This is 1990s autoconf bs that has not yet been excised from the Linux ecosystem. Every modern build system, even the really obtuse ones, are less insane than autoconf.

And the original purpose of this was not for efficiency, but to support a huge variety of target OSes/distros/architectures, most of which are no longer used in any real capacity.

None of this would have been visible in commits or diffs at all.

There are many potential reasons to publish a release tarball where some of these steps are already done. It could be done in a reproducible way. Look at sqlite for an example of an extremely well maintained open source library that publishes not one but two source code tarballs for various stages of the build.

These calls to change source code distribution just because it was a small part of the attack vector in this particular case seems misguided to me. It may still be a good idea but only as part of a much larger effort for reproducible builds. In itself it would accomplish nothing, apart from a wake of uncertainty that would only make future attacks easier. Especially in this case, where the maintainer could have acted in a number of other ways, and indeed did. The entirety of the backdoor was added in a regular git commit a long time ago.

I believe what happens in debian is that they host their own mirror of source tarballs, since going to some random website or git repo means it could be taken down from under them. So I guess if the package is built straight from a repo they'd probably make a tarball of it anyway.

Until last month, would you refuse a tar package from the official maintainer, I wouldn't, especially when there was a mention of a bugfix that might have been tripping our build system

For example nginx is using mercurial (with admittedly a github mirror for convenience), and a lot of OSS are still using subversion and CVS, and my guess is that there are some project which might run with less free source control software ( most likely for historical purpose, or use case that might be the strong point of that software).

Other than that, why wouldn't the user be the one to build their own software package.

when binary files are part of a test suite, they are typically trying to illustrate some element of the program being tested, in this case a file that was incorrectly xz-encoded. Binary files like these weren't typed by hand, they will always ultimately come from something plaintext source, modulo whatever "real world" data came in, like randomly generated numbers, audio or visual data, etc.

Here's an example! My own SQLAlchemy repository has a few binary files in it! https://github.com/sqlalchemy/sqlalchemy/blob/main/test/bina... oh noes. Why are those files there? well in this case I just wanted to test that I can send large binary BLOBs into the database driver and I was lazy. This is actually pretty dumb, the two binary files here add 35K of useless crap to the source, and I could just as easily generate this binary data on the fly using a two liner that spits out random bytes. Anyone could see that two liner and know that it isn't embedding a malicious payload.

If I wanted to generate a poorly formed .xz file, I'd illustrate source code that generates random data, runs it through .xz, then applies "corruption" to it, like zeroing out the high bit of every byte. The process by which this occurs would be all reviewable in source code.

Where I might be totally off here is if you're an image processing library and you want to test filters on an image, and you have the "before" and "after" images, or something similar for audio information, or other scientifically-generated real world datapoints that have some known meaning. That might be difficult to generate programmatically, and I guess even if said data were valid, payloads could be applied steganographically. So I don't know! But just like nobody would ever accept a PR that has a "curl https://some_rando_url/myfile.zip" inside of it, we should not accept PRs that have non-cleartext binary data in them, or package them, without really vetting the contents of those binary files. The simple presence of a binary file in a PR can certainly be highlighted, github could put a huge red banner BINARY FILES IN THIS PR.

Downstream packagers for distros like Debian, Redhat etc. would ideally be similarly skeptical of new binary files that appear in the source downloads, and tooling can be applied to highlight the appearance of such files. Packagers would be on the hook to confirm the source of these binary files, or ensure they are deleted (even disabling tests if necessary) before the build process is performed.

A lot of them malformed (or output is slightly different than standard output), because they need to ensure they can work even with files generated by other programs. Bugs like ' I tried to load this file and it failed, but works in XYZ' are extremly common.

These formats are often very complex and trying things like'zeroing out a high bit' doesn't cut it. Youvwould end up with binary code encoded in source.

Edit: one of simple improvements github/other forges could do is show content of archives in a diff. The payload was hidden in a archive test file and it would be displayed in a diff instead of "binary file change, no idea what is in it"

That works if the archives are valid as checked in, but not if they’re corrupted in a predictable way such that they can trivially be “un-corrupted” as needed, perhaps by something as simple as tr.

Even if that’s not exactly what happened here, I think it’s pretty obvious how eminently doable that is, given the sophistication of so many aspects of this attack.

For instance, in Go it’s quite common to generate files and check them in, for eg ORMs. If I run `rm -r ./gen` and then `go generate`, git will report a clean working dir if all is dandy. It’s trivial to add a CI hook for this to detect tampering. Similarly, you could check in the code that generates garbage with a seed, and thus have it be reproducible. You still need to audit the generators, but that’s acceptable.

I don't think treating those binary files in the repo as red flags is in any way useful.

Packagers like deb and rpm (I work for Red Hat and have done some rpm packaging) should modify their build processes so that while they may run test suites ahead of time which use binary files, the post testing build phase should start from zero with all binary files fully removed from an untouched source download. There can be steps that attempt to build from a tar distro vs a GitHub source tree and compare. There's lots of ways a lot more caution can be provided around binary files, and I'm talking about downstream packagers for which there are a lot of resources to work on this (at Red Hat we're paid for this kind of work).

I have only a cursory experience with rpm builds, but with the normal debhelper process that should be quite easy: just switch the order of the dh_install and dh_auto_test targets, and then make sure the debian/ directory is read-only before running the tests.

Others have pointed it out thos is a normal procedure. One would think these tests should result in a binary hash and that hash gets compared with the production build.

Ie, the build for production doesn't need to pass the tests, it just needs the hash of the files that passed.

EDIT:

Also, Slackware -current neither doesn't link sshd against xz, nor uses systemd.

IIRC only the "binary" files where added secretly, right? But the build script was there for people to inspect? If so, I have to say, it's not that obfuscated, to someone who actually knew .m4, I guess. At least the grep line should have raised the question of why. I think, part of the problem is normalization of arcane, cryptic scripts in the first place, where people sign off on things they don't fully understand in the moment, since - c'mon - "knowledge" of these old gibberish scripting languages only lives transiently between Google searches and your working memory.

Without looking it up, can you tell me what this does in bash: `echo $(. /tmp/file)` ?

I think, I've seen at least one "xz backdoor detection script" by "someone trusted" in one of the xz threads here, which was at least as cryptic as the .m4 script, containing several `eval`s. I mean, you could probably throw your head onto the keyboard and there is a good chance it's valid bash or regex, or at least common bash can be indiscernible from random gibberish until you manually disassemble it, feeling smuck and dopaminergic. The condensed arcane wizardry around Linux (bash, autotools, CMake, ...) and C (macros, single letter variable culture, ...) is really fun in a way, but it's such a huge vulnerability in itself, before we even talk memory safety.

Yes, but it is important to note that these malicious m4 scripts were only present in the tar file. They were not checked into the git repo, which is why distros that actually built from git were not affected.

Totally agree with the problem of cryptic scripts in the build process, but unfortunately, if you maintain a project that needs to support a ton of different platforms, you don't have that much choice in your build tools. Pretty much everyone agrees that the 'autoconf soup' and its tooling (essentially m4, perl, shell, make) are all horrible from a readability perspective, and the amount of people who know these tools and can review changes is getting smaller, but switching to a more modern build system often times means dropping support for some platforms.

Looks like I got it backwards then. I thought, the test-files where the sneaky addition. Guess nobody cared for them...

> if you maintain a project that needs to support a ton of different platforms, you don't have that much choice in your build tools

Yeah, but, if possible, we could start porting those things into better frameworks instead of adding new features to this problematic Linux legacy code base. And maybe we could also retro-fix some of it with a better meta-layer, which generates the problematic code verbosely and standardized. If it can be done for JS a thousand times, it can be done for the *nix ecosystem once.

Lastly, part of it is cultural, too. Some people seem to get a kick out of reduced, arcane code, instead of expressive "prose". See, my example above... why the fuck is dot a shortcut for `source`?! Btw. I stumbled into this in Docker documentation[1]:

    echo \
      "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
      https://download.docker.com/linux/debian \
      $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
      sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

    $(. /tmp/os-release && echo "$VERSION_CODENAME") |  sudo tee ...

    sudo backdoor
    VERSION_CODENAME=bookworm

Normalizing shit like this is just bad practice.

[1] https://docs.docker.com/engine/install/debian/

1. It might not be there in the place where you're looking. It exists in the m4 in the release tarballs, not in the git repo.

2. It's highly obfuscated.

This: 'gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/`' seem to match the '####Hello####', but, as far as I can see, that's supposed to be the already converted script?! I presumed the binary files not to contain human readable strings, maybe that's the whole confusion.

From what I've read, the payload isn't stored in the archive, but rather the test file itself is a sandwich of xz data and payload: There are 1024 bytes of xz archive, N bytes of payload, another 1024 of xz, etc.

Do you have a (safe web view) version of those files? I would like to see what they look like to a casual observer. Judging by the 'tr' assembly command I would expect the bad-3-corrupt_ligma2.xz to be somewhat recognizable as script.

    commit 74b138d2a6529f2c07729d7c77b1725a8e8b16f1
    Author: Jia Tan 
    Date:   Sat Mar 9 10:18:29 2024 +0800
    
        Tests: Update two test files.
        
        The original files were generated with random local to my machine.
        To better reproduce these files in the future, a constant seed was used
    to recreate these files.



    diff --git a/tests/files/bad-3-corrupt_lzma2.xz b/tests/files/bad-3-corrupt_lzma2.xz
    index 926f95b0..f9ec69a2 100644
    Binary files a/tests/files/bad-3-corrupt_lzma2.xz and b/tests/files/bad-3-corrupt_lzma2.xz differ
    diff --git a/tests/files/good-large_compressed.lzma b/tests/files/good-large_compressed.lzma
    index 8450fea8..878991f3 100644
    Binary files a/tests/files/good-large_compressed.lzma and b/tests/files/good-large_compressed.lzma differ

If you looked at strings contained within the bad file, you might notice that this was not random:

    7zXZ
    ####Hello####
    7zXZ
    w,( 
    7zXZ
    ####World####

well lets all agree that now, if we see commits affecting / adding binary data with "this was generated locally with XYZ", that now we will bat an eye at it.

Speaking of obfuscation; I'm not a programmer but I did some research in Windows malware RE and what stuck with me is that every code that is obfuscated or every code that is unused is automatically suspicious. There is no purpose for obfuscated code in the open source non-profit software project and there is no purpose for extra code that is unused. Extra/redundant code is most likely junk code meant to confuse the reverse engineer when s/he is debugging the binary.

[1] https://lwn.net/Articles/846272/ [2] https://xkcd.com/2347/

This kind of goes against the whole "free" thing.

Idk if this specific individual that backdoored XZ had a track record of contributing to other open source projects(in a good will) or if s/he just out of the blue starting contributing to this project. I read somewhere that somebody else recommended him or vouched for him. Somebody needs to fill me in with the details.