For context, I've really only worked on "fun" side projects, namely emulators and game remakes, where I've explicitly avoided any mention of donations or similar. Both as it's intended to be a distraction from my job, not become part of it. And generally avoid as many issues as possible around said money distribution within the project, and possible copyright problems that such things often skirt.

And those sort of projects tend to be extremely limited in progress not by total interest, but skilled interest with the ability to contribute. I wish as many as 1% of the regular discussion participants in the community could contribute to that level.

There's a natural belief that unless someone is egregiously out of line, all discussion around a project from users is allowed, even encouraged.

But many people just can't seem to help themselves making "suggestions" (or, another reading, "demands"), and that can have a significant impact on motivation of volunteers. I believe the vast majority are from based in good intentions, wanting a "better" result, but arguably are mostly self-defeating in that I find the discussion around much of these things to be draining and demotivating. I want to have "fun" coding game remakes, not defusing drama in discord threads.

But it's expected to have a "community" for all such projects, and not exclude non-direct contributors. Even writing it down here I'm worried I sound like I'm trying to curate a closed community of ego-boosting yes men.

But I sometimes wonder if allowing this sort of open community is better in the long run.

Engage with people that have earned it in your eyes, whether by contributing to your project via code, assets, bug triage, writing a good and effortful bug report, whatever. Just ignore what the larger internet has to say about you and your creations.

Most people want to be helpful to others. This works fine when it is with a limited no of in real life interactions with mostly reasonable people. It can get wrecked by one or two unreasonable people, but the more fundamental problem is that it does not scale well, and the internet takes it to a far greater scale.

It is one reason so many of us essentially work for free for FB and the like: we do things to help others (advice and information), and the side effect is creating content and data for FB.

I don't think it has anything to do with the internet (whose scale pose many problems but not this one): anyone that founded a product/start up can tell you that people will give you “advices” and “suggestions” all the time when you talk about your project, and “don't listen to random suggestions from people you've just met” is one of the first advice you're being given when you're talking to other founders.

It doesn't even need to scale above a handful of people to be a nuisance that harm your project if you listen to them.

* lack information

* have a conflicting vision

* are being dishonest

I'm not a founder but I imagine that being integral to your own vision is the way to go. Attempting to cater to the conflicting needs of a diverse set of users isn't feasible for a startup.

But anyway this isn't valid only for founders but also to life, friends and family.

And yes they will go cry on reddit or HN or their own blogs and call you names. Ignore it. Be happy in the knowledge of the thousands or millions of people whose lives you have improved by your labor. Most of them wont bother to say thank you, but a few will, rememebr those. And be happy in the knowledge that the worlds is a better place because you exist in it, in however small a way.

The key is to realize that human psychology is by default not well equipped to deal with the internet. We tend to react much more strongly to criticism than praise. Recognize that in yourself, and gradually change it.

The distribution game is kind of stacked for the most likely to burn out to be the most likely to be important in the stack and in desperate need of help.

You're not wrong, and that's the first mistake: the individual running a FOSS project wants to see their project succeed and willingly bends over backwards to make it so.

The more you care about popularity, or even acceptance, the more vulnerable you are.

Especially considering the responsiveness here is responsiveness to the peanut gallery.

It would probably be good for the maintainers too. If being responsive is seen as a way to have a bigger impact, people might be inclined to become more responsive and burn out.

If you're working on open source as a hobby (as I do), you should make this very clear to your users. You work on stuff you want to, at the pace you find enjoying. You don't take contributions you think will be hard to maintain. You make it clear you don't owe anyone anything. You also accept that, often, the best answer to an issue/contribution/disagreement is a fork.

> How do you stop people from being mad that you’re ignoring them? (IME these types of people are likely to take things personally and start harassing you or complaining loudly in other forums…)

You don't (and can't) stop people from being unreasonable, mad, angry, or upset. That is their own choice. What you describe is bullying, plain and simple. Being nice to them because they might escalate later is counter-productive to your own mental health and gives them agency over your own time and energy. With bullies, the only way to win is to not play the game.

Don't reward incompetence with a response -- are they paying attention, have they done their homework? (paraphrasing Maria Popova)

https://serenityos.org/ apparently only makes source code available. There are no binary images of the OS to install

I think Andreas said this functions like a little test -- if you're not willing to build it from source, then you might not be a good contributor. Or you might not be seriously interested in the project's goals.

---

Likewise, my shell project provides source tarballs only, right now - https://www.oilshell.org/release/0.21.0/

It is packaged in a number of places, which I appreciate. That means some other people are willing to do some work.

And they provide good feedback.

I would like it to be more widely available, but yeah I definitely see that you need to "gate" peanut gallery feedback a bit, because it takes up a lot of time.

Of course, it's a tricky balance, because you also want feedback from casual users, to make the project better.

There's this weird trend that's been happening for some time now that tries to make non-fully-open groups look wrong, but honestly, has anything ever actually been done by such open groups? As far as I can tell, "closed community" is a necessary (but not sufficient) condition for any kind of quality creative output (this includes individual projects as single-person closed group). Having a core of creators surrounded by distinct group of fans and secondary contributors is a natural organization that forms spontaneously.

I don't think insults are the solution, but giving a clear no, is a skill many people struggle with. And if some cannot cope with that, blocking individuals also works.

I believe this masochistic behaviour stems from a combination of both general optimism and dedication to a particular cause. The more that developers are passionate about what they are doing, the more they are willing to engage with the peanut gallery.

The author there does the absolute minimum of interaction.

Issues not following exactly the template or lack a repro gets closed immediately without any comment, actual bugs also see very limited comment or openness to discussion.

It felt a bit weird at first, but I do get the author and the library is succesful, maybe exactly because of this approach.

It made me reconsider some of the effort I spend in my open source projects with useless issues / comments.

I'm not saying the community shouldn't be there, it's just that the barrier to entry is so low, that people can "contribute" with very little effort, that's become a significant problem as Internet usage (and acceptance of shit-posting trolls as just being a cost of anything online), so the signal/noise ratio is awful and we end up in an easy slide to controversy and spite and the good actors wanting to walk away.

I think this is why I don't do social media any more. I think it's why I'm not on the mailing lists I once was, or have interest in discord or a great many open community sites. It's just exhausting. HN, slashdot and some group chats in Signal with people I have known for many years is the closest I get to anything like that any more.

I've thought about doing some blogging without comments and a means to contact me directly another way privately. I've thought about making contributions to other projects with my only conversations being about my own commits, but that doesn't work for leading my own projects if I want to be "nice".

I think leading a project for me might involve _not_ letting others contribute to my version without clearing some arbitrary bar. No mailing list, no discord, no developer community. That doesn't feel optimal on some metrics, but it might be optimal for the metrics I care about.

I don’t really know how I’d classify the differences specifically…

Needless to say, I did not assist him in any way and closed the issue as wontfix.

I've also had people demand that I expand the narrow scope of my project to cover a whole class of devices with completely different APIs, instead of the sole focus on the series from the same manufacturer of devices I own and use. Closed as wontfix as well with a clear statement that I will not be expanding the scope, but am more than happy to link to their project covering it.

But we all should realize that any interaction also comes with a cost. If you report a problem and don't manage to describe it well it doesn't help the maintainers one tiny bit, in fact it might just create work and a bad feeling.

So my approach is to "just" write stellar bug reports including ways of bisecting where the problem might come from.

This both makes a bar that the bug creator has to pass, filtering some of the drive by time wasters, and gives a reward to the developers for fixing it.

I'm not saying everyone need to be Linus Torvalds circa 2012, but I do think more people need to be a bit less precious and sensitive, especially when receiving direct communication about their abilities.

You don't need ego-boosting yes men, you just need to work with more experienced folks, and that's okay. But the problem is that there is an army of people online who will take offense to that, and I don't know what the solution is. Best of luck.

It is necessary to establish clear boundaries of what can and can't be provided by the maintainers. If not done at an earlier stage of the project, the support burden becomes too much to bear at which point the maintainer transfers ownership, and the project suffers from catastrophic consequences such as the xz backdoor we're talking about here, or other cases where the project mostly stalls and serves as an ego-boosting platform for the new maintainer, as was the case with PhantomJS[6] before it was shut down.

This can also happen in your life, where a "friend" sees that you possess a certain skill, and then gradually tries to push an inordinate amount of their personal work related to this field onto you.

Personally, I think it's best to use an approach with extremely clear communication as to what the maintainer can and cannot provide. This can be seen, for example, in yt-dlp[1], where the consumer is clearly informed upfront that not providing detailed information as requested will lead them to block said consumer; or sqlite where their position regarding contributed patches[2] and support[3] is similarly made clear.

Having a shouty BDFL like Torvalds can also help improve code quality[4] and questionable contributions[5], though it is better that the shouty BDFL makes statements that are professional and do not show as much aggression; so for example, "Mauro, shut the fuck up"[7] would become "Mauro, your response is completely unbecoming for a Linux kernel maintainer, and is not in line with the promise of not breaking userspace."

[1] https://github.com/yt-dlp/yt-dlp/issues/new?assignees=&label...

[2] https://www.sqlite.org/copyright.html

[3] https://www.sqlite.org/support.html

[4] https://www.theregister.com/2024/01/29/linux_6_8_rc2/

[5] https://cse.umn.edu/cs/linux-incident

[6] https://github.com/ariya/phantomjs/issues/14541

[7] https://lkml.org/lkml/2012/12/23/75

This is just PC corporate speak. Linus is an actual human being who says what he really means.

There is also no telling if the person is interacting in good faith and just doesn't know, in which case the aggression is a bit rude.

You can tell people to fuck off without saying that explicitly, and it also allows you to save face in case the situation isn't what you expected it to be.

[1] https://arstechnica.com/gadgets/2018/09/linus-torvalds-apolo...

I mean sure, but how do you intend to stop people? You can't stop them from setting up a forum somewhere.

This is why github discussions is useful though. A feature request in a ticket is something that needs to be handled. A feature request in a discussion is just some users talking about stuff they think would be cool.

I’m not much of a gamer at all. But the once in a blue moon where I dip my toes in, it’s not that long before I’m met with a forum post or reddit comment making some technical assertion that sounds incredibly presumptuous if not downright incredibly unlikely.

They haven't poked around with RenderDoc, they don't have the debug symbols to profile the CPU code, they have zero idea what kind of work the software needs to do, they simply know the lazy devs have failed to "optimize" their game.

So if your project's audience is gamers, of course it's going to attract more people which also means lower quality community almost automatically.

WHO FUCKING GRANDSTANDS ENDLESSLY ABOUT CYBERTHREATS AND THE NEED FOR BETTER SECURITY?

WHO FUCKING SITS ON BILLIONS IF NOT TRILLIONS OF DOLLARS IN FUNDING?

That would be the FUCKING GOVERNMENTS OF THE FIRST WORLD. Whose modern economies increasingly rest on open source to a degree they possibly don't understand.

Yes I understand part of this comes from funding from another part of these governments. Oh well, cat and mouse. Once we wrung our hands over these governments having power over these. Well, these days we are facing far more totalitarian state threats and totalitarian wannabes in the private sector.

Government save me, as the less totalitarian option!

Linux and BSD the operating systems should UNQUESTIONABLY be funded to the degree of multiple billions per year by, I'll just put it, "NATO". Here's the true value of open source in this model: it is the perfect public record, unlike untold other aspects of government output.

You must produce public vetted code in Linux/BSD.

This person should not have been "a person". This should have been 5 people, funded likely at least 10-50k/year for their roles.

The described "hit" by the intel services of course is frighteningly easy. But what is most horrid is that this poor person is being subject to very common abuse that comes from non-state-actor manipulation. The article says it:

"this is where our software comes from" -- this poor dude that is trying to help and being abused by state actors on one hand, and ridden thanklessly and for no monetary or fame benefit by massively deep pocketed corporations, governments, and billionaires.

For all of Linus's famous swearing abilities, he hasn't used it to address this publicly. He should be dressing down all corporations and governments at this point that need Linux. There isn't a "going back" from Linux at this point to some closed source option.

Linus and Linux have massive sway here. They could threaten to stop work on the kernels.

I guess fundamentally, this makes me angry because this is basically bullying the nerd in high school to get his homework/answers. The nerds need to realize their power here.

Other vulnerabilities snuck through the commit chain? I get that, usual spy stuff. They actually kind of messed up here. Usually it is within the "cordiality" of things, just quietly ride the overworked unpaid nerd to benefit. Here they inadvertently exposed the real truth of everything: the hidden contempt we treat these people with as a society.

  > You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?

Worthwhile noting: It happened in the open, archived for the world to see at any time! The attacker needed to be extra careful not to raise suspicion, both acutely and for accumulated evidence in history. Of course, easy to say "Hindsight is 20/20", but we can probably agree in actual hindsight, we easily see a lot of suspicious acts around the issue. An individual incident may be chance, but as a whole things become clear more easily.

So, I think there is a teachable moment here: When something seems just a tiny bit fishy, do a background check, investigate. You may not catch all something, but you probably won't miss everything.

Now, looking at the level of sophistication and risk sneaking people and code into open infrastructure, imagine chances of getting caught infiltrating a large company writing closed source binary blobs for drivers, firmware, ...

Sometimes it's even true, some projects really die and stop actually addressing real issues.

But that's just inherent downside of the "bazaar" model. I don't think how we can "treat maintainers better" without going full corporate/without going full "cathedral".

But important software needs to be identified and proportionally more scrutinized by multiple independent parties, that's the lesson. Identification is the hard part. You can't easily determine that half of the world relies on this particular piece of software, or that it enables access to desirable targets.

Yes, this almost succeeded... but can you imagine how many scenarios where someone such as Andres Freund would have found irregularities, but then.. what? Just had to report it to some webpage's contact page? Without being able to even dig further?

Would he have known what was happening, or would it have just ended up as an oddity, with no source code, and with the binary purposefully obscured, and so on?

Or... even worse, it's reported directly to the 2 guy team, and the guy who put the back door in... takes the bug report?!

From where I sit the sleeper/mole problem exists in companies, and can be far harder to detect.

Yes, you get more eyes and people like Andres Freund.

However, if this had been a mole in a company, he wouldn't be able to hide behind a possibly anonymous fake persona and (likely) be immune from any consequences/fallout from this attack. It would be harder to gain entry in the first place, he would have needed a real identity. Background checks may not be a big hurdle, but they're at least something more than signing up for a GitHub account. He also wouldn't have had his posse of anonymous sock puppet accounts to add pressure to the original maintainer.

I also think that just being able to ramp up on liblzma and its dependencies to undertake this effort in the first place is a huge head start vs. trying to execute the same attack on a closed source corporate product.

At the same time, there are probably lower hanging fruits to attack if you really do get a mole inside a company, in addition to there being fewer eyes/opportunities for the attacks to be discovered as you pointed out.

I honestly don't know how all of these factors add up. I expect the argument opposite to yours to be raised (again) in the wake of this incident. I'd personally be hesitant to raise this argument (not that it matters here on HN).

You can save a lot of time reading John LeCarre novels and take a short summary like this one of Adam Curtis [0].

If you watched the recent Oppenheimer film you'll know the name Klaus Fuchs. But what about Guy Burgess, Kim Philby and Anthony Blunt? If MI5, MI6 and GCHQ are. almost by tradition stacked to the rafters with defectors and spies, enclaves of enemies within, and enemies within enclaves of enemies... how does anyone expect a commercial company motivated by money and with such a weak perimeter as a "job market", to do better?

Trust does not have an organisational solution.

[0] https://www.bbc.co.uk/blogs/adamcurtis/entries/3662a707-0af9...

Lots of companies hire remote workers sight unseen, and not all require proof of identity, and where they do, that proof can possibly be easily faked by someone willing to break the law.

Larger Open Source projects - e.g. Debian Developers, also have real-identity verification through chain of trust.

> Background checks may not be a big hurdle, but they're at least something more than > signing up for a GitHub account.

GitHub doesn't allow more than one free account per person. Companies might not look for employees sharing an IP address as much as GitHub does.

> He also wouldn't have had his posse of anonymous sock > puppet accounts to add pressure to the original maintainer.

Depending on the software, it might not be that hard to be a customer (or even pretend to be an employee of a known customer, if they aren't validating incoming requests claiming to be from customers enough) and add pressure for features that will provide cover for a backdoor, or even for a product that isn't a commercial success to be handed over to an external developer.

You should check the thread posted yesterday, the Lastpass guy who raised a PR for a go binding for xz but was otherwise unrelated to this fiasco already faced a bit of questioning regarding their motivations from their employer based on a user reporting them from a contact form.

Moreover, many companies already have information from background checks, and in certain countries, they also have the tax identification number of the employee which can pretty much identify who put in the backdoor.

If this happened on a proprietary software project, …I have verified proof of identity documents for everyone on my team. These can of course be faked or fraudulently obtained, but it heightens the barrier to entry and leaves more of a paper trail. That’s not something that we have in this scenario. There’s way more give and take between open and closed source than open source purists make out. They just like the idea of being able to see source code.

Did it? At least in Debian, where it was found in the first place, it only existed in sid/unstable and I think perhaps in testing. It never made it into stable which is the only thing you should be running on a production system.

I think in Fedora it was also only in a testing or RC of the next stable version but I'm not sure. Also unsure about others. I have seen several distros putting out advisories (Gentoo, Arch, OpenSuse and more) but I didn't go into the specifics.

The end goal seems to have been to get it into the LTS releases of rhel and Ubuntu. Since this was a valuable it's probably something kept around for when other methods fail. I doubt it would be used before the really valuable systems are compromised (rhel and Ubuntu in cloud and governments)

There's nothing wrong with Debian stable having years-old packages either. In any case:

> The current "stable" distribution of Debian is version 12, codenamed bookworm. It was initially released as version 12.0 on June 10th, 2023 and its latest update, version 12.5, was released on February 10th, 2024.

https://www.debian.org/releases/

That's just over 1 year. Sure, packages themselves will be older than that since they take some time soaking frozen in testing before they become part of stable but that's generally a good thing.

In any case, if there's pressing need for some newer packages, it's always possible to use apt pinning to pull those on top of a stable base.

> > It made it to production, for a long time

I never said it didn't make it to production. I was asking if that's the case, and giving one data point.

It apparently didn't make it into Ubuntu LTS either, so there's another one.

If you have information that proves or disproves either point, please present it.

That is pretty hard to defend against. I wonder if they were planning this all along, or if they wound up with a greedy plan later, or what.

"I don't feel like it, if it's important to you then feel free to fork".

That's really all that's needed. "I don't feel like it" is all the justification you need.

Some guy just made a compression tool, because some people like doing that kind of thing, or because it was useful for him. He didn't ask to be made "critical infrastructure" or to be responsible for the security of sshd or to have some business depend on it, or anything like that. No one even asked him.

That's really all that's needed.

that's much harder than it sounds.

having someone fork your project can give you the feeling of loosing control over the project as potentially all your users might go with the fork. that fear is often strong enough to push yourself to do things that will avoid a fork.

it's a desire for harmony and a fear of conflict

That's the issue right there. Why would you care?

Clearly, the maintainer is invested in having a "community". Why? They must expect something positive coming out of it, so they are invested in having that community, which then means they have something to lose if that community moves somewhere else and they are left behind.

That's what enables this social exploit. A takeaway can be not to get so invested in having a community. These are users of your software, and their "utility" is in helping find bugs, perhaps suggest improvements or even provide patches, making the thing you as the original author made better. But that can clearly become a burden.

And that doesn't really change that things really are that simple, kind of. How do you stop smoking? By not lighting up any cigarettes. Of course it's not that simple, but ... it also kind of is.

this is an important observation.

a change of culture is really what is needed here, because it implies that not only maintainers change their behavior, but everyone involved, and the question is not so much what any individual can do for themselves, but how we can help others with that change and spread it

For the most part, the people who care already care (you, me, most others commenting here) are not really the problem.

So ultimately the only thing you can do as a maintainer is set your own boundaries for yourself. And the original message in that thread ("there is no activity, what are your plans?") is a fair question, although the follow-ups are not.

All of this is true for a lot of the internet by the way. The best thing of the internet is that anyone on the planet can talk to anyone else on the planet. When I was a kid I did some ham radio with scouting, and you could talk to people from places like Russia and the US. Wow!

The worst thing of the internet is that anyone on the planet can talk to anyone else on the planet. You're constantly exposed to a seemingly never-ending stream of assholes and idiots, and a single asshole can ruin the day of dozens, hundreds, or even thousands of people every day. Maybe this is just 1% of people, but 1% of 8 billion is 80 million.

being nice to each other is just one of the many things that we as humans need to work on. and we are working on it, and hopefully some day we will be able to achieve it.

> Redict is a Finished Product

> Drew is a controversial person

(he's been rude/mean in the past)

xz should be pretty much finished as well, major overhauls like the "ifunc" feature to inject alternate function implementations are not really justified. Beware of busybodies and this whole "the community demands vibrant evolution of the project" thing. xz does its job already!

And being rude/mean is not a plus, it's a minus, but it does seem to correlate strongly with leading a high-quality project. Even if it's not the best way, this person has the guts to say no in unequivocal terms. You might just have to accept the downside of a rude/mean maintainer as a common unwanted side-effect of an effective maintainer. (Linus Torvalds also comes to mind of course.)

If someone had slapped me down for being an idiot I'd have most definitely found something else to spend my free time on.

Today, I'm not super great at coding but can hunt down segfaults like a truffle pig and submit bug reports (with code) that demonstrates the exact issue if it's something I can't figure out on my own.

And sure, I understand why people feel a responsibility. And it's fine to take this responsibility too. I'm just saying: there is no need to.

The entire point of this Free Software/Open Source is to give people the freedom to do whatever you want with some piece of software, without having to be beholden to the original author. That's pretty much the entire point.

Anyone in the world can be a maintainer for xz. By forking it and applying useful patches.

It was definitely another world. I do agree that maintainership issues were already there, but I think they were smaller in number. Between the explosion of projects and the explosion of users, they are now on a different scale.

The others I don't know, but MySpace was not a particularly big operation. Before WordPress, most sites started as PHP at some point were expected to migrate to something else, because maintainability of PHP3/4 projects was a big challenge (hence the "shame"). Facebook was the first company that simply refused to do that, and focused on improving PHP instead.

What PHP lacked and Ruby had was rails like framework , the team from 37signals singlehandedly shifted the momentum out of PHP with their work on rails.

It was almost 20 years ago, and no, I generally agree with the person you're responding to that OSS has changed significantly in these regards since the early-to-middle 2000s.

I'm not even disagreeing with the rest of your take, just poking at this idea that time hasn't passed and changed things. Some days I look around our industry and feel like it's nowhere near the one I was working in before.

I don't think the maintainer is at fault to any degree here. Sure, this could have been avoided if the maintainer refused to be pressured and kept sitting on the project and letting it die, but it's not his fault that he didn't do that, and I wouldn't want that to be the default for maintainers either.

All the pro-social benefits with a side-dish of the nuclear option. That’s coherent I have to admit.

In that case one can limit one’s interactions to other invested parties, i.e. contributors. Granted then you are still interacting with the attacker but you’re spared from the peanut gallery.

In real life volunteering you don’t get random drive-by input from outsiders. The input (and whatever peer pressure) is only from other invested parties.

> I don't think the maintainer is at fault to any degree here.

I’m having a hard time understanding moral arguments. “Fault” and “blame”. Everyone is condemning the peanut gallery for complaining about passing on the maintainer stick to someone else. Yes, including people who say that he could have just “not got peer pressured”. To be clear it’s not about the maintainer having “fault” or the peanut gallery/the attacker being wrong. Both can have “fault” in different ways. Like, clearly the attacker is the one who did something bad. Now there’s only a question of what other people could have done differently.

> Sure, this could have been avoided if the maintainer refused to be pressured and kept sitting on the project and letting it die, but it's not his fault that he didn't do that, and I wouldn't want that to be the default for maintainers either.

The maintainer could have done something different but he didn’t and that’s not his fault. It seems that we all agree that he had a live option. You just want to not associate it with “fault”.

In another comment[1] I asked what moral obligation a maintainer has to herself. Only to herself.[2] Focusing on that angle seems more fruitful than talking about “fault” in the abstract since that just leads to back and forths about whether people should protect their wallets better or whether or not people should just stop pickpocketing people.

The goal of this subthread seems to be about how maintainers might protect themselves (for their own sake) from this kind of thing. Laying out the options that are in their hands (and not just how the world around them should become better) seems pertinent to the issue.

[1] https://news.ycombinator.com/item?id=39882721

[2] Like asking about whether someone has a moral obligation to eat healthy. It’s not about other people.

Peer pressure among adults is far more widespread and powerful than the limited peer pressure among teenagers.

I hate this trend of shouting "victim blaming!" once someone tries to explain things or analyse anything. Not everything needs to be a value judgement. "X happened, and Y could have prevented it" is not a judgement.

If I were in the maintainer's shoes, and was feeling ambivalent about handing over maintenance to a fairly unknown person, this kind of social attack would definitely push me over the edge, exactly as it was planned to.

But you can insert "I'm not trying to blame anyone, but here are some suggestions to modify cultural norms so these things are less likely to happen in the future" if you want. Or you can just assume good faith and take that as implied unless demonstrated otherwise.

Also: as far as I'm concerned there is no "peer pressure" here because these people aren't "peers". They're just some random people who, as near as I can tell, have done fuck all. There is not even an attempt to help out. Not even the question on how to help out. These people are supposed to be the maintainer's peers? Yeah nah. They're just shouty entitled internet nobodies that have not even attempted to contribute anything constructive or signal any willingness to do so (not even "I have been using the patch in production for half a year without problems", which would actually be a small but useful way to help out).

When it comes to these types of things you need to accept that you can't change every person in the world; you can only change yourself. If you cycle a lot you better learn to anticipate assholes doing asshole things. Is that fair? No. But it beats being run over and getting hospitalized, or worse.

That’s what I find incoherent about it.

1. The consumers are little ants that have no real power over anybody, certainly not to help the project itself

2. On the other hand they are powerful enough to peer pressure the only person who has the power to drive the project forward

I think their paraphrasing was a fair representation. You want to have it both ways. You said some incendiary victim-blaming thing, but now you're backpedaling with a "no no, you misunderstood".

The reason I ask is maybe the people who do are (in general) self-selected from the subgroup of people who don't just say "sod off" when someone is rude or inconsiderate.

If that was their stance in life, they would likely not put up with maintaining a popular open source project for very long.

"I feel a huge responsibility to please every user who reports a problem, shortcoming, or asks for a feature, which I need to address ASAP" is one attitude.

"I work on it whenever I feel like it, and if you don't like that then I don't care" is another.

Those are on the extreme end and for most people it's somewhere in-between.

It's very common for people doing volunteer work to lean too much towards the first. They have trouble saying "no" and bite off more they can chew, even without direct pressure. Learning when to say "no" and setting boundaries for yourself is absolutely a vital skill for any kind of volunteer work – without it sooner or later you will burn out.

When I was a scout leader burnout due to this was a major cause of attrition. We made it very clear and very explicit there was no pressure for anyone to do anything they didn't want to, and that there was no shame in not wanting to do something just because you didn't feel like it. But it still happened, because people still feel this pressure, even when it doesn't actually exist.

I think its far easier to make $3,000,000 as a hacker than a worker/entrepreneur.

Its way easier to find flaws/bugs than to do the entire Capitalism thing correctly.

Then I see that half of these major attacks required social engineering.... Maybe being a hacker is significantly easier. I only need to fool 1 person, there are a lot of people, and merit isnt exactly how everyone got to their position.

Anyway point being: People are amazed by hacking, they shouldn't be, its relatively easy if you are a mere 10 year programmer. Most of us pick relatively moral work, so the number of attacks are small. It is also why we really need to treat security on computers like its no stronger than your home's front door lock. There are too many attack vectors to be perfectly safe.

And if you do make that $3m as a hacker, now you have to spend your time worrying about whether someone will eventually catch you because of that one time you logged into a service without using enough VPNs.

+1

Between roughly 1999 and 2013 I was primarily a test engineer for networking switches/routers/telephony products. I found bugs for a living and wrote them up, and in the process I found plenty of security vulnerabilities and wrote them up. Security bugs aren't really that different from other bugs. Yet for some reason we lionize people who find security bugs.

Most security issues are simply quality issues. But by calling them security issues we shift the focus away from the software producer creating shit code to an attacker doing something bad.

The case with xz is a little different, because we have someone who intentionally added bad code and tried to hide it. But for unintentional bugs that rise to security vulnerabilities it's 99% a QA problem.

The argument around blame shifting is apt. The same case has been made for the usage of the term 'bug' (aka an externality). It's 2024, we don't have moths crawling into relays on our computers. We have implementation faults, invalid designs, unsound architecture, inaccurate documentation, ambiguous requirements, and a myriad of other ways to express how software may be defective.

Using those terms hurts, and may even invoke some level of concern from those outside of the engineering org - this is a great reason to embrace them.

It would be pretty poor QA who only tested happy paths . Any competent quality analyst will be expected to test for failure states, boundary conditions and so on .

Even if we say "no payment, no customer," it won't prevent determined attackers from paying significant amounts of laundered money in order to be treated as customers.

In fact, Occam’s razor says that the malicious code was injected by a compromised account and not a malicious actor who spent years steadily getting into position to attack?

The normal commentator who complains is probably not malicious, probably not aware of the pain they might cause, is probably just not even thinking of that angle.

Let's take on face value that it was the Chinese, and that China is communist. I mean, "Kumar" and "Tan"? Maybe it wasn't, but it doesn't matter for my purposes:

They took an overworked peon of the capitalist enemy that provides a ...

... do I even need to expound? well it's fun ...

... collectively and idealistically produced common operating system "for the people"

... that is exploited and neglected by the rich and powerful, to a degree that society, not just computers, overall society operates on this operating system, and the profits from that are hoovered up by the powerful.

The communists attacked the exploited proletariat to get to the enemy. And they are forcing the enemy capitalists to either pay the proletariat properly (they won't) or continue to be vulnerable to the growing communist power in the far east.

That's what this distills so well, within a political/ideological conflict that has now spanned 100 years: capitalism vs communism.

To wit, a proper functioning capitalist system to reward market value for produced value would properly pass compensation to this poor soul, and incentivize others to help him. Barring that, a proper functioning government would recognize the public good of this and provide support to the core infrastructure software to enable other private enterprise to produce tax revenue.

THOSE AREN'T HAPPENING, so the Communists can attack this with impunity and in perpetuity.

Here's the thing folks, we've been living in, as they used to say "uninteresting times". Post-WWII Pax Americana, even the Cold War was basically peace, has been cranking for 80 years now.

People, that is coming to an end:

- Russia / China are destabilizing demographically and simultaneously becoming militant and totalitarian

- Global Warming will ramp up the pressure on populations, food shortages, production

- The US will likely retreat to a more regional focus as production is onshored, regionalized (or at least centralized to our hemisphere)

There's going to be more state conflict, the Ukraine war is just the beginning. The world stakes are rising.

(I guess in the case of xz, Jia Tan has earned the cred before going rogue; the one-off “maintainer needs to be replaced” campaigners, however, haven’t.)

It's not an either/or proposition. I definitely think it was state sponsored, AND one method used was social engineering a burned out maintainer.

If there was a simple way of advertising the need for help to a community of experts that I/we could trust, I'd take that any day.

That's an excellent mind set when reviewing code, no matter security or not. But especially for security. How could this be wrong? What are the corner cases? How could anyboy break this? What do we need to test? That kind of scrutiny is crucial for keeping the quality of your code base high, no matter who posts the PR.

In any case, I agree technologist aren't thinking about it because the only way to extract massive amount of wealth is by centralization/monopoly. Hard problem though, the appropriate amount of redundancy is hard to know.

It is implicit in your statement that you know some "proper" way to think about trust. If it is the case, please do share with us your thoughts on trust.

So, not so much a judgement of the quality of the thought but rather of its existence in the first place.

Unfortunately the Open Source community has a long history of ties to anonymity culture, which is fundamentally untrustworthy.

Because that's an insanely hard problem that's outside our area of expertise.

Consider how much money governments spend on all the red tape they add to increase trust. If there was naturally perfect objective alignment and trust I bet any infrastructure project would cost about 10% of what it does.

Laws are highly regulated. The DMV policy manual is, like most government day to day policy, run through “managers” who report to a elected official (ie it can be very well written and considered or made up on the good after a bad newspaper headline)

Anyway I think I am saying there is likely to be a consolidation of FOSS - where we spend a fortune trying to bring it under some “approved and controlled” process.

We can call that trust.

I am not sure.

In my book, software quality is much more important than emotions.

On the other hand, if I were a non-chinese hacker trying to do evil, then using a chinese handler does make more sense (China is evil, blah blah blah)

1. Might not have the vocabulary to express themselves correctly

2. Not understand the nuance and connotations that their word choices imply

I've seen my colleagues get frustrated in meetings while simulatenously having to assume good intent.

Choosing names like the bad actor(s) did gives them an advantage in this scenario.

- First, we're not discussing the English level of the general Chinese population, but specifically of the group that's also good at programming. That conditional probability is much higher.

- Even if we did speak of the general Chinese population, that is a lot of people so something rare in that group is true for many people still!

- It's not like everyone in the US has great English either. (Though open source maintainers are again probably more likely to.)

That's the group I was discussing. Higher for sure, but still not high. (Remember we're talking about Chinese nationals, not Chinese ethnicity.)

> so something rare in that group is true for many people still

Yes but we're talking probabilities so the total population size is irrelevant. You're the one making the statistical mistake here.

> It's not like everyone in the US has great English either. (Though open source maintainers are again probably more likely to.)

Answered your own question there.

What would raise my hackles would be someone with a name that codes western who had grammatical idiosyncrasies common to ESL Asians.

A better assumption imo is they chose their identity first and foremost with the intent to garner trust first with Lasse Collin.

They also used a sock puppet with a seemingly German name (Hans Jansen). However "Hans" has not been a popular baby name in German speaking countries for many decades. You'd expect any "Hans" to be over 70 or 80 by now. Unless they are American, like Hans Niemann.

This could be a cultural oversight on the attacker's side, which points towards any culture that has a wide spread belief that typical German names are Hans or Fritz. From my experience, that is especially true for English speaking countries, mainly because of stereotypes displayed in Hollywood WWII movies. I wouldn't be surprised if cultural perception of Germany was different in China.

Edit: some clarifications

Edit: Another country that seems to have retained popularity for "Hans" for a longer time than Germany is the Netherlands. According to [2], its popularity seems to have gone down significantly as well, with a further sharp dropoff in the 90s.

There are plenty of other names to choose from if you want to create a fake personality. It's curious they chose one that sticks out among the age cohorts you would expect participating on Github.

[1] https://www.dst.dk/en/Statistik/emner/borgere/navne/navne-ti...

[2] https://www.babynamen.nl/naam/hans/

1) Why not?

2) You'd likely want a relatively common-sounding name, I suppose. Chinese names are relatively common-sounding.

Opposite is true. His activity is focused around midnight in UTC+0800, with significant activity at 2AM in that timezone.

Culpability also must be laid at RedHat's feet for sanctioning the practice of side loading libraries into such a critical service's address space. Their drive to cellularize systems management has overtaken their common sense.

The idea that they could not be bothered to answer the call of the sole maintainer of a library used in such a critical path in his time of need is, well, astonishing.

2) A potential explanation for "why now" is that systemd DID prevent these dependencies from loading automatically in a patch one month ago [0], and the patches to lzma enabling the backdoor merged a few days later, followed by (as we know) an immediate and somewhat heavy push to get distros to upgrade driven by sockpuppets. It could be a total coincidence, or it could be that the attacker jumped to pull the trigger before the window of vulnerability started closing on them

[0] https://github.com/systemd/systemd/pull/31550#issuecomment-1...

I think it's a bit cheap to blame systemd here, and systemd does not equate directly to Red Hat either.

Choosing a distro is nothing but choosing where you place your trust.

I can understand debian cutting corners here and there. But RH have little excuses with the money they make. Yet, even a superficial analysis, show them to be less trustworthy than the anime-avatars maintaining gentoo or arch.

It's not obvious what to do about it to me either. Which is why I'm concerned to talk about it.

That this attack was run by someone who had been participating in the project for possibly years before making the attack -- is not what i would have expected, and makes it even much harder to defend against. Before I was thinking it was about awareness of "don't just turn over the thing to someone new who just showed up, they might be an attacker." But not that easy in this case.

I suppose "try to get users to be less mean, by doing our part by being less mean individually" is arguably one piece of strenghtening defenses to this kind of attack, I guess, ok.

Yes, by denying cover traffic.

And then you need to debug something. What do?

You could also in some cases have a debug container you pull in on demand, say if the host OS is an appliance-style k8s runtime like TalosOS.

E.g. if you are running an immutable k8s runtime distro like TalosOS or Metropolis, you can choose to pull and mount a debug container with strace and gdb into the same process namespace of a problem pod, and when you are done all those debugging tools evaporate.

System mutability is why most bugs happen in the first place.

There should be no tools or services in prod except those required to run or monitor prod at runtime.

Systemd itself doesn’t even use lzma/xz… libsystemd does, and that’s a library meant to help other software integrate with systemd. It’s not really the same thing.

Help in maintainship how? Patches had already been made and were awaiting to be reviewed and merged. This was up to the maintainer to do and requestor couldn't help with it.

At the same time, this attempt nicely illustrated that the chain is only as strong as the weakest link since, as I understand it, no part of the backdoor was committed to the git repository in cleartext. Instead, the part of the backdoor that was at least somewhat identifiable was only included in the tarballs that would be downloaded and used by Debian/Fedora when building the packages for these distributions, thus giving a very nice trade-off between the chance of someone detecting what was going on and the potential impact of the backdoor.

Depends upon your perspective.

Hacker: "Oh it was hilarious, you should have been there! They donated $1M to the project after I hacked the code, so I took the $1M too."

Time is another factor. It takes time to maintain software, improve the codebase, add features, etc. Then there are the other tasks such as answering questions, reviewing PRs, triaging bugs and feature requests, etc.

So getting more contributors, people to assist with bugs and bug investigations, etc. is arguably more important. Especially projects developed by a single person, or a small number of people. That's the avenue that opened up this attack.

It is easy to get burned out implementing features that end up being more complex than expected, interacting with users that want different things from a project, and having a growing list of issues and PRs. That's the scenario that happened with xz, and is common with popular software that is maintained by a solo developer.

The other aspect to this is the direction the maintainer wants to take the project in. If another maintainer has a different direction in mind, that's going to cause tension.

Time and money are not actually 100% fungible, but there is a lot of truth to it, especially given enough money.

Maintainers are human. They need to eat, to sleep, to visit the doctor, to rest when they get sick, to participate in activities that reduce stress and foster human relationships. Money makes all of that much easier.

Besides, the maintainer in this case was already taking time off regularly, not to work on xz, but to get away entirely from any kind of programming work. Throwing money in his general direction probably wouldn't have helped with the burnout, unless you were offering to help him hire somebody.

Of course if the developers don't want to be paid, then that's that. But otherwise, there is a very heavy atmosphere in the open source community of excommunicating anyone who dares to ask for payment as heathens of the vilest order.

I fully agree that forcing payment or using dual licensing is unfortunately heavily frowned upon. But a voluntary Patreon/donation option is perfectly acceptable to the same anti-payment people.

But it's not a solution for all problems.

I am thinking a lot about this. One of the issues is scale and proof. I suspect that I am interested in introducing gated ability to comment / participate in a community

Say for example github introduces “gates”. The first might be add a test to the test suite that generates a haha of the version number and test output. Then adding that number to your profile means github trusts you have at least downloaded and run make test. It shows some level of commitment. (I suppose the zero level is logging in to github and commenting on some maintainers mental health)

I mean, the idea of commitment -> privilege isn’t bad. But it’s got nothing to offer in the story of xz.

Why? My guess is that this new culture is promoted to extract free labor from hobby developers. Take for example, Github. It has a bot to close stale issues after a given interval of inactivity. But what if the issue was valid and unresolved? What's the problem in leaving it open for someone interested to take it up? But what GH wants to promote is a culture where open issues are considered as bad performance on the maintainer's side. It indirectly prompts the maintainers to take open issues too seriously.

Well, also a lot of people get into FOSS for altruistic reasons. They want to pay it forward and make the world a better place.

For these people, applying good operations principles allows them to more efficiently do good -- just like it allows a commercial company to deliver more end-user value for lower cost.

Running an efficient operation in FOSS isn't necessarily bad. Sometimes it takes an obligation viewpoint to get there. It can be unhealthy, but it can also give us crazy levels of neat software, as we see proof of daily.

- every Fortune 500 company tracks exactly which FOSS code it includes in its ecosystem (usually code scanning and fingerprinting - can’t remember the usual Provider of such)

- this is essentially the software BOM that Biden signed a while back.

- this (made public) would give a real time map of the dependancies of all organisations - and linking that to things like the above thread (“cry for help”) would be an interesting place for “intervention” - anything from plain old cash to “here are three interns doing two years in gov.uk. They will help for the next 2 years

(It’s not a great idea that last one but we need to start somewhere- no way can government “pick winners” but also no way can society just sit back and hope. And commercial incentives break this horribly.

Essentially we are going to find at some point we treat some developers like lawyers - this stuff is our societies laws, rules, processes

Having temp workers come and go into all kinds of various open source projects.. does that help? :)

I am not claiming I have perfect solutions, just that we need to be more mindful of our supply chains. And when we are we start demanding things of those supply chains - and if we are sensible we support those suppliers

Why not play into the strengths of OSS instead of trying to replicate frictionous trust models from elsewhere?

Like, listen to ourselves. We are admitting here that open source software is not feasible to audit? Like did we just accept that an adversary can basically smuggle exploit payloads in plain sight?

Can we at least try to simplify and reduce the messiness of checked in generated code, irreproducible builds, and magical binary blobs, conditional compilation, “ifuncs”? Wtf? Granted I’m not a domain expert in this area, but I’m horrified by all the complexity for what I understand to be a compression library.

I would much rather play whack-a-mole with obscurity, which as a side effect is great for software quality overall.

God knows if I could. It that has to be the first place to start

They already are BDFL for themselves or one other person. The problem is a chronic deficiency of maintainers, which is at the root of this attack.

Your comment implies there is a wide pool of peoplecwho want to do that, along with mythical "someone".

Here is a little story:

There was an important job to be done and Everybody was sure that Somebody would do it.

Anybody could have done it, but Nobody did it.

Somebody got angry about that because it was Everybody’s job.

Everybody thought that Anybody could do it, but Nobody realized that Everybody wouldn’t do it.

It ended up that Everybody blamed Somebody when Nobody did what Anybody could have done.

- Charles R. Swindoll

I don’t know where you got that? I didn’t say there are a lot of people who can or would step up. It’s the opposite: competent, motivated people are rare… and why would they step in to maintain someone else’s project?

I was implicitly hinting at this by saying you need a Steve Jobs. We all know how rare such people are.