Published: 2021-12-23 22:17:43 -0800 PST
Last updated: 2022-04-24 21:06:24 -0700 PDT
I think I saw something about pkgsrc pop up randomly in my Mastodon feed a few months ago. Before that, I had never heard of it. And now…well now I’m currently writing these words in a version of Emacs that I built from source with pkgsrc.
Pkgsrc is NetBSD’s package manager, but thanks to NetBSD’s portability philosophy, pkgsrc is fully supported as a package manager on other Unix-like operating systems like Linux, Mac, and Solaris. It supports installing binary packages, but the reason I was interested in it is that pkgsrc supports building packages from source as well.
For some context on other similar projects, pkgsrc was originally forked from FreeBSD’s Ports collection decages ago, so for people familiar with the other BSD’s, this is a ports collection that works on other platforms. Even better, pkgsrc and NetBSD get the terminology right and correctly call these things packages instead of ports (and leave “port” as the term for cpu architecture support). Other similar-ish projects include MacPorts, Gentoo’s emerge, GNU Guix, and Ravenports. I probably missed some, but you get the point. There are multiple options out there if you’re interested. I’m not going to compare/contrast them because honestly the only other one I’ve ever used was MacPorts like 10 years ago when I didn’t know what I was doing.
Why did I use it?
I’ve always disliked needing to install a new package to get some
program to run. The main reason why is that the new package could pull
in other packages and they all end up polluting my system folders like
/bin and /lib with extra stuff. Half the time I delete the program
I was testing out anyway, but then I’m stuck with this extra stuff in
my base system. I never really even stopped to think that there might
be way to separate out the “base system” from the other crap I’ve
installed along the way.
But this is exactly what NetBSD does. NetBSD’s base system for /bin
and such is really only for the core NetBSD system code like the stuff
you can build from their source code repo. Anything else is supposed
to be installed somewhere else. By default, pkgsrc will install stuff
into /usr/pkg where it created its own subdirectory tree of
/usr/pkg/bin and /usr/pkg/lib and so on. This way that specfic
pkg directory is like a self-contained userland separate from the
base operating system. I’m actually using pkgsrc on Debian rather than
NetBSD itself, but it works the same way.
Even better, pkgsrc lets you use it in unprivileged mode and install
stuff locally in your user’s home directory like ~/pkg. This can
give your user their own separate custom userland from other
users. Depending on your use case this may not be what you want, but I
found it particularly cool that I could run almost a completely
separate system just for my user. Because I didn’t want to need
sudo/doas to install packages, I am actually using pkgsrc in
unprivileged mode with ~/pkg as my base directory.
So that was a long explanation to this answer. Why did I start using pkgsrc? Because I can install packages locally in my home directory, I can build those packages from source, and I know this system will work on any operating system I will ever run into (and I’ve used some weird ones considering how relatively young I am).
How it works
Pkgsrc isn’t really a package repository itself. It is simply a collection of Makefiles that define how to build each package as well as the dependencies needed for a package. Each package’s Makefile defines where to download the source code for the package, how to apply any necessary patches, and then how to build it. This actually answered a long-standing question for me: why do GitHub “releases” build a tarball/zip snapshot of the source code? Well, systems like pkgsrc will use that tarball to download the source code for a specific version without needing to clone a full repository (or even needing the source control software installed at all).
Pkgsrc reaches out to various websites like GitHub and Sourceforge (or wherever else a project’s source code is hosted) to download the specified version of a package to build it. This is highly decentralized because pkgsrc does not rely on a single repository server, instead relying on each project’s own servers. (As a random side note, this is similar to how the Go programming language handles dependencies). This could lead to the potential problem where a single project’s website is down so you can’t build their software. However, the Makefiles also specify alternative distfile mirrors for where to download the source code. New specific terminology for me here was a “distfile” which is that source code tarball snapshot for a specific version of the package.
Tinfoil Hat Time
One potentially “irrational” requirement of mine was that I wanted to be able to install software from my own network in case the internet ever goes down. Not just my internet, but THE internet. And I know that the internet as whole will probably never really go down, but I do think it may get to the point where the internet is fragmented into completely different silos because of government censorship.
In a more practical manner, I also wanted to be able to install software from my local network to save bandwidth when updating multiple computers. This is why I host some mirrors on my server.
One of those mirrors is the pkgsrc distfile mirror. I was able to use
pkgsrc with the MASTER_SITE_OVERRIDE option to tell it to use my own
distfile mirror. This allows me to build all of the supported pkgsrc
packages offline because the distfile mirror has already reached out
and downloaded all of the source code for every project (or rather,
I’m mirroring from a mirror that has already done that).
I realize this is a weird almost “prepper’ doomsday mentality, but I really wanted to be able to install software locally on my own network. Pkgsrc gave me the solution to be able to build the software from source for any operating system on any cpu architecture that I could find and do it all from my own server.
More Articles
After searching for more information on pkgsrc, I found these other cool links:
Followup
To keep this from ballooning into one gigantic article, I’ll keep this one as simply why I think pkgsrc is cool and why I wanted to use it. I’ll follow up with another article about how exactly I’m using it.
I should probably also write another article about using Clang as the default pkgsrc compiler since I messed around with that until I finally got it to work. Their documentation about it is a bit scattered and misleading.
I did end up writing more posts about pkgsrc!
