In which I painstakingly remove functionality from a Linux live image. Like a normal person.
But first, I can haz ISO at all
One of NixOS' cooler party tricks is that you can
trivially take some configuration and run it as a virtual machine (VM).
nixos-rebuild build-vm will give you one for your system
configuration. But you can also do it for any given configuration
you want, system or not!
let pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/567a49d1913ce81ac6e9582e3553dd90a955875f.tar.gz";
sha256 = "1vq77hlx8mi3z03pw2nf6r5h7473r1p9yxyf58ym3fh01zppmfln";
}) {};
in pkgs.nixos {
system.stateVersion = "26.05";
services.getty.autologinUser = "root";
}
That's enough to get a minimal VM, you can run it yourself with
$(nix-build basic-vm.nix --attr vm --log-format bar --no-out-link)/bin/run-nixos-vm.
This will create a "thin" VM: there's a disk image, but it only
contains the files you've made yourself once you're inside of the
VM. Everything else (cough /nix/store) gets mounted in from your
host OS instead.
And that's great when it fits, but sometimes we just need something a bit more normal. Something we can just run in libvirt, or even ship off to some remote host that might not have Nix. It might not even run Linux at all. Hell, there might not even be a hypervisor!
In short, I just want a damn ISO.
Thankfully, NixOS still has my back here:
let pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/567a49d1913ce81ac6e9582e3553dd90a955875f.tar.gz";
sha256 = "1vq77hlx8mi3z03pw2nf6r5h7473r1p9yxyf58ym3fh01zppmfln";
}) {};
in pkgs.nixos ({ lib, ... }: {
system.stateVersion = "26.05";
services.getty.autologinUser = "root";
imports = [ "${pkgs.path}/nixos/modules/installer/cd-dvd/iso-image.nix" ];
image.baseName = lib.mkForce "nixos";
})
We'll need to run QEMU ourselves this time, but it's not too bad:
qemu-system-x86_64 --cdrom $(nix-build basic-iso.nix --attr isoImage --log-format bar --no-out-link)/iso/nixos.iso -m 1G --accel kvm
Trouble in paradise
So we're done here, right? But... we haven't even touched the
salad actual topic of the title! So, uh, about that...
$ ls -lh $(nix-build basic-iso.nix --attr isoImage --log-format bar --no-out-link)/iso/nixos.iso
-r--r--r-- 1 root root 458M jan 1 1970 /nix/store/kg8mv6296hbhm8als26r400nj1s7ry1n-nixos.iso/iso/nixos.iso
Uh oh. 458MiB‽ And it doesn't even do anything yet!
[root@nixos:~]# vim
-bash: vim: command not found
That's almost 10x larger than what my childhood's Damn Small Linux needed to run a surprisingly complete desktop environment! We're probably not going to reach those levels, but surely there's room to do at least a little better.
For comparison Alpine's VM ISO currently sits at around 66MiB. So I guess that's a reasonable baseline to compare against.
What else even is there?
Well, let's have a look at what we're paying for...
$ sudo mount $(nix-build basic-iso.nix --attr isoImage --log-format bar --no-out-link)/iso/nixos.iso iso --mkdir
$ du iso --all --block-size=1M | sort -n | tail -n10
3 iso/isolinux
13 iso/boot/nix/store/92id8yn2g9kj7bskld42p222pmk3y3ms-linux-6.18.35
13 iso/boot/nix/store/92id8yn2g9kj7bskld42p222pmk3y3ms-linux-6.18.35/bzImage
26 iso/boot/nix/store/zvn61hpw86ncvgsr8vjrfi3ahnk2c9hb-initrd-linux-6.18.35
26 iso/boot/nix/store/zvn61hpw86ncvgsr8vjrfi3ahnk2c9hb-initrd-linux-6.18.35/initrd
39 iso/boot
39 iso/boot/nix
39 iso/boot/nix/store
416 iso/nix-store.squashfs
458 iso
So.. 416MiB for the main userspace, 26MiB for the early boot environment, and 13MiB for the kernel itself. At least we can dismiss the latter two for now. So let's crack that squash open too, and see what secrets it hides...
$ sudo mount iso/nix-store.squashfs squash --mkdir
$ du squash --max-depth=1 --block-size=1M | sort -n | tail -n20
11 squash/bizyfqdw0h67wzqmp10knmf9s2pqahdb-file-5.47
13 squash/pa0x6m662kr9vr875fcp1cl9wwkswsa8-coreutils-full-9.11
14 squash/dscn63ni6fp8p58y99b3ywk478b4bvmv-texinfo-interactive-7.2
14 squash/gf6i4cbisapj28y2dnqhpk1s95vd2r36-util-linux-2.42-lib
15 squash/qmajd4nyy8zf70g9p4x2lcq45gq5gzy3-boost-1.89.0
16 squash/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1
21 squash/cfjm0s2jnlaiz9y3byvfa0fc6fp2la20-systemd-minimal-260.1
22 squash/92id8yn2g9kj7bskld42p222pmk3y3ms-linux-6.18.35
24 squash/dis2sflz0lifcji3gb81rbr7896dw39l-nix-manual-2.34.7
26 squash/zvn61hpw86ncvgsr8vjrfi3ahnk2c9hb-initrd-linux-6.18.35
28 squash/ryi73l6ic3pz9vh69nj0i4l188vkqgaq-nixos-manual-html
30 squash/r1nzk3ga4fk9q2xw77md2ln88m98d2vc-grub-2.12
32 squash/cl0a76f140zrrbx0n0qpjx07794q5gzn-grub-2.12
34 squash/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61
39 squash/clpq5c7bysml4vqpa1x60a5yk3nzkfj4-icu4c-76.1
56 squash/6plwsm6pkq79yjv4xvy8csk2pd4hzr67-perl-5.42.0
60 squash/a8avqfxd649rfgfpqldja6v38ljb8fj5-systemd-260.1
128 squash/60m4rxhg2fldqaak400c0lry96ijrzqn-python3-3.13.13
144 squash/f060awdpif3c41v4wbshd6h6jzbxdv66-linux-6.18.35-modules
1064 squash
Well, that looks an awful lot like a Nix store... And since we built
it on the host machine, anything inside of it should be in our host
store, too! So we can just replace squash/ with /nix/store/, and
then use nix why-depends to find out where the dependencies come from!
For example, we can ask where the Boost dependency comes from:
$ nix why-depends $(nix-build basic-iso.nix --attr config.isoImage.storeContents --no-out-link) /nix/store/qmajd4nyy8zf70g9p4x2lcq45gq5gzy3-boost-1.89.0 --precise
/nix/store/w27j1xagd7cb5qxm1phmvffdwnk4b3wc-nixos-system-nixos-26.11pre-git
└───activate: …fsx38qi-setup-etc.pl /nix/store/7nvwsgl5cplzal9y5m9nnwjjrbbvzcjv-etc/etc..if (( _localstatus > 0…
→ /nix/store/7nvwsgl5cplzal9y5m9nnwjjrbbvzcjv-etc
└───etc/tmpfiles.d/00-nixos.conf -> /nix/store/0zl414yp4h3ppb63h20pljy5rn5v10w4-tmpfiles.d/00-nixos.conf
→ /nix/store/0zl414yp4h3ppb63h20pljy5rn5v10w4-tmpfiles.d
└───nix-daemon.conf -> /nix/store/hqwkw2nala59avjximpdmn1yi474n4h7-nix-2.34.7/lib/tmpfiles.d/nix-daemon.conf
→ /nix/store/hqwkw2nala59avjximpdmn1yi474n4h7-nix-2.34.7
└───bin/nix: …1.3.2.GLIBCXX_3.4.21./nix/store/a90px0h1szz9y9c67ww1sa163z44flf9-nix-util-2.34.7/lib:/nix/store/…
→ /nix/store/a90px0h1szz9y9c67ww1sa163z44flf9-nix-util-2.34.7
└───lib/libnixutil.so.2.34.7: …5-libcpuid-0.8.1/lib:/nix/store/qmajd4nyy8zf70g9p4x2lcq45gq5gzy3-boost-1.89.0/lib:/nix/store>
→ /nix/store/qmajd4nyy8zf70g9p4x2lcq45gq5gzy3-boost-1.89.0
Ah! So we install the Nix daemon, and that brings in Boost with it.
Let the slimmening commence
After looking through the ever-valuable
search.nixos.org for a moment, we should be
able to disable shipping Nix entirely, by setting nix.enable = false.
While we're at it, there's also a killswitch for disabling
documentation entirely, documentation.enable = false. So let's try
both of those...
let pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/567a49d1913ce81ac6e9582e3553dd90a955875f.tar.gz";
sha256 = "1vq77hlx8mi3z03pw2nf6r5h7473r1p9yxyf58ym3fh01zppmfln";
}) {};
in pkgs.nixos ({ lib, ... }: {
system.stateVersion = "26.05";
services.getty.autologinUser = "root";
imports = [ "${pkgs.path}/nixos/modules/installer/cd-dvd/iso-image.nix" ];
image.baseName = lib.mkForce "nixos";
nix.enable = false;
documentation.enable = false;
})
So.. great success, let's go home and celebrate?
$ ls -lh $(nix-build sans-nix.nix --attr isoImage --log-format bar --no-out-link)/iso/nixos.iso
-r--r--r-- 1 root root 384M jan 1 1970 /nix/store/rm5zhcmxmypgg4rl2yb2c9lcpaqda85y-nixos.iso/iso/nixos.iso
Well, it's a start. But we're not done. Annoyingly, we can also see that we didn't quite manage to vanquish Boost yet.
$ nix why-depends $(nix-build sans-nix.nix --attr config.isoImage.storeContents --no-out-link) /nix/store/qmajd4nyy8zf70g9p4x2lcq45gq5gzy3-boost-1.89.0 --precise
/nix/store/aznzf6j6mjllgxllmx92wgmjc977ryin-nixos-system-nixos-26.11pre-git
└───activate: …fsx38qi-setup-etc.pl /nix/store/3kmvk7vy9gr83xnpsd4kg7zvscpa6nr1-etc/etc..if (( _localstatus > 0…
→ /nix/store/3kmvk7vy9gr83xnpsd4kg7zvscpa6nr1-etc
└───etc/systemd/system -> /nix/store/3h0kw6ryg8d7zqwhj5aza7imi4n94631-system-units
→ /nix/store/3h0kw6ryg8d7zqwhj5aza7imi4n94631-system-units
└───register-nix-paths.service -> /nix/store/8jqrvyk58rfixir8lgz2hxqnabhvss6i-unit-register-nix-paths.service/register-nix-paths.service
→ /nix/store/8jqrvyk58rfixir8lgz2hxqnabhvss6i-unit-register-nix-paths.service
└───register-nix-paths.service: …nged=false.ExecStart=/nix/store/cc50wj89l6j4fyw43hvnwqwh0zn590q7-unit-script-register-nix-paths-…
→ /nix/store/cc50wj89l6j4fyw43hvnwqwh0zn590q7-unit-script-register-nix-paths-start
└───bin/register-nix-paths-start: …tabase in the tmpfs../nix/store/bvkx110ylicifcgl0xiid5f100hx3ar7-nix-2.34.7/bin/nix-store --load…
→ /nix/store/bvkx110ylicifcgl0xiid5f100hx3ar7-nix-2.34.7
└───libexec/nix-nswrapper -> /nix/store/97zxp9j00zcjmkn3zv9karhwj86q7x5w-nix-nswrapper-2.34.7/libexec/nix-nswrapper
→ /nix/store/97zxp9j00zcjmkn3zv9karhwj86q7x5w-nix-nswrapper-2.34.7
└───libexec/nix-nswrapper: …X_3.4.GLIBCXX_3.4.20./nix/store/a90px0h1szz9y9c67ww1sa163z44flf9-nix-util-2.34.7/lib:/nix/store/…
→ /nix/store/a90px0h1szz9y9c67ww1sa163z44flf9-nix-util-2.34.7
└───lib/libnixutil.so.2.34.7: …5-libcpuid-0.8.1/lib:/nix/store/qmajd4nyy8zf70g9p4x2lcq45gq5gzy3-boost-1.89.0/lib:/nix/store/f15…
→ /nix/store/qmajd4nyy8zf70g9p4x2lcq45gq5gzy3-boost-1.89.0
Looks like we still have some paths that can reach Nix (and Boost). In this case, it will try to register the contents of the ISO's store on boot.
That's of course required for people to be able to use Nix once the image is booted.. the very same Nix that we already removed!
So let's try blanking it out...
let pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/567a49d1913ce81ac6e9582e3553dd90a955875f.tar.gz";
sha256 = "1vq77hlx8mi3z03pw2nf6r5h7473r1p9yxyf58ym3fh01zppmfln";
}) {};
in pkgs.nixos ({ lib, ... }: {
system.stateVersion = "26.05";
services.getty.autologinUser = "root";
imports = [ "${pkgs.path}/nixos/modules/installer/cd-dvd/iso-image.nix" ];
image.baseName = lib.mkForce "nixos";
nix.enable = false;
systemd.services.register-nix-paths = lib.mkForce {};
documentation.enable = false;
})
Will this finally work better?
$ ls -lh $(nix-build sans-nix-redux.nix --attr isoImage --log-format bar --no-out-link)/iso/nixos.iso
-r--r--r-- 1 root root 360M jan 1 1970 /nix/store/gk39b6lnq2dbg85dzksqhwb93ijshhf3-nixos.iso/iso/nixos.iso
$ nix why-depends $(nix-build sans-nix-redux.nix --attr config.isoImage.storeContents --no-out-link) /nix/store/qmajd4nyy8zf70g9p4x2lcq45gq5gzy3-boost-1.89.0 --precise
'/nix/store/mnn0zxj3ccwkzh4mpj3zfwf2z1spm2xv-nixos-system-nixos-26.11pre-git' does not depend on '/nix/store/qmajd4nyy8zf70g9p4x2lcq45gq5gzy3-boost-1.89.0'
Looks like it, phew! Though the size win didn't turn out to be that big.
Sans Undertale SSH?
After following on for a bit in a similar pattern (as it turns out,
environment.defaultPackages is right there and right for the
clearing), ssh turns out to be a bit trickier than I'd have liked to
get rid of.
modules/programs/ssh.nix
adds
it to environment.corePackages, which sounds like the sort of option
that I'd rather not screw with wholesale right now. Normally, that'd
be controlled by an option like programs.ssh.enable, but I can't seem
to find one.
We can exclude modules wholesale with
disabledModules,
but that leads to a bunch of knock-on errors from other modules that
expect it to be there.
error: The option `programs.ssh' does not exist. Definition values:
- In `/nix/store/pfwrb65dsv8phlsf1m98bvz11cvgb290-source/nixos/modules/services/desktop-managers/plasma6.nix':
{
_type = "if";
condition = false;
content = {
askPassword = {
...
Did you mean `programs.zsh', `programs.bash' or `programs.fish'?
We could disable services/desktop-managers/plasma6.nix too, but it
also has a bunch of reverse dependencies. That seems like a pretty deep
rabbit hole to go down, for just one package. Instead, we can provide a
stub for the programs.ssh options that doesn't actually use it for
anything. That should be enough to keep us going for at least a bit
longer..
let pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/567a49d1913ce81ac6e9582e3553dd90a955875f.tar.gz";
sha256 = "1vq77hlx8mi3z03pw2nf6r5h7473r1p9yxyf58ym3fh01zppmfln";
}) {};
in pkgs.nixos ({ lib, ... }: {
system.stateVersion = "26.05";
services.getty.autologinUser = "root";
imports = [
"${pkgs.path}/nixos/modules/installer/cd-dvd/iso-image.nix"
{
# A lot of modules assume that the programs.ssh option exists,
# so let's provide a stub for them.
options.programs.ssh = lib.mkOption {};
}
];
image.baseName = lib.mkForce "nixos";
nix.enable = false;
systemd.services.register-nix-paths = lib.mkForce {};
documentation.enable = false;
documentation.man.enable = false;
networking.firewall.enable = false;
disabledModules = [
# Injects openssh as a core package without a clear disable switch
"programs/ssh.nix"
];
environment.defaultPackages = lib.mkForce [];
})
Wait, why are we importing the option?
A NixOS module really has three parts:
{ lib, ... }: {
# module-level options
imports = [ ... ];
disabledModules = [ ... ];
# option definitions
options.programs.ssh = lib.mkOption {};
# implementation (sets other options)
config.networking.firewall.enable = false;
}
But if a module doesn't define any options then there's a shorthand
where you don't have to put config. before every implementation
attribute. That's why you haven't seen config. in your generated
configuration.nix, or in the rest of this post. I'd rather not give
up on that shorthand for the rest of the post, so I'm going to isolate
all the options into a separate module instead.
Chicken on a raft
One of the bigger fruits left is around 62MiB of GRUB (the bootloader). Now, we do want a bootloader (and GRUB is a pretty decent one), but we probably don't need to bundle all of its installation tools. We especially don't really need to bundle them twice. For some reason, NixOS' ISO preset bundles both UEFI and BIOS versions.
Now, we don't have a clear knob to disable this, but we can take the
pretty blunt path of just.. resetting system.extraDependencies and
environment.systemPackages to get rid of it. We can't reset
systemPackages entirely though, let's keep environment.corePackages
to keep the shell at least somewhat usable.
let pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/567a49d1913ce81ac6e9582e3553dd90a955875f.tar.gz";
sha256 = "1vq77hlx8mi3z03pw2nf6r5h7473r1p9yxyf58ym3fh01zppmfln";
}) {};
in pkgs.nixos ({ lib, config, ... }: {
system.stateVersion = "26.05";
services.getty.autologinUser = "root";
imports = [
"${pkgs.path}/nixos/modules/installer/cd-dvd/iso-image.nix"
{
# A lot of modules assume that the programs.ssh option exists,
# so let's provide a stub for them.
options.programs.ssh = lib.mkOption {};
}
];
image.baseName = lib.mkForce "nixos";
nix.enable = false;
systemd.services.register-nix-paths = lib.mkForce {};
documentation.enable = false;
documentation.man.enable = false;
networking.firewall.enable = false;
disabledModules = [
# Injects openssh as a core package without a clear disable switch
"programs/ssh.nix"
];
environment.defaultPackages = lib.mkForce [];
environment.systemPackages = lib.mkForce config.environment.corePackages;
system.extraDependencies = lib.mkForce [];
})
Do we really need all those kernel modules?
There's also 144MiB of just kernel modules. That's about a quarter of our total size, and more on its own than the entire Alpine image! I'm sure some of them are nice to have, but we can't need.. all of them, surely?
NixOS doesn't really (to my knowledge) have a good hook for limiting
which modules are available at runtime, like it automatically does for
the initrd. But we can just remove the modules folder from the
"system" completely. Of course, the tradeoff here is that we
effectively disable runtime module loading entirely. Any modules that
you do want to load will have to go in
boot.initrd.kernelModules instead.
let pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/567a49d1913ce81ac6e9582e3553dd90a955875f.tar.gz";
sha256 = "1vq77hlx8mi3z03pw2nf6r5h7473r1p9yxyf58ym3fh01zppmfln";
}) {};
in pkgs.nixos ({ lib, config, ... }: {
system.stateVersion = "26.05";
services.getty.autologinUser = "root";
imports = [
"${pkgs.path}/nixos/modules/installer/cd-dvd/iso-image.nix"
{
# A lot of modules assume that the programs.ssh option exists,
# so let's provide a stub for them.
options.programs.ssh = lib.mkOption {};
}
];
image.baseName = lib.mkForce "nixos";
nix.enable = false;
systemd.services.register-nix-paths = lib.mkForce {};
documentation.enable = false;
documentation.man.enable = false;
networking.firewall.enable = false;
disabledModules = [
# Injects openssh as a core package without a clear disable switch
"programs/ssh.nix"
];
environment.defaultPackages = lib.mkForce [];
environment.systemPackages = lib.mkForce config.environment.corePackages;
system.extraDependencies = lib.mkForce [];
system.systemBuilderCommands = lib.mkAfter "rm $out/kernel-modules";
})
This seems to have lost us the ability to switch to a slightly more comfortable display resolution, but otherwise I'd still call this fully bootable. And we're now down to 197MiB!
Perls for swine
We're also still spending 56MiB on Perl.
Returning to nix why-depends for a moment, it turns out that it's
only used to configure users and /etc during system activation. We
can't throw those away entirely, but thankfully there are now
experimental features for managing /etc with an overlay, and users
with the native userborn. Both
are listed as experimental, but.. uh.. I think we're deeply into the
off-label use territory already.
let pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/567a49d1913ce81ac6e9582e3553dd90a955875f.tar.gz";
sha256 = "1vq77hlx8mi3z03pw2nf6r5h7473r1p9yxyf58ym3fh01zppmfln";
}) {};
in pkgs.nixos ({ lib, config, ... }: {
system.stateVersion = "26.05";
services.getty.autologinUser = "root";
imports = [
"${pkgs.path}/nixos/modules/installer/cd-dvd/iso-image.nix"
{
# A lot of modules assume that the programs.ssh option exists,
# so let's provide a stub for them.
options.programs.ssh = lib.mkOption {};
}
];
image.baseName = lib.mkForce "nixos";
nix.enable = false;
systemd.services.register-nix-paths = lib.mkForce {};
documentation.enable = false;
documentation.man.enable = false;
networking.firewall.enable = false;
disabledModules = [
# Injects openssh as a core package without a clear disable switch
"programs/ssh.nix"
];
environment.defaultPackages = lib.mkForce [];
environment.systemPackages = lib.mkForce config.environment.corePackages;
system.extraDependencies = lib.mkForce [];
system.systemBuilderCommands = lib.mkAfter "rm $out/kernel-modules";
system.etc.overlay.enable = true;
system.etc.overlay.mutable = false;
services.userborn.enable = true;
})
183MiB. Less massive, but still nice.
Calling it quits
So that's where I think I'll leave things, for now at least. Compared to where we started, it's almost a third of the original image. But still not something that I'd call good. Oh well, maybe I'll revisit the topic in the future.
So should you actually.. do this?
It.. depends, I guess?
For anything serious, or a desktop that you're going to actually use? Pretty definitely not, those things are all there for a reason.
But sometimes you just need something tiny that you can boot for an experiment, that just needs to perform some tiny task. Perhaps this can serve as some sort of inspiration for that. But it's still not just a question of copying and pasting my final configuration; there's probably something stripped out here that you actually do need for what you're doing.
I don't know.
At some point I just kept going because I got curious.
Homework
This whole post has focused on the things that can "just" be removed, or that have straightforward replacements.
But there are also plenty of things you could do that require a deeper
effort. For example, right now we're still bundling both
systemdMinimal and systemd, which seems pretty silly. But trying to
drop either seemed to screw up other parts of the build in ways that I
don't have the time to explore right now.
There are also probably plenty of smaller things that could be stripped out, that still add up to significant sizes together.
Maybe that could be considered homework for some intrepid reader? If anyone does end up taking it on, it'd be cool to hear about in the comments!