Oct 31^st, 2023 @ justine's web page

After nearly one year of development, I'm pleased to announce our version 3.0.1 release of the Cosmopolitan library. The project is an entirely new animal. For starters, Mozilla sponsored our work as part of their MIECO program. Google also awarded me an open source peer bonus for my work on Cosmopolitan, which is a rare honor, and it's nice to see our project listed up there among the greats, e.g. curl, linux, etc. In terms of this release, we're living up to the great expectations you've all held for this project in a number of ways. The first is we invented a new linker that lets you build fat binaries which can run on these platforms:

• AMD64
  • Linux
  • MacOS
  • Windows
  • FreeBSD
  • OpenBSD
  • NetBSD
• ARM64
  • Linux
  • MacOS
  • Windows (non-native)

It's called apelink.c and it's a fine piece of poetry that weaves together the Portable Executable, ELF, Mach-O, and PKZIP file formats into shell scripts that run on most PCs and servers without needing to be installed. This is an idea whose time has come; POSIX even changed their rules about binary in shell scripts specifically to let us do it. So we've been using it to create a "Fat Linux Distro" which I've named the "Cosmos". In the Cosmos, every program is statically linked and contains a PKZIP central directory where its /usr/share dependencies are embedded. You can think of it as a coalition of individualistic executables, where each program can be separated from the whole and run on other OSes. So far it includes programs like Emacs, Vim, CoreUtils, Curl, Git, etc.

cosmos-3.0.1.zip
213mb - PE+ELF+MachO+ZIP+SH executables
For AMD64+ARM64 on Linux+Mac+Windows+FreeBSD+NetBSD+OpenBSD
ce256ededf106748a09f13bf47ace9ef0e6f115d963353d3d63c21302c5f28f4

More specifically, the above zip file contains fat binaries for ape, python, lua, qjs, vim, emacs, emacsclient, nano, llama, bash, dash, less, git, grep, curl, wget, tidy, zip, unzip, zstd, bzip2, sqlite3, life, nesemu1, make (GNU + SECCOMP + Landlock), gmake (GNU), redbean, greenbean, datasette, assimilate, rusage, ctags, wall, pledge, verynice, find, tree, basename, chgrp, cp, date, du, expr, groups, ls, mknod, nl, pathchk, pwd, rm, seq, shred, stat, tee, uname, users, who, basenc, chmod, csplit, dd, echo, factor, head, mktemp, nohup, pinky, rmdir, shuf, stty, test, true, unexpand, vdir, whoami, awk, chown, df, false, id, link, mv, nproc, pr, rsync, sleep, sum, truncate, uniq, yes, b2sum, chroot, dir, fmt, install, md5sum, numfmt, printenv, readlink, runcon, sort, sync, timeout, tsort, unlink, base32, cat, cksum, cut, dircolors, env, fold, join, ln, mkdir, od, printf, realpath, script, split, tac, touch, tty, wc, base64, chcon, comm, dirname, expand, kill, logname, mkfifo, nice, paste, ptx, sed, sha1sum, sha224sum, sha256sum, sha384sum, sha512sum, tail, tr, ttyinfo, and uptime.

This only became possible in the last few months, in part thanks to Gautham Venkatasubramanian, who spent a few weekends of his PhD studies modifying the C language so it's possible to build conventional software with Cosmopolitan. Since then it's been out of the frying pan and into the fire, testing our library to see if it can support some of the most complex and mature projects in the open source community. Running the ./configure scripts and make check rules of projects (e.g. GMP) has done so much to help us fix bugs and battle test new features.

One of the things we're most happy with, is that Cosmo's cross platform support is now good enough to support Cosmo development. We've traditionally only compiled code on x86 Linux. Devs using Cosmo would build their programs on Linux, and then copy the binaries to other OSes. Focusing on Linux-only helped us gain considerable velocity at the start of the project; the Cosmopolitan monorepo has two million lines of code. Today was the first day the whole thing compiled on Apple Silicon and Microsoft Windows systems, and using Cosmo-built tools.

Windows Improvements

In order to get programs like GNU Make and Emacs to work on Windows, we implemented new libraries for POSIX signals emulation. Cosmopolitan is now able to preempt i/o and deliver asynchronous signals on Windows, using a SetThreadContext() trick I learned from the Go developers. Cosmo does a considerably better job spawning processes now too. For example, we wrote a brand new posix_spawn() function that goes 10x faster than the posix_spawn() included with Cygwin. Cosmo's execve() can now reparent subprocesses, inherit non-stdio file descriptor, and our read() function now contains a termios driver which, for the first time, lets us poll() standard input on consoles. Cosmo binaries cleanly integrate with WIN32, depending pretty much only on KERNEL32, and your fat binaries won't have to live on a separate partition like WSL.

MacOS Improvements

While MacOS may not be the prodigal child of our support vector, this release brings improvements to MacOS users that are equally important. For starters, we now have first-class native ARM64 support. APE Loader also now dynamically links the officially blessed Apple libraries (e.g. libSystem.dylib) on ARM64, so there's less chance that Apple will break your binaries. We've also made semaphores and futexes much better on XNU, thanks to Grand Central Dispatch, and ulock on AMD64.

Portability and Performance (Pick Two)

The end result is that if you switch your Linux build process to use cosmocc instead of cc then the programs you build, e.g. Bash and Emacs, will just work on the command prompts of totally different platforms like Windows and MacOS, and when you run your programs there, it'll feel like you're on Linux. However portability isn't the only selling point. Cosmo Libc will make your software faster and use less memory too. For example, when I build Emacs using the cosmocc toolchain, Emacs thinks it's building for Linux. Then, when I run it on Windows:

It actually goes 2x faster than the native WIN32 port that the Emacs authors wrote on their own. Cosmo Emacs loads my dotfiles in 1.2 seconds whereas GNU Emacs on Windows loads them in 2.3 seconds. Many years ago when I started this project, I had this unproven belief that portability toil could be abstracted by having a better C library. Now I think this is all the proof we need that it's not only possible to make software instantly portable, but actually better too. For example, one of the things you may be wondering is, "these fat binary files are huge, wouldn't that waste memory?" The answer is no, because Cosmo only pages into memory the parts of the executable you need. Take for example one of Linux's greatest hits: the Debian Almquist shell.

$ ls -hal /usr/bin/dash
-rwxr-xr-x 1 root root 107K Nov 21  2022 /usr/bin/dash
$ ls -hal /opt/cosmos/bin/dash
-rwxr-xr-x 1 jart jart 983K Oct 15 19:14 /opt/cosmos/bin/dash

Here we see Cosmo's six OS + two architecture fat binary dash is 30% bigger than the one that comes with Alpine Linux (which only supports x86-Linux and dynamically links a separate 600kb Musl library). But if I run them:

$ rusage /usr/bin/dash -c true
took 231µs wall time
ballooned to 688kb in size
needed 183us cpu (0% kernel)
caused 34 page faults (100% memcpy)

$ rusage /opt/cosmos/bin/dash -c true
took 217µs wall time
ballooned to 544kb in size
needed 172us cpu (0% kernel)
caused 36 page faults (100% memcpy)

Here we see Cosmo's fat binary version of dash went faster and used less memory than an x86-Linux-only binary built for Musl Libc. This is due to (1) the magic of modern memory management, where CPU MMUs lazily load 4096 byte blocks at a time; and (2) how carefully apelink plans your executable layout. For example, all that code which is needed to support Windows (it takes a lot of code to support Windows) gets linked into its own special section of the binary, far away from what the MMU will want to page on UNIX systems. The same goes for the embedded ARM64 build. Since I'm running on AMD64 here, the ARM64 code is never loaded off disk.

You no longer need to choose between the amalgamation release or the cosmo monorepo. We now have a third preferred option, which is our new cosmocc command. It works pretty much the same as the cc command you already know. The Cosmopolitan README file has getting started instructions, which basically boil down to this:

sudo mkdir -p /opt
sudo chmod 1777 /opt
git clone https://github.com/jart/cosmopolitan /opt/cosmo
export PATH="/opt/cosmo/bin:/opt/cosmos/bin:$PATH"
echo 'PATH="/opt/cosmo/bin:/opt/cosmos/bin:$PATH"' >>~/.profile
ape-install        # optionally install a faster systemwide ape loader
cosmocc --update   # pull cosmo and rebuild toolchain

If your development environment isn't x86_64 Linux, then you'll need to download the latest toolchain, rather than using the one that's vendored inside the cosmo repository.

cd /opt/cosmo
rm -rf o/third_party/gcc
wget https://github.com/jart/cosmopolitan/releases/download/3.0.1/cosmocc-0.0.16.zip
unzip cosmocc-0.0.16.zip

If you're on Windows, then you need a shell before you can do any of the above. If you download the cosmos.zip file at the top of the page, it'll have a bin/ and libexec/ folder. Just extract those to your C:\ drive. Then install Terminal Preview from the Microsoft Store, and configure it so that it launches C:\bin\bash -l as your shell. You'll naturally need a good unzip program in order to do this: one that supports symbolic links (a.k.a. reparse points). The cosmos zip includes InfoZIP, but you can download it directly here: unzip.exe.

Now that your cosmo compiler is installed and added to your $PATH, you can build standard C and C++ programs as follows:

cosmocc -o foo foo.c
cosmoc++ -o bar bar.cc

If you want to build programs as fat binaries, there's a special compiler now which can make the process easier than using the x86_64-unknown-cosmo-cc, aarch64-unknown-cosmo-cc, fixupobj, and apelink commands directly. This compiler works by manipulating your compiler flags so that a concomitant .aarch64/foo.o gets created for every foo.o you compile. It runs the x86_64 and aarch64 compilers in parallel too, so it doesn't go much slower than cosmocc at the end of the day.

fatcosmocc -o foo foo.c
fatcosmoc++ -o bar bar.cc

Integrating with GNU Autotools projects is easy as well.

export CC=cosmocc
export CXX=cosmoc++
./configure --prefix=/opt/cosmos
make -j
make install

Refer to the README file for more detailed instructions. Also be sure to check out the ahgamut/superconfigure repository for the largest source of high-quality examples on how to build open source software using Cosmo.

This cosmos release includes the latest version of the redbean web server. This is a fat single-file forking Lua+SQLite+MbedTLS stack written by the Cosmopolitan authors originally to showcase the capabilities of the library for greenfield development. Cosmopolitan is good for more than just building old GNU code and redbean proves that. It's the third most upvoted hobby project in Hacker News history. A few weeks ago Berwyn Hoyt independently determined it to be the fastest Lua web server too!

One of the reasons why redbean is a forking web server is because we didn't develop our own POSIX Threads implementation until last year. So we put a lot of thought into writing an example of how you can build a bare minimal threaded web server that's even faster than redbean, and it's called greenbean. There's a prebuilt fat binary for it in the cosmos.zip distribution above. Greenbean is 400 lines of liberally commented perfection. I still can't believe how good we got its memory usage, thanks to tricks like MAP_GROWSDOWN. On Linux, if I ask greenbean to spawn over 9,000 persistent worker threads:

$ sudo prlimit --pid=$$ --nofile=18000
$ sudo prlimit --pid=$$ --nproc=18000
$ rusage greenbean 9001
listening on http://127.0.0.1:8080
listening on http://10.10.10.237:8080
greenbean workers=0 connections=0 messages=0 ^C shutting down...
took 7,959,324µs wall time
ballooned to 40,352kb in size
needed 929,917us cpu (92% kernel)
caused 10,039 page faults (100% memcpy)
54,689 context switches (99% consensual)

Then it somehow only uses 40mb of peak resident memory, and according to htop, greenbean's virtual memory usage is 76,652kb. That's for 9,001 threads. Like redbean, greenbean is able to handle hundreds of thousands of requests per second on my Intel Core i9-9900, except (1) greenbean has better shared memory support, (2) it sets up and tears down connections faster, and (3) it lets you experience the joy of using Mike Burrows' *NSYNC library, which is the basis of Cosmopolitan's POSIX synchronization primitives. If you're not familiar with the man, he's the guy who coded Chubby and Altavista, a global search engine which was so efficient it only needed to operate on a single server. But you wouldn't think *NSYNC is as prolific as it is if you're only going off star count.

My favorite thing about greenbean is how elegantly it reacts to CTRL-C. When you interrupt greenbean, it'll use the POSIX pthread_cancel() API to immediately terminate all the worker threads, so that shutdown happens without delay. Cancelation is one of the trickier concepts for a C library to get right. It's up there with threads, signals, and fork in terms of how its ramifications pervade everything. So similar to Musl Libc, we've put a lot of thought into ensuring that Cosmo does it correctly. Cosmo avoids cancelation race conditions the same way as Musl and Cosmo also implements Musl's PTHREAD_CANCEL_MASKED extension to the POSIX standard, which has seriously got to be one of Rich Felker's most brilliant ideas. Read the commentary in greenbean.c if you want to learn more.

Some of you might view greenbean as just a toy example. In that case, if you want to see something based on greenbean that is actually running in production, then pay a visit to https://ipv4.games/ whose source code is in net/turfwar/turfwar.c and net/turfwar/.init.lua. This is a hybrid redbean + greenbean service, where redbean does the HTTPS frontend, and greenbean does the HTTP backend. Hackers love to unleash their botnets on the IPv4 Games, and it honestly isn't that hard to withstand a DDOS with 49,131,669 IPs when your web server can do a million requests per second. That's for a service where 99% of the requests are write requests. If you want to have fun with this server, then you're also welcome to check out our monitoring metrics while you do it.

This cosmos.zip release includes several games you can play which have been vetted on all our supported platforms. For example you can actually play Nintendo in the terminal and it'll even work inside the Windows 10 command prompt. Getting poll(stdin) to work in Windows is a messier problem than even naming or cache invalidation, so it's super sweet that Cosmopolitan Third Edition is now able to abstract that complexity. Although in the case of our port of Bisqwit's fabulous NESEMU1 program, it turned out we didn't need poll() at all! Just calling read() and letting it be EINTR'd by setitimer() sixty times a second to pipe audio did the trick. So playing these games helped battle test our new signals implementation too.

We may not have GUI support yet, but you can use your mouse in the terminal. On Windows, your mouse cursor will generate the same ANSI XTERM style control codes as it does on Linux, MacOS, BSD, etc. Try running the life program that's included in Cosmos. Left click draws cells. Space runs an iteration of the life game. Right click can drag the display. You can also ctrl+wheel to zoom the display in and out. To read the source code to this simple program, check out tool/viz/life.c.

Boulder startup dylibso just announced a few weeks ago that they've adopted Cosmopolitan for their new product Hermit: Actually Portable Wasm. Hermit lets you create secure cross-platform executables for WebAssembly modules. It's worth a try! Cosmopolitan has a long history of serving the needs of the Wasm community. Our first major adopter back in early 2021 was actually the wasm3 project which provides a similarly great solution to running Wasm outside the browser. I'm happy to see these projects benefiting from the advantages Cosmopolitan has to offer. Will you be the next adopter? If so, feel free to reach out to me personally and I'll see what I can do to help you be successful: [email protected]. You're also invited to join our Discord community.

Funding for Cosmopolitan Third Edition was crowdsourced from Justine Tunney's GitHub sponsors and Patreon subscribers, the backing of Mozilla's MIECO program, and the generous contributions of our developer community on Discord. Your support is what makes projects like Cosmopolitan possible. Thank you!