Jul 31, 2024

I talk a lot here about using computers freely, how to select programs to use, how to decide if a program is trustworthy infrastructure one can safely depend on in the long term. I also spend my time building such infrastructure, because there isn't a lot of it out there. As I do so, I'm always acutely aware that I'm just not very good at it. At best I can claim I try to compensate for limited means with good, transparent intentions.

I just spent a month of my free time, off and on, rewriting the core of a program I've been using and incrementally modifying for 2 years. I've been becalmed since. Partly this is the regular cadence of my subconscious reflecting on what just happened, what I learned from it, taking some time to decide where to go next. But I'm also growing aware this time of a broader arc in my life:

• Back in 2015 I was suspicious of abstractions and big on tests and version control. Code seemed awash in bad abstractions, while tests and versions seemed like the key advances of the 2000s. I thought our troubles stemmed from bad incentives, using abstractions too much, and not using tests and versions enough. Mu1 was an attempt at designing a platform with tests and layers (more like versions, less like abstractions) as foundational constraints influencing everything else.
• In 2017 I started reworking Mu1 into the current Mu. At the start I used all my new ideas for tests and layers. But over time I stopped using them. Mu today has tons of tests, but they are conventional tests, and I never got around to porting over my infrastructure for layers.
• In 2022 I started working on Freewheeling Apps. I started out with no tests, got frustrated at some point and wrote thorough tests for a core piece, the text editor. But I struggled to find ways to test the rest, and also found I was getting by fine anyway.
• Now it's 2024, and a month ago I deleted all my tests. I also started radically reworking my text editor, in a way that would have made me worried about merge conflicts with other Freewheeling Apps. In effect I stopped thinking about version control. Giving up tests and versions, I ended up with a much better program. The cognitive dissonance is now impossible to ignore.

After mulling it over for a few days, I think my current synthesis on programming durable things is:

1. Building durably for lots of people is too hard, just don't even try. Be ruled by what you know well, who you know well and Dunbar's number.
2. Most software out there is incurably infected by incentives to serve lots of people in the short term. Focus as far as possible on software without lots of logos on the website, stuff that is easy to build, has few dependencies, definitely doesn't auto-update. Once you filter by these restrictions, the amount of durable software humanity has created so far is tiny.
3. Small changes in context (people/places/features you want to support) often radically change how well a program fits its context. Our dominant milieu of short-termism doesn't prepare us for this fact.
4. Given this tiny body of past work and low coverage per program, any new program you decide to build is quite likely striking out into the unknown in some way or other. You often won't know quite what you're doing in some direction or other. (In my example above, I was trying to insert special "drawing lines" in a text editor. Questions that raised: can the cursor lie on a drawing? Can I try to draw in one line while the cursor is on another? Drawings are taller than text lines. Can a drawing be partially visible at top of screen? Can I draw on a partially visible drawing? My answers to these questions were sub-optimal for a long time, leading to hacks piled on hacks.)
5. Types, abstractions, tests, versions, state machines, immutability, formal analysis, all these are tools available to us in unfamiliar terrain. Use them to taste.
6. You'll inevitably end up over-using some of these tools, the ones you gravitate towards. The ideal quantity to use these tools is tiny, much more miniscule than any of us is trained to think by our dominant milieu of short-termism. The excess is tech debt. It keeps us from noticing that a program is unnecessarily complex, less durable than it could be, harder to change when the context shifts.
7. When your understanding of the context stabilizes, there's value in throwing away vast swathes of a program, and redoing it from scratch.
8. Before you set out to rewrite, you have to spend some time importing everything into your brain at once. Everything you want from the program, all the scenarios the program has to cater to. This is hard. The goal is to get to a point where you can build everything all at once.
9. Build everything all at once.

In my case, tests and versions actively hindered getting to the end of this evolution. Tests let me forget concerns. Version control kept me attached to the past. Both were counter-productive. It took a major reorientation to let go of them.

All the software I've written in my life — and all my Freewheeling Apps so far — are at level 6 in this trajectory. Only the output of the past month feels like it might have gotten to level 9. We'll see.

It seems likely that a program can grow so complex it becomes impossible to import into memory in level 8. That seems to describe most software so far, certainly most software written by more than a couple of people. Even my text editor, small as it is, was daunting enough I spent much of the month girding myself to face the terror.

Not all software necessarily needs to get to level 9. I think many of my Freewheeling Apps are simple enough and evolve slowly enough that they would stabilize to a bug-free state with just a handful of people using them, regardless of my initial design choices. Particularly now that I know how to streamline one complex piece at their core. Still, it's good to be aware of how things might be improved, if it becomes worthwhile.

One thing that feels definitely useful in getting to level 9 is data-oriented design. It's not a tool you can blindly apply but a way of thinking you have to grow into, to look past immediate data structure choices at the big picture of how your program accesses data. Just don't let tools like ECS blind you to the essential intellectual activity.

These levels are probably not quite right. I'm probably under-estimating tools I have less experience with.

I wonder what levels lie beyond these.

(I last wrote some thoughts on how I program back in 2019. It's nice to see signs of evolution.)

Comments gratefully appreciated. Please send them to me by any method of your choice and I'll include them here.