We've found a new way to speed up parallel Golang continuous integration workflows by taking advantage of the Golang build cache. Replacing GitHub's official actions/setup-go action with a drop-in equivalent cut our test job runtimes by 69%. We're open-sourcing cloudx-io/setup-go (opens in a new tab) so you can do the same.

GitHub's official actions/setup-go step makes parallel jobs interfere with each other's performance, and it continuously loads stale cache values. Backtesting in our monorepo, which has the common situation of a few parallel Golang test jobs (one for lints, one for tests, one for builds), suggests that 86% of the work the default action does is completely unnecessary.

If you manage a moderately complex Go project, you can expect similar performance improvements; see our CI measurement methodology or just try it for yourself.

We Care About Fast CI

We've been shipping a lot of new products and features, and the pace at which we do it is actually increasing over time. This is no accident — we invest heavily in the tools and processes required to make this possible. At the center of every "software factory" are the test suites and Continuous Integration (CI) workflows that ensure code changes won't break in production. If our tests run reliably, and quickly, on every change, we can build at fantastic speed without worrying about breaking things for our customers.

This is important to us, so we measure and invest in the speed of our CI jobs. If you push code to a CloudX repository, our goal is that you get a clear answer as to its acceptability — whether it builds, its tests pass, and it abides by our linter rules — within 90 seconds.

Speed can be achieved in a number of ways, but at the end of the day if you want things to be fast you have to make algorithmic improvements. We're already using Warp Build (opens in a new tab) to run our CI jobs on fast, cost-efficient machines. As our test suite has scaled with our product surface area, we realized that actions/setup-go was not setting us up for success.

How actions/setup-go fails for parallel jobs

GitHub's actions/setup-go (opens in a new tab) is the GitHub-encouraged way to install and run Go in GitHub Actions. It uses actions/cache internally to save and restore the local Go module cache and build cache directories. In principle, that should make downloaded module source code and build/test artifacts from one job run available to all the subsequent job runs in your repo. Here's the default actions/setup-go cache key construction:

This cache key is woefully incomplete: in a typical product under active development, only a tiny minority of code changes modify the target operating system, architecture, Go version, or go.mod files.

The first time a job computes this hash key, it persists the final cache state to the GitHub cache service. Until the next change that modifies one of those key elements, every single CI run will load that first value. As you change your application, the restored go build module archives from this first run weaken — each subsequent build does more work from scratch. The restored go test outputs go stale too, so each subsequent job reruns more tests. CI degrades until you update go.mod!

Moreover, multiple parallel jobs running actions/setup-go race to write different local cache states to the GitHub cache service — different because the final Go cache state on a runner depends both on the source code and on the commands run. For example, you might run separate lint and test jobs in parallel:

Both jobs resolve the same default cache key, then race to write its value. Suppose the lint job finishes first: it saves a value without an updated test cache state. Subsequent test jobs will keep using that stale value until the cache key changes, and therefore re-run tests unnecessarily.

Linting, building, and testing a codebase are ideal candidates for memoization: their outputs (linter messages, built binaries, and test results respectively) should be pure functions of the source code. You can store outputs and reuse them rather than recomputing them, so long as the inputs haven't changed.

Several parts of the standard Go toolchain save their outputs to the filesystem and check if they can reuse an existing output instead of recomputing a new one from scratch:

Go's module cache saves time spent downloading source code for your module dependencies, which you can trigger explicitly with go mod download but also implicitly with go build. There's nothing mysterious here, just source code organized by the package identifiers in your go.mod:

You trigger fresh downloads when you change your go.mod, e.g. to add a new dependency or upgrade an existing one.

Go's build cache and test cache are actually located together in the GOCACHE directory and share a general structure. Both build and test processes hash their full inputs for use as a cache key. Those hashes are organized into subdirectories by prefix, and used as filenames for the reusable process outputs:

Files with the suffix -d are data payloads, and the -a-suffixed files serve as indexes. Of course, build and test processes yield different data payloads:

• go build stores package archives, intermediates that are linked into a final binary.
• go test stores stdout, stderr, and the final exit code of the test execution.

The Go test runner spies on the test process, automatically detects what files it reads, and incorporates their contents as inputs to the cache key.

The principles underlying these tool caches are the same: they maximize hit rates by making keys of complete but minimal sets of dependencies, so misses only occur when absolutely necessary. Whenever there's a miss, the new result is always persisted to the cache so future processes can reuse it.

This works brilliantly in a single persistent filesystem, but CI runners don't have the benefit of a single persistent filesystem. In GitHub Actions, these toolchain caches are smuggled from one ephemeral runner to the next by stowing them in yet another cache — one with very different design priorities.

The GitHub Actions cache

GitHub's base actions/cache (opens in a new tab) just knows keys and filepaths. You give GitHub's cache service a key of your own design. If the cache service recognizes the key, it loads the corresponding cached files into your runner; otherwise, it loads nothing. If and only if this primary-key lookup missed, actions/cache saves these files to the cache service after your CI job completes.

Once you write an object to the GitHub cache service under a certain key, that key-value pair is immutable. Any subsequent calls that would persist a different value for that key are rejected.

There is nothing wrong with any of that. Indeed, actions/cache is indispensable, and it uses GitHub's cache access restrictions (opens in a new tab) to prevent cache poisoning.

The hard part is picking good keys.

Improving setup-go

cloudx-io/setup-go is effectively a drop-in replacement for actions/setup-go; here's why we actually prefer it for our web monorepo:

• We're happy to pay a premium to keep engineers and coding agents unblocked. That means parallelizable work must run in parallel (even if this increases billed runner time by repeating setup work), and we gladly pay a few bucks per month for extra cache space.
• Our build, test, and lint workloads are much faster when they can reuse prior cached values. If all our tests were wicked fast (maybe one day they will be!) or all our lint rules wimpy, we wouldn't sweat our GOCACHE hit rates.

Our main insight is just that the Go toolchain is really, really good; a good CI caching strategy has to preserve that toolchain's most important properties across lots of ephemeral runner instances, while working within GitHub's constraints — i.e. still adapting actions/cache.

Let's revisit the important properties one by one.

They maximize hit rates by making keys of complete but minimal sets of dependencies, so misses only occur when absolutely necessary.

GitHub's default setup-go keying is incomplete because it doesn't capture what a given job actually does. That's why the test and lint jobs in the example above race to write a single, partial cache entry.

cloudx-io/setup-go solves this by making the job identity (or any arbitrary cache-key-prefix input) part of the Actions cache key. The lint job and test job save and restore separate caches without conflicts.

Whenever there's a miss, the new result is always persisted to the cache so future processes can reuse it.

GitHub's default setup-go only saves a new cache entry when go.mod changes, even though there's new data written to the runner's local cache directories every time you build or test a new version of your source code.

Instead of discarding that incremental effort, cloudx-io/setup-go writes a cache entry every single time: the final element in its key is the GitHub Actions run ID. The fully-qualified cache key includes several other elements to encourage prefix-matching in a git-aware way:

Measuring performance

Late last year, while we still used the default action, we encountered exactly the race condition discussed above: our parallel lint job saved a Go cache without test results, which slowed our test jobs from a 76-second median runtime to an unacceptable 180-second median. Remember, this slowdown represents exactly zero value: the jobs slowed down to re-test logic completely unchanged from the run before.

Eliminating the race by separating caches for our various jobs immediately solved this problem: we introduced cloudx-io/setup-go, test jobs resumed loading appropriate caches, and the median job runtime fell to 41 seconds, a 69% improvement.

Even if we lint and test in series, cloudx-io/setup-go would outperform the default because it saves an updated cache state after every run. Using the GitHub default, the loaded cache grows progressively staler between key changes (go.mod changes). With our new strategy, the loaded cache is always fresh from the run before; the test run for a commit only exercises test packages genuinely modified by that commit.

In aggregate, we wait for 86% fewer test packages to run now that we load a fresher cache. To run the counterfactual comparison on real data, we took a sequence of 4,000 real commits, calculated the action IDs for each snapshot's test packages, and modeled cache-hit rates under the old and new key constructions.

Of course, your mileage will vary (according to how often you change go.mod). To be transparent, we've seen two downsides to the switch, both because we save so many more cache objects:

1. Initially our cache blobs grew linearly with each run; eventually they grew so large that cache-load times became a major factor in our overall CI time. This is an issue present in the actions/setup-go default behavior too: Go's cache doesn't prune itself; it grows until you clear it. We save the cache more often, so it grows faster. We solved this with automatic pruning.
2. You may need an expanded GitHub Actions cache capacity. This is offset by making your jobs faster — runners bill by the minute — but locating the necessary settings in GitHub is a pain.

cloudx-io/setup-go (opens in a new tab) has been stable internally since November of last year. We hope it saves your team some time, and we look forward to hearing what you think!

(opens in a new tab)Did you read this while waiting for your CI to finish?
Explore careers at CloudX!