Disk I/O bottlenecks in GitHub Actions

ValdikSS · 2025-03-28T15:57:48 1743177468

`apt` installation could be easily sped-up with `eatmydata`: `dpkg` calls `fsync()` on all the unpacked files, which is very slow on HDDs, and `eatmydata` hacks it out.

nijave · 2025-03-28T15:59:13 1743177553

Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit

If you corrupt a CI node, whatever. Just rerun the step

wtallis · 2025-03-28T16:03:02 1743177782

CI containers should probably run entirely from tmpfs.

jacobwg · 2025-03-28T16:07:16 1743178036

We're having some success with doing this at the block level (e.g. in-memory writeback cache).

candiddevmike · 2025-03-28T16:10:46 1743178246

We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching:

https://etcha.dev/etchaos/

jacobwg · 2025-03-28T16:04:36 1743177876

I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.

suryao · 2025-03-28T16:08:27 1743178107

TLDR: disk is often the bottleneck in builds. Use 'fio' to get performance of the disk.

If you want to truly speed up builds by optimizing disk performance, there are no shortcuts to physically attaching NVMe storage with high throughput and high IOPS to your compute directly.

That's what we do at WarpBuild[0] and we outperform Depot runners handily. This is because we do not use network attached disks which come with relatively higher latency. Our runners are also coupled with faster processors.

I love the Depot content team though, it does a lot of heavy lifting.

[0] https://www.warpbuild.com

（评论） (comments)

（评论）
(comments)