While that’s great, in that it gives us a nice performance boost, what’s also worth calling out is that the number of splats being drawn per frame was now truly variable, and decided entirely on the GPU. The pipeline had grown into the right kind of shape a level-of-detail system needs; drawing a different subset every frame, decided at the last possible moment.

I hadn’t set out to build an on-ramp to LODs. But that’s what it turned out to be.

Hubris: Redux

Lifting that instance cap felt like the finish line. However, in less than a week I hit the next wall.

Because the GPU Scene exfiltration had changed how many splats I could draw, but not how many I could hold: every splat in the scene still lived in GPU memory, all the time. That's completely fine when the whole scene fits. But obviously a splat isn't free in terms of memory. Position, scale, orientation, opacity, spherical harmonic coefficients, all told about 32 bytes per-splat once I'd packed it as tightly as I dared.

Which implies you can throw a lot of splats at VRAM, until you meet a real photogrammetry capture of a real site, at which point you discover that reality is annoyingly high-resolution and doesn’t care about your VRAM budget.

The captures we actually wanted to show people were bigger than the capacity of the machines, and there was no more clever packing to be done. So the whole scene can't be resident at once. Which means only part of it can be. Which means the renderer has to decide, every frame, which part to render, go and fetch it and then draw it on-screen. Stop me if you’ve heard this before.

What LOD even means for splats

If you've done anything in real-time graphics, you know about LODs: far-away things get a cheaper stand-in. A distant tree mesh becomes a lower-poly tree, then a billboard, then a smudge. Things get cheaper over distance.

Splats don't have polygons to throw away, but they have something better: there are simply too many of them, and a lot of that density is wasted on a region that's forty metres away and six pixels wide. So a splat LOD is a decimated copy of a region; the same blob-cloud, with most of the blobs removed, and the survivors enlarged to cover the gaps. At LOD level 0, we show every splat, and at higher LOD levels we show progressively fewer.

The format we settled on, PlayCanvas's Streamed SOG, bakes this for you. It essentially carves the scene into a spatial tree of regions ("leaves"), and each leaf describes several decimated LOD levels. A little lod-meta.json at the root then describes the overall tree structure and where each chunk lives.