A scaling law converts a compute budget into a model: fix the FLOPs, minimize the predicted-loss curve, read off a parameter count and a token budget. It is the most useful instrument pretraining has, and it is denominated in a currency no cluster accepts. Clusters do not bill FLOPs. They bill GPU-hours.
I have been pulling at this thread on this blog for a while. The post on model size and training horizon ended by converting compute into machine time,
and the pretraining research map filed MFU-aware comparison under where scaling laws get hard: two designs can tie on loss-per-FLOP and differ on loss-per-GPU-hour, and the second number is the one on the invoice. Our new paper with Soumajyoti Sarkar and Yuxin Tang takes that observation to its conclusion. The framework, MOSAIC, folds the systems stage into the scaling-law stage: one optimization picks the architecture, the token budget, and the distributed execution layout, for a specific cluster and a specific training window. The title gives away the finding: compute-optimal is not cluster-optimal.
Optimizing the numerator
Look at the denominator of that fraction. Goodput — how often the job runs, after checkpointing, node failures, and input stalls — is a property of the fleet, fixed before any architecture is chosen. Peak throughput is a property of the hardware. MFU — how well the job runs while it runs — is a property of the architecture under its best feasible parallel layout, and it is not a small correction: two designs with identical predicted loss can differ several-fold in realized throughput.
The convention treats it as a constant anyway. A scaling-law stage picks the architecture and the token budget in FLOPs; a systems stage inherits the architecture and tunes the implementation around it. (The Kimi K2 report, for example, treats sparse-MoE scaling laws and systems tuning as separate stages.) That workflow optimizes the numerator while the cluster charges for the whole fraction.
Sparse MoEs are where it breaks
A mixture-of-experts model replaces the dense FFN with a set of experts and a router, which decouples active parameters — the per-token FLOPs — from total parameters — the memory footprint. That introduces axes a Chinchilla-style law cannot see: sparsity, and the expert split factor , which measures how finely the FFN width is partitioned into experts. The paper fits a joint law over total parameters, sparsity, tokens, and on roughly 150 from-scratch MoE pretraining runs, spanning 104M to 2.7B active parameters and total sizes up to 79B.
The fitted law delivers an uncomfortable result. Under a fixed model-FLOPs budget, predicted loss falls monotonically with sparsity across the entire calibrated range. There is no interior optimum. The FLOPs-only answer is “as sparse as your search space allows,” which is not a design principle so much as the absence of one.
Price the same grid on hardware and the ordering inverts. At a world size of 512 GPUs, wall-clock per delivered model FLOP rises monotonically with sparsity, up to 1.70× the densest configuration. The design that ranks first on loss per FLOP is the one the cluster runs slowest.
The same design grid, scored two ways. On loss per model FLOP (left), sparser and finer-grained wins. On wall-clock per delivered FLOP (right), the same picks are the slowest — 1.70× on the sparsity axis, 2.66× on the expert-split axis. The winners sit at opposite ends. From Figure 8 of the paper.
Putting the cluster inside the problem
MOSAIC replaces the model-FLOPs budget with deliverable FLOPs: the cluster’s raw ceiling times the MFU that candidate can attain under its best feasible layout, times goodput. A candidate is feasible only if the FLOPs its recipe requires fit inside the FLOPs the cluster can deliver while executing it.
The MOSAIC workflow. Cluster inputs enter on the left; a geometry ladder proposes realizable architectures; the performance model prices each in MFU and the scaling law prices each in loss. What comes out is a model configuration together with the layout to execute it. From Figure 2 of the paper.
Selection becomes a bilevel problem: the outer loop minimizes predicted loss over a discrete grid of realizable geometries and token budgets; the inner loop maximizes deliverable FLOPs over parallel layouts — tensor, pipeline, data, context, and expert degrees, micro-batch size, activation checkpointing — subject to device memory. The MFU and memory surfaces are measured lookups rather than symbolic functions, so the paper solves it by structure-exploiting enumeration instead of a generic solver.
A worked example shows what the constraint buys. Prescribe model FLOPs and give it four p6-B200 nodes for five days. The law alone pushes sparsity to the boundary of the search space. But on this cluster, in this window, configurations past a sparsity of about 0.96 cannot deliver the FLOPs their own recipe requires. The boundary pick is not merely expensive. It is infeasible.
Left: under a fixed model-FLOPs budget of 3.3 × 10²¹, predicted loss falls all the way to the edge of the calibrated range, so the optimum is the boundary. Right: the same prescription priced on four p6-B200 nodes for five days — past a sparsity of about 0.96, configurations deliver fewer FLOPs than the recipe requires. From Figure 1a of the paper.
Re-optimize under the same envelope, with every candidate evaluated at the token count its own attainable MFU supports, and the optimum moves inside: an interior sparsity whose predicted loss is 0.031 nats lower than the boundary pick. Nobody capped a model dimension by hand. The optimum emerged from the cluster.
Minimizing the same loss law subject to deliverable FLOPs produces an interior optimum (green star), 0.031 nats better than the model-FLOPs pick (red star) under an identical four-node, five-day envelope. From Figure 1b of the paper.
The estimator that makes the inner loop cheap is an operator-level performance model: analytical FLOPs, collective-communication equations, and the 1F1B pipeline schedule, calibrated with microbenchmarks on the target hardware, with exactly one labeled fitted correction term. On p6-B200 nodes, across models from 700M to 18B active parameters, its predicted-MFU error stays under 15% per sweep — and, the property a selector actually needs, its top-5 and top-10 sets coincide with the measured ones on every 5.4B and 18B sweep. It is open source as ScalePlan.
The candidate with the most FLOPs is not the best model
Give every candidate the same 32 nodes for 20 days and they all draw on the same raw ceiling, FLOPs. What differs is conversion, and the expert split factor orders it: the finest geometries () sit near 8% predicted median MFU while coarser ones () reach 13—20%, because expert dispatch and combine communication leaves fewer cycles for the model — a penalty tied to the Megatron-Core-style stack the performance model mirrors. Read on model FLOPs alone, fine-grained experts look cheapest. Under a fixed hardware ceiling, they are the ones the cluster feeds least.
The consequence is direct. The candidate that converts the ceiling into the most model FLOPs, at , lands at a higher predicted loss than the winner, which needs — roughly 36% fewer. Ranking designs by the FLOPs they emit misorders them.
Every feasible configuration under the 32-node, 20-day envelope, colored by expert split factor. All draw on the same raw ceiling, so a point sits further right only by sustaining a higher MFU. The gold star — the configuration MOSAIC returns — is conspicuously not the rightmost point. From Figure 10 of the paper.
Two quieter results from the sweeps. Optimal sparsity does not sharpen as compute grows; it sits on a wide plateau, roughly 0.915 to 0.963. And optimal active parameters move slowly — near 14—15B as the training window stretches, reaching 34B and 58B only at the two largest cluster sizes, with near-ties at neighboring budgets leaving the exact crossovers open.
A fitted surface can be validated on the values it predicts or on the ordering it induces, and the ordering is what a practitioner acts on. So the paper ends with real runs: four configurations spanning the trade-off, selected against an eight-node, ten-day envelope, each launched as a staged pretraining run — up to 250B total parameters, trained past the selection envelope itself.
The measured MFU reproduced the predicted ordering exactly. The loss trajectories then showed the thesis in a single plot: between the two leading configurations, the ordering flips with the axis. Against model FLOPs, the finer-grained, lower-active-parameter run leads over much of the lower range. Against peak-equivalent hardware compute — charging each run for the hardware time it actually consumed — the other run leads, and the crossover comes far earlier. It sustained a normalized MFU of 0.84 against 0.69, so the same hardware bought it about 1.2× the model computation. Same runs, same data; the axis you price them on decides which one you would ship.
Smoothed loss trajectories of the four staged runs. Left: against model FLOPs, ID 2 (green) leads over much of the range. Right: against peak-equivalent hardware compute, which divides out the MFU each run actually sustained, ID 1 (blue) takes over far earlier. The same four runs, ranked two ways. From Figure 12 of the paper.
The honest hedge: on the model-FLOPs axis the two curves converge near the end of the runs — the paper reads that lead as a tendency over the budget range covered, not a separation that holds to the end of training.
What it does not settle
The specific geometries MOSAIC returns are conditional on the geometry ladder, the hardware platform, and the training recipe, and some reported optima sit beyond the regime the law was fit on. Uncertainty in the law’s non-identified coefficients is not propagated through the discrete search. The performance model mirrors a fairly basic Megatron-Core stack, so the absence of a clean trend in optimal may be that stack’s artifact rather than a property of MoEs — better kernels could change it. The evidence supports the hardware-aware ranking and the co-design principle more strongly than it supports any single geometry as a universal answer.
The claim I would defend beyond MoEs
If MFU depends on the architecture, then the architecture decision is partly a systems decision, and the two-stage convention has the wrong org chart baked into it. The pretraining teams that win are hybrid ML-and-systems teams, and this is the workflow that makes the hybrid concrete: every candidate design gets priced on loss and on MFU before any compute is spent. The cluster was always going to charge for both.