利用按位零训练-推理不匹配在 TPU 上实现稳定同策略强化学习
Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

原始链接: https://developers.googleblog.com/unlocking-stable-on-policy-rl-on-tpu-with-bitwise-zero-traininference-mismatch/

Zero-TIM(零训练–推理不一致)是面向基于 TPU 的大语言模型强化学习的 Tunix 实现,它使采样器解码、服务端预填充和训练器前向传播在相同权重下产生按位相同的对数概率。这消除了由不同归约顺序、中间精度和执行路径造成的隐藏数值离策略差距。该方法采用四部分配方:具有固定累加顺序和 float32 部分和的批不变 Pallas 内核;用于跨分片归约的显式源秩排序;共享的对数概率评分路径和保留的 token ID;以及复用服务内核进行前向评估同时对匹配副本求导的自定义 VJP。长轨迹使用分页 KV 缓存状态分块。在 64 颗 TPU v5p 芯片上验证,该方法在 FrozenLake 上 200 步内实现了采样器–训练器对数概率平均绝对差为零,并且比标准和 TIS 基线具有更高的求解率。模块化的 JAX/XLA/Pallas 模式旨在更广泛地支持多主机 TPU 强化学习采用。

一个 Hacker News 帖子链接到一篇 Google 博客文章,内容是关于通过消除训练-推理不匹配,在 TPU 上解锁稳定的 on-policy 强化学习。该帖有 10 分和一条评论。评论者提出另一种方案:让模型从略微 off-policy 的 rollout 中学习,并将人类 on-policy 学习与运动员观看自己近期尝试的学习作比较。他们指出,ML 可能避免使基于视频的学习对人类效果较差的感官限制。
相关文章

原文

The problem

Reinforcement learning (RL) for large language models is notoriously sensitive to numerical instability: even tiny discrepancies between rollout generation and gradient updates can destabilize training or stall convergence. In LLM RL, the model serves as the policy, π(action | previous tokens), and true on-policy learning requires every action in an update to be generated and evaluated by that exact same policy. While a synchronous loop ensures each rollout batch is scored using the same weights that generated it, a critical numerical requirement remains: the sampler and trainer must evaluate that policy identically.

In practice, production RL stacks run sampling and training through separate, specialized execution paths. The sampler chooses each action token during incremental decoding and records its behavior logprob. The trainer evaluates the same token in batched forward-backward passes to compute its current logprob and gradients. Under the same weights, these differing execution paths can still assign different logprobs to the same token and context. The training schedule is on-policy, but the numerical execution has introduced a hidden off-policy gap that compounds over multi-turn trajectories.

Our zero train–inference mismatch (Zero-TIM) implementation, developed in Tunix and verified at multi-host scale across 64 TPU v5p chips (DP8×TP8), closes this numerical gap. At the same weights, the sampler and trainer assign bitwise-identical logprobs to the same action and token history—bringing state-of-the-art, true on-policy stability to TPU RL training.

Why it happens

A reduction is a tree of floating-point additions. Floating-point addition is not associative, so changing the tree changes where rounding occurs. The effect is easier to expose in bfloat16, which retains less precision than float32.

The following bfloat16 example isolates the effect of reduction order:

figure-1-reduction-trees

Figure 1. Near 2.0, each 0.0078125 input is half the distance to the next representable bfloat16 value. Round-to-nearest-even discards each input when added separately; grouping the same inputs first reaches the next value.

The same issue appears in model reductions: matrix products accumulate partial sums, normalization computes statistics, and tensor-parallel shards combine partial outputs. In our original stack, sampler decode and trainer forward used different token-array shapes and forward implementations. JAX specializes computations to their input shapes, and the training forward also runs inside a differentiated program.

Across these paths, reduction order and intermediate precision were not consistently aligned. Floating-point rounding turned those arithmetic differences into different logprob bits. The weights and model equations were identical; the arithmetic was not.

The alignment contract

The contract is simple: across sampler decode, serving prefill, and trainer forward, all three paths must produce bitwise-identical logprobs for every sampled action—comparing raw floating-point bytes under the same rollout weights, token history, and sampling settings, with zero rounding or tolerance.

Why three paths instead of two? In a decoupled RL stack, a direct sampler–trainer disagreement shows that a numerical gap exists, but not where it enters. Introducing serving prefill as a middle checkpoint turns alignment into a systematic two-step diagnostic that any RL stack can adopt:

Sampler decode vs. prefill verifies that the serving engine agrees with itself between incremental token generation and full-sequence re-scoring, while prefill vs. trainer verifies that the serving and training engines agree across compilation and differentiation boundaries.

figure-2-policy-contract

Figure 2. Bitwise comparisons of sampled-action logprobs within the serving engine and across the serving–training boundary, evaluated at the same rollout weights.

How we built zero-TIM on TPU

Achieving Zero-TIM does not require merging the sampler and trainer into a single monolithic engine. Instead, we use a generalizable four-part recipe that bridges separate serving and training pipelines across multi-host TPU meshes: locking within-shard kernel accumulation, ordering cross-shard reductions, standardizing the logprob scoring tail, and wrapping serving kernels with custom differentiation rules.

Fix within-shard arithmetic

Inside each tensor-parallel rank, batch-invariant Pallas kernels control accumulation order and intermediate precision. Projection kernels traverse fixed contraction blocks in order, keeping the reduction blocking independent of token-row count. Partial products accumulate in float32, with explicit output casts.

The production kernel initializes its scratch accumulator before the tile traversal, then applies this core to each tile:

input_tile = x_ref[...]
weight_tile = y_ref[...]

partial = jnp.dot(
    input_tile,
    weight_tile,
    preferred_element_type=jnp.float32,
)
acc_ref[...] = acc_ref[...] + partial
out_ref[...] = acc_ref[...].astype(out_ref.dtype)

Python

The next tile continues from the float32 accumulator, not the cast output. This construction covers the attention and MLP projections and the LM head. Dedicated kernels fix RMSNorm reductions and SwiGLU’s intermediate casts. The existing Pallas paged-attention forward is retained.

Order cross-shard reductions

For row-parallel projections, each tensor-parallel rank produces a partial output. We separate communication from addition: ppermute transports the partials, then every rank adds them in the same source-rank order.

The reduction order is explicit in the program rather than selected by a collective implementation.

Share the action-logprob path

Alignment must extend beyond model logits to the probability of the sampled action. Decode, prefill, and trainer apply matching sampling transforms, use the same fixed Pallas log-softmax construction, and select the recorded action token.

Prefill and trainer score the action generated during rollout rather than choosing another action. We also preserve sampled token IDs across environment turns so that every path evaluates the same token history.

Differentiate the serving forward

The trainer reuses the serving prefill computation. Custom vector–Jacobian products retain the serving kernels for forward evaluation and provide their backward rules. The matmul example shows this separation:

@jax.custom_vjp
def serving_matmul(inputs, weights):
    return pallas_matmul(inputs, weights)

def matmul_forward(inputs, weights):
    output = pallas_matmul(inputs, weights)
    return output, (inputs, weights)

def matmul_backward(saved, output_gradient):
    inputs, weights = saved
    _, pullback = jax.vjp(
        differentiable_matmul,
        inputs,
        weights,
    )
    return pullback(output_gradient)

serving_matmul.defvjp(matmul_forward, matmul_backward)

Python

The differentiable replica mirrors the kernel’s accumulation and casts, but is used only to construct the pullback. Forward values still come from the serving Pallas kernel.

Training places that forward inside a differentiated JAX program. We pass --xla_allow_excess_precision=false through XLA_FLAGS to the compiler process, constraining excess intermediate precision in the operations surrounding the Pallas kernels.

Long trajectories are evaluated in fixed-size chunks, carrying the paged KV cache forward. All action logprobs feed the complete train-batch RL loss. The reverse pass propagates gradients through those cache connections and combines parameter-gradient contributions before the optimizer update.

figure-3-chunked-differentiation

Figure 3. KV-cache state connects the forward chunks; cache gradients connect their reverse passes. Parameter-gradient contributions are combined before the optimizer update.

Bringing frontier RL numerical alignment to TPU

Across the AI systems community, eliminating train–inference mismatch has rapidly become a state-of-the-art requirement for stabilizing large-scale RL convergence. Recent open-source efforts—including vLLM with TorchTitan and RadixArk/LMSys Miles—have demonstrated how batch-invariant kernels and bitwise numerical alignment prevent off-policy drift during policy optimization.

Our work delivers an open-source, modular JAX/XLA and Pallas implementation of this contract for TPU in Tunix. Verified across a multi-host 64 TPU v5p (DP8×TP8) mesh, these kernel and collective patterns provide a reusable blueprint that teams can adopt to achieve bit-exact on-policy RL across distributed JAX/TPU stacks.

Evaluation and results

To benchmark our TPU kernels, we run a 200-step numerical verification test on the synthetic FrozenLake environment using a public Qwen3-8B checkpoint as a baseline test case using 64 TPU v5p chips with a DP8×TP8 mesh. The recipe uses a GSPO-token loss with leave-one-out advantages. We compare Standard training with trainer-old logprobs and no additional sampler correction, token-level truncated importance sampling (TIS), and Zero-TIM.

The Zero-TIM configuration also uses token-in/token-out (TiTO) handling to preserve sampled token IDs across environment turns. The comparison runs retain their existing tokenization paths.

figure-4-results

Figure 4. Single-seed training runs on FrozenLake grids from 2×2 through 9×9, with up to five interaction turns. Both panels show the first 200 training steps. Logprob differences are plotted without smoothing; solve rates show faint raw measurements and a ten-step trailing average. Endpoint labels show the final ten-step averages.

Zero-TIM records zero mean absolute sampler–trainer logprob difference at every logged step in this window, while its training solve rate increases. Standard and TIS retain nonzero differences: TIS adjusts their contribution to the update without removing the underlying mismatch. Zero-TIM reaches higher observed solve rates in these runs.

Code and updates

We developed Zero-TIM prototype as part of Tunix, Google's open-source, JAX-native library for LLM RL and post-training on Cloud TPU. The kernel patterns and three-way verification contract are modular by design—making them straightforward to adopt across existing multi-host JAX/TPU serving and training pipelines—and we are actively extending support to linear-attention and Mixture-of-Experts (MoE) architectures.

Preview the current implementation

联系我们 contact @ memedata.com