Google details bitwise-aligned TPU training for stable LLM reinforcement learning

Google has described an open-source Tunix implementation designed to eliminate numerical differences between sampling and training during reinforcement learning for large language models on TPU. The company says its Zero-TIM system produced bitwise-identical action logprobs across separate execution paths, including in tests spanning 64 TPU v5p chips.
- Google's Zero-TIM targets numerical differences between LLM sampling and training.
- The contract compares sampler decode, serving prefill and trainer logprobs bit for bit.
- The Tunix implementation uses Pallas kernels and explicitly ordered reductions.
- Google verified the approach across 64 TPU v5p chips in a DP8×TP8 mesh.
The hidden numerical gap in on-policy learning
On-policy reinforcement learning requires actions to be generated and evaluated by the same policy. In large language model systems, however, rollout sampling and gradient updates commonly use separate execution paths.
The sampler generates tokens incrementally and records their behavior logprob. The trainer then evaluates those tokens in batched forward and backward passes. Google says the two paths can produce different logprobs even when they use identical weights. This creates a numerical gap that can compound across multi-turn trajectories, making training less stable or preventing convergence.
The cause is floating-point arithmetic. Addition is not associative, so changing the order of a reduction changes where rounding occurs. Google says the problem is particularly visible with bfloat16, and can arise in matrix products, normalization, tensor-parallel reductions and other model operations.
A three-way contract for matching logprobs
Google's zero train-inference mismatch, or Zero-TIM, requires sampler decoding, serving prefill and trainer forward passes to produce bitwise-identical logprobs for every sampled action. The comparison uses raw floating-point bytes under the same rollout weights, token history and sampling settings, with no rounding tolerance.
Serving prefill provides a diagnostic checkpoint between the sampler and trainer. Sampler-versus-prefill comparisons test whether incremental generation and full-sequence rescoring agree within the serving engine. Prefill-versus-trainer comparisons test whether serving and training agree across compilation and differentiation boundaries.
The implementation keeps the serving and training systems separate. Its recipe aligns within-shard kernel accumulation, explicitly orders cross-shard reductions, standardizes the action-logprob calculation and adds custom differentiation rules around serving kernels.
How Zero-TIM works on TPU
Within tensor-parallel ranks, Pallas kernels use fixed accumulation orders and explicit intermediate precision. Projection kernels accumulate partial products in float32 before casting outputs, while dedicated kernels handle RMSNorm reductions and SwiGLU intermediate casts. Google says the approach covers attention and MLP projections as well as the language-model head.
For row-parallel projections, communication is separated from addition. The implementation transports partial outputs with ppermute, then adds them in a fixed source-rank order rather than relying on the order selected by a collective implementation.
The sampler, prefill and trainer use matching sampling transforms and the same Pallas log-softmax construction to score the recorded action token. Sampled token IDs are retained across environment turns so each path evaluates the same token history. The trainer reuses the serving prefill computation for forward evaluation, while custom vector-Jacobian products supply backward rules.
Google also says long trajectories are processed in fixed-size chunks while carrying the paged KV cache forward. It used the implementation in Tunix and verified it across a multi-host 64 TPU v5p mesh with a DP8×TP8 configuration. The company presents the work as a reusable blueprint for teams building distributed JAX and TPU reinforcement-learning systems.
Google for Developers: Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch- Google Developers Blog


