Artificial intelligence news.
Straight from the original announcements.

Google publishes TPU guide for faster LLM and MoE workloads

November 16, 2026By the AI News Bites Editorial TeamSource: Google for Developers
Google publishes TPU guide for faster LLM and MoE workloads
Image: Google for Developers

Google has launched the open-source AI-Hypercomputer/tpu-codesign guide, offering hardware-specific recommendations for designing and tuning models on its TPU systems. The guide targets large language models, Mixture-of-Experts models and other hardware-intensive workloads.

Key points
  • Google launched the open-source AI-Hypercomputer/tpu-codesign guide.
  • The guide recommends aligning key model dimensions to 256 for TPU systolic arrays.
  • LatentMoE compresses routed tokens before expert dispatch.
  • Google measured 2.8× lower permute time in a 30B, 128-expert MoE test.
  • The recommendations cover Ironwood, TPU 8t, TPU 8i, TPU 8tx and TPU 8i Max.

Design models around TPU architecture

Google says TPUs are built for machine learning workloads, with wide systolic arrays, dedicated SparseCore engines and custom interconnect fabrics. Those features can deliver strong performance per total cost of ownership, but they also create architectural constraints that developers must account for when designing models.

The guide covers training and inference on Ironwood, identified in the post as TPU v7x, as well as future TPU generations. Google divides its recommendations into three groups: shape rules, architecture choices and ways to express parallelism. Shape rules are presented as practices developers should adopt, while architecture choices should be tested through ablations because they can involve model-quality tradeoffs.

Shape alignment is a central recommendation

One of the guide's main rules is to align reduction and output dimensions to 256, matching the dimensions of TPU's 256×256 MXU systolic array. Google specifically highlights head_dim, saying that values of 64 or 128 can leave the MXU at least 50% idle during flash-attention QK products.

The guide recommends using head_dim = 256 with proportionally fewer query and key-value heads, while preserving total projection widths and logical KV cache bytes. According to Google's measured Ironwood profiling, reshaping Qwen3-30B MoE attention from 32 × 128 to 16 × 256, with identical parameters and FLOPs, improved end-to-end training speed, with larger gains at longer context.

Google also recommends static shapes because XLA compiles each program for exact tensor shapes. For dynamic workloads, it describes using buckets, masking and packed training sequences, while keeping buffers fixed and representing changing data extents with small integer arrays.

LatentMoE can reduce routed-token traffic

Another highlighted design choice is LatentMoE. Instead of dispatching every routed token at the full d_model width, LatentMoE projects tokens into a narrower latent width before sending them to experts, then projects them back after combining the results.

Google says this approach can reduce expert weight reads and all-to-all dispatch traffic. Its TPU-specific recommendations include routing on the full d_model before compression, keeping the latent width a multiple of 256, using FP8 for dispatch and BF16 for the lightweight projections, and placing expert parallelism on one short 3D torus axis.

In Google's measured 30B, 128-expert MoE test, a latent width of 768, described as 2.67× compression, cut on-device permute time by 2.8× and increased end-to-end training throughput by 14% in tokens per second. The reported comparison was 322 versus 270 TFLOP/s per chip, and the compressed latent representation freed about 25 GB of activation HBM per device.

For users adapting an existing model, Google says the guide's parallelism section can be applied without changing the checkpoint. The company says the broader rules also apply to dense LLMs, multimodal models and world models, while the accompanying repository will add model- and workload-specific material over time.

GoogleIronwoodTPU v7xTPU 8txTPU 8i MaxAI-Hypercomputerchipsdata centerslanguage models

Related news

A CatBoost model predicts fine-tuning time far better than FLOPs
ResearchNovember 18, 2026, 19:53

A CatBoost model predicts fine-tuning time far better than FLOPs

A CatBoost model predicted the end-to-end duration of fine-tuning jobs far more accurately than conventional FLOP calculations in a Hugging Face study of 272 production…

AWS brings managed NVIDIA Dynamo inference to SageMaker HyperPod
Products and toolsNovember 18, 2026, 00:33

AWS brings managed NVIDIA Dynamo inference to SageMaker HyperPod

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining the open source framework's distributed inference capabilities with HyperPod's infrastructure…

Google details bitwise-aligned TPU training for stable LLM reinforcement learning
ResearchNovember 16, 2026

Google details bitwise-aligned TPU training for stable LLM reinforcement learning

Google has described an open-source Tunix implementation designed to eliminate numerical differences between sampling and training during reinforcement learning for…