vLLM reports 58x throughput for DeepSeek-V4.1-Flash on Vera Rubin

vLLM says it is serving DeepSeek-V4.1-Flash at 1.7M tokens per second per GPU on an NVIDIA Vera Rubin NVL72 rack, while delivering 387 tokens per second to each user. The company reports 58x the throughput of a GB300 NVL72 system, with peak throughput reaching 2.52M tokens per second.
- vLLM reports 1.7M tokens/s per GPU on Vera Rubin NVL72.
- Each user receives 387 tokens/s, according to vLLM.
- The company reports 58x the throughput of GB300 NVL72.
- Peak throughput reaches 2.52M tokens/s.
- A single user reaches 1,810 tokens/s at batch size 1.
Rubin hardware drives the performance gains
The results come from adapting vLLM's DeepSeek-V4.1-Flash serving stack to Vera Rubin, which the company says introduces several optimization opportunities. The rack's GPUs have two memory domains, and streaming bandwidth-bound data from an SM's own domain is faster than accessing interleaved or remote memory.
vLLM rebuilt or reworked the model's MoE, attention, GEMM, mHC and indexer kernels for Rubin. The company also uses SM107's wider block-scaled matrix multiplication instructions, including K64 mixed-precision FP8×FP4 MMA and K128 FP4 MMA. According to vLLM, MegaMoE gains 8, 11% throughput at 8K or more tokens per rank, while a low-batch routed GEMM runs 12% faster.
The system also uses a rack-wide NVLink domain. vLLM says this allows TP8 to span two compute trays with almost no extra all-reduce cost, while prefill instances can read one another's HBM efficiently.
Kernel changes reduce prefill and decode time
For prefill, vLLM reports about 28% less time per DEP2 step at 32K tokens per rank. The changes include keeping the mHC residual stream local to each memory domain, improving attention pipelines and filtering experts before scoring them. MegaMoE tile sizing was also changed to account for the larger of local and data-parallel average token counts.
For decode, the company reports about 25% less time per DEP32 step compared with its first Rubin build. The changes include die-local expert weights and a Wide-EP dispatch design that reduces remote memory traffic. vLLM says the DEP2 decode forward is 10.1% faster at batch size 32 and 5.5% faster at batch size 168.
A specialized tensor-parallel decode path spreads the model over two trays with TP8. In that configuration, vLLM reports 1,810 tokens per second at the 90th percentile for a single user at batch size 1.
Rack-scale memory supports higher concurrency
Vera Rubin's speed allows the system to handle substantially higher concurrency, according to vLLM. The company pools rack memory through Dynamo workload-aware routing. Its Mooncake KV pool extends from DRAM to spare HBM, while Engram tables are sharded across GPU HBM.
vLLM reports a prefix hit rate of 96.5, 97% even at very high concurrency, and says routing alone balances the ranks. The benchmark is SemiAnalysis AgentX, which vLLM uses as a representative agentic serving workload and measures with cached input, uncached input and output tokens.
The results extend vLLM's earlier DeepSeek-V4.1-Flash work. The first post covered day-0 bring-up and kernel fusions, while the second scaled the model to a GB300 NVL72 rack using P/D disaggregation and Wide-EP. The Vera Rubin results show how the same serving stack changes when it can use Rubin's memory layout, FP4 tensor cores and rack-wide NVLink.


