Artificial intelligence news.
Straight from the original announcements.

vLLM reports 58x throughput for DeepSeek-V4.1-Flash on Vera Rubin

November 12, 2026By the AI News Bites Editorial TeamSource: vLLM
vLLM reports 58x throughput for DeepSeek-V4.1-Flash on Vera Rubin
Image: vLLM

vLLM says it is serving DeepSeek-V4.1-Flash at 1.7M tokens per second per GPU on an NVIDIA Vera Rubin NVL72 rack, while delivering 387 tokens per second to each user. The company reports 58x the throughput of a GB300 NVL72 system, with peak throughput reaching 2.52M tokens per second.

Key points
  • vLLM reports 1.7M tokens/s per GPU on Vera Rubin NVL72.
  • Each user receives 387 tokens/s, according to vLLM.
  • The company reports 58x the throughput of GB300 NVL72.
  • Peak throughput reaches 2.52M tokens/s.
  • A single user reaches 1,810 tokens/s at batch size 1.

Rubin hardware drives the performance gains

The results come from adapting vLLM's DeepSeek-V4.1-Flash serving stack to Vera Rubin, which the company says introduces several optimization opportunities. The rack's GPUs have two memory domains, and streaming bandwidth-bound data from an SM's own domain is faster than accessing interleaved or remote memory.

vLLM rebuilt or reworked the model's MoE, attention, GEMM, mHC and indexer kernels for Rubin. The company also uses SM107's wider block-scaled matrix multiplication instructions, including K64 mixed-precision FP8×FP4 MMA and K128 FP4 MMA. According to vLLM, MegaMoE gains 8, 11% throughput at 8K or more tokens per rank, while a low-batch routed GEMM runs 12% faster.

The system also uses a rack-wide NVLink domain. vLLM says this allows TP8 to span two compute trays with almost no extra all-reduce cost, while prefill instances can read one another's HBM efficiently.

Kernel changes reduce prefill and decode time

For prefill, vLLM reports about 28% less time per DEP2 step at 32K tokens per rank. The changes include keeping the mHC residual stream local to each memory domain, improving attention pipelines and filtering experts before scoring them. MegaMoE tile sizing was also changed to account for the larger of local and data-parallel average token counts.

For decode, the company reports about 25% less time per DEP32 step compared with its first Rubin build. The changes include die-local expert weights and a Wide-EP dispatch design that reduces remote memory traffic. vLLM says the DEP2 decode forward is 10.1% faster at batch size 32 and 5.5% faster at batch size 168.

A specialized tensor-parallel decode path spreads the model over two trays with TP8. In that configuration, vLLM reports 1,810 tokens per second at the 90th percentile for a single user at batch size 1.

Rack-scale memory supports higher concurrency

Vera Rubin's speed allows the system to handle substantially higher concurrency, according to vLLM. The company pools rack memory through Dynamo workload-aware routing. Its Mooncake KV pool extends from DRAM to spare HBM, while Engram tables are sharded across GPU HBM.

vLLM reports a prefix hit rate of 96.5, 97% even at very high concurrency, and says routing alone balances the ranks. The benchmark is SemiAnalysis AgentX, which vLLM uses as a representative agentic serving workload and measures with cached input, uncached input and output tokens.

The results extend vLLM's earlier DeepSeek-V4.1-Flash work. The first post covered day-0 bring-up and kernel fusions, while the second scaled the model to a GB300 NVL72 rack using P/D disaggregation and Wide-EP. The Vera Rubin results show how the same serving stack changes when it can use Rubin's memory layout, FP4 tensor cores and rack-wide NVLink.

Related news

PyTorch team integrates SGLang with IBM Spyre in less than a week
For developersNovember 12, 2026, 15:45

PyTorch team integrates SGLang with IBM Spyre in less than a week

A two-person team integrated SGLang with IBM Spyre in less than a week, using AI coding agents to build an open source plugin of roughly 1,500 lines. PyTorch says SGLang…

Cline hits 11,840 tokens per second on one GPU with Nemotron 3.5 Super VL
For developersOctober 27, 2026, 20:12

Cline hits 11,840 tokens per second on one GPU with Nemotron 3.5 Super VL

Cline says it reached 11,840 aggregate output tokens per second on a single GB300 running NVIDIA's Nemotron 3.5 Super VL with 256 concurrent requests. The team published…

vLLM Hits 954K Tokens/s per GPU on DeepSeek-V4.1-Flash with GB300 NVL72
For developersOctober 23, 2026

vLLM Hits 954K Tokens/s per GPU on DeepSeek-V4.1-Flash with GB300 NVL72

The vLLM team reports that its engine, running with P/D disaggregation and Wide-EP on one NVIDIA GB300 NVL72 rack, serves DeepSeek-V4.1-Flash at 954K tokens/s per GPU on…