Artificial intelligence news.
Straight from the original announcements.

vLLM cuts Elastic EP scaling pause from 100 seconds to 0.82 seconds

October 19, 2026By the AI News Bites Editorial TeamSource: vLLM
vLLM cuts Elastic EP scaling pause from 100 seconds to 0.82 seconds
Image: vLLM

The vLLM project says it has reduced the serving pause during Elastic Expert Parallelism reconfiguration to under two seconds. In its DeepSeek V4 Pro benchmark, scaling from 16 to 20 GPUs now stops serving for 0.82 seconds, down from 100 seconds in the earlier Elastic EP version.

Key points
  • vLLM's Elastic EP now scales DeepSeek V4 Pro from 16 to 20 GB200 GPUs with 0.82 seconds of downtime, according to the project.
  • Scaling back from 20 to 16 GPUs pauses serving for 1.96 seconds, versus 56 seconds before.
  • A full restart for the 16 to 20 GPU change stopped serving for 197 seconds.
  • The gains come from asynchronous background preparation and NIXL EP, a library that expands EP communication in place.
  • Retained workers reuse their existing CUDA graphs after both scale-up and scale-down.

Why scaling used to be costly

According to vLLM, changing the number of GPUs in a service traditionally meant adding or removing whole model replicas, or restarting the engine with a different GPU count. As models grow, each replica needs more GPUs, so scaling by replica happens in ever larger steps.

The project gives a concrete example: restarting vLLM to scale DeepSeek V4 Pro from 16 to 20 GB200 GPUs stopped serving for 197 seconds, leaving expensive GPUs allocated without generating tokens.

The first Elastic EP release let vLLM add or remove GPUs in a running DEP deployment (data-parallel attention with expert-parallel MoE) without a restart. Serving still paused, though, while new workers were initialized, communication groups prepared, weights loaded, and kernel warmup and CUDA graph capture repeated. In the same benchmark that pause lasted 100 seconds when scaling from 16 to 20 GPUs and 56 seconds when scaling back to 16.

Two phases: background preparation and a short commit

vLLM reorganized reconfiguration into background preparation and a brief commit. Existing workers use separate background threads to prepare and warm up the target communication, transfer model weights, and synchronize execution information such as autotuning results with joining workers. Meanwhile they keep serving requests with their current communication groups and CUDA graphs.

Joining workers, in parallel, initialize, receive weights and prepare their communication and GPU execution. During the commit, vLLM briefly pauses its schedulers, activates the prepared topology and resumes serving. Preparation covers four tasks:

  • Initialize joining workers: start processes and allocate storage for the model and execution.
  • Prepare communication: set up target communication groups and expand the EP communicator.
  • Transfer model weights: move weights from existing workers over NVLink or RDMA.
  • Prepare execution: finish compilation, kernel warmup, tuning setup and CUDA graph construction on joining workers.

After the commit, asynchronous Expert Parallel Load Balancing (EPLB) lets new workers begin handling attention while existing workers handle expert computation. EPLB then redistributes experts across the expanded group in the background.

NIXL EP and preserved CUDA graphs

Asynchronous preparation alone was not enough. Earlier EP communication backends in vLLM used fixed membership, so adding or removing ranks required a new instance with its own buffers, memory registrations, IPC handles and RDMA Queue Pairs. Switching communicators also invalidated captured CUDA graphs. According to vLLM, even with asynchronous preparation, recapturing graphs during the commit caused pauses of 58 seconds on scale-up and 44 seconds on scale-down.

To address this, vLLM designed NIXL EP, an EP communication library built for Elastic EP. It adds or removes ranks in place, reusing the same NIXL agent, buffers and connections, and only joining peers need new connections and registrations. It also keeps captured EP dispatch and combine entries valid, so retained workers can reuse their CUDA graphs.

vLLM also changed its own code so the surrounding compute kernels keep stable arguments, launch configurations and tensor addresses. It reuses fused-MoE execution objects on retained workers, keeps addresses and layouts of graph-visible workspaces and dispatch outputs fixed, and preserves the storage of expert-routing tensors while updating mappings in place.

Joining workers prepare independently. NIXL EP lets each one mask every remote EP peer during warmup and graph capture, and vLLM no longer makes these passes wait for DP batch information from other workers. After capture, peers can be reactivated without invalidating the graphs.

Results and what they mean

On DeepSeek V4 Pro with GB200 GPUs, TP=1 and DP/EP sizes of 16 and 20, vLLM reports downtime of 0.82 seconds when scaling from 16 to 20 GPUs and 1.96 seconds when scaling back to 16. The project says this allows scaling up and down with almost no impact on serving performance.

For operators of large MoE deployments, the practical point is that capacity can be adjusted in smaller GPU increments without a long stop in token generation. These figures come from vLLM's own benchmark on a single model and hardware setup, and the source text provided does not include further measurements.

Related news

PyTorch team integrates SGLang with IBM Spyre in less than a week
For developersNovember 12, 2026, 15:45

PyTorch team integrates SGLang with IBM Spyre in less than a week

A two-person team integrated SGLang with IBM Spyre in less than a week, using AI coding agents to build an open source plugin of roughly 1,500 lines. PyTorch says SGLang…

vLLM reports 58x throughput for DeepSeek-V4.1-Flash on Vera Rubin
Chips and infrastructureNovember 12, 2026

vLLM reports 58x throughput for DeepSeek-V4.1-Flash on Vera Rubin

vLLM says it is serving DeepSeek-V4.1-Flash at 1.7M tokens per second per GPU on an NVIDIA Vera Rubin NVL72 rack, while delivering 387 tokens per second to each user.…

vLLM Adds Day-One Support for Thinking Machines’ Inkling 2 Small
ModelsNovember 12, 2026

vLLM Adds Day-One Support for Thinking Machines’ Inkling 2 Small

vLLM has added day-zero support for Thinking Machines’ Inkling 2 Small, a multimodal mixture-of-experts model that accepts text and images and produces text. The…