Artificial intelligence news.
Straight from the original announcements.

AWS brings managed NVIDIA Dynamo inference to SageMaker HyperPod

November 18, 2026, 00:33By the AI News Bites Editorial TeamSource: AWS
AWS brings managed NVIDIA Dynamo inference to SageMaker HyperPod
Image: AWS

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining the open source framework's distributed inference capabilities with HyperPod's infrastructure management. AWS says the integration is designed to help serve large language models across multiple nodes and separate prompt processing from token generation.

Key points
  • SageMaker HyperPod Inference now supports NVIDIA Dynamo.
  • Dynamo enables multi-node, aggregated and disaggregated LLM serving.
  • HyperPod manages GPU scheduling, networking, health monitoring and node replacement.
  • AWS reports 2.05 times the highest aggregated throughput for one Qwen3 benchmark.
  • Users can deploy through a DynamoGraphDeployment or InferenceEndpointConfig.

Managing large models and mixed workloads

AWS says serving large language models efficiently requires enough GPU memory and careful coordination between prefill, which processes prompts, and decode, which generates tokens. The company gives GLM-5.3 as an example: its FP8 weights exceed the 640 GB of GPU memory available on a single ml.p5.48xlarge instance, before the memory needed for KV cache and runtime overhead is included.

HyperPod Inference's NVIDIA Dynamo support enables multi-node deployment, disaggregated prefill and decode, and KV-aware routing. Dynamo works with inference engines including vLLM, SGLang and TensorRT-LLM. In disaggregated serving, separate prefill and decode worker pools can be scaled and tuned independently, while KV cache is transferred between them over Elastic Fabric Adapter networking.

Dynamo also supports aggregated serving, where prefill and decode run on the same worker. In both arrangements, workers can run on one instance or span multiple instances through tensor and pipeline parallelism. Its KV-aware router considers reusable prefix KV cache and active worker load. AWS says prefix reuse can reduce prompt recomputation and improve time to first token, while load awareness can help avoid overloading workers.

What HyperPod manages

The HyperPod Inference add-on installs the Dynamo custom resource definitions and the components needed to deploy and schedule the serving system. The HyperPod Inference Operator manages endpoint deployment through an InferenceEndpointConfig resource, while the Dynamo operator manages resources in a DynamoGraphDeployment, including frontends, model workers, replica counts and resource requirements.

Grove coordinates related groups of pods, including multi-node workers and their scaling relationships. Kubernetes AI Scheduler places pods on GPU resources using gang scheduling, so the pods required for a multi-node worker are scheduled together. AWS Deep Learning Containers provide the Dynamo runtime, and the add-on manages integrated component versions.

HyperPod also provides node, GPU and EFA health monitoring, manages node replacement and works with the orchestration components to recreate affected workers. Deployments configured through InferenceEndpointConfig with dynamoSpec can be invoked through SageMaker endpoints.

Two deployment paths and benchmark results

Existing Dynamo users can bring their own DynamoGraphDeployment and continue using upstream documentation, recipes and manifests. Teams using HyperPod Inference can instead configure an InferenceEndpointConfig with dynamoSpec, using the configuration format and OpenAI-compatible interface they already know. Both paths support single-instance and multi-node deployments. vLLM is the default inference engine, with SGLang and TensorRT-LLM also available.

In an AWS deployment example for GLM-5.3, aggregated serving runs one model-worker replica across two ml.p5.48xlarge instances, with two frontend replicas. A disaggregated example uses one prefill replica and two decode replicas, each spanning two instances, for a total of six instances.

AWS benchmarked Dynamo on HyperPod with two models. For Qwen3-235B-A22B-FP8, using TensorRT-LLM on two p5en.48xlarge instances with 4,000 input tokens and 200 output tokens, disaggregated serving reached 207 output tokens per second per GPU, compared with 101 for the highest measured aggregated result. AWS reports this as 2.05 times the aggregated throughput, with the tested points meeting its targets for P90 time to first token and median generation speed.

In a separate test using Qwen3.5-397B-A17B-FP8 with vLLM, a two-node configuration delivered 126.4 output tokens per second per GPU, retaining 99.1 percent of single-node throughput under an agentic workload with high prefix reuse. The result illustrates the multi-node serving approach described by AWS for models and deployments that require more than one instance.

Related news

AWS SageMaker AI Adds Monitoring for Inference Endpoint Drift
Products and toolsNovember 19, 2026, 22:28

AWS SageMaker AI Adds Monitoring for Inference Endpoint Drift

Amazon SageMaker AI can now identify latency drift, underused GPUs and missed serving optimizations on eligible generative AI inference endpoints. The monitoring is…

AWS adds GPU diagnostics and observability to SageMaker HyperPod
Products and toolsNovember 19, 2026, 17:41

AWS adds GPU diagnostics and observability to SageMaker HyperPod

AWS has introduced enhanced infrastructure observability in Amazon SageMaker HyperPod, bringing GPU diagnostics and profiling together for distributed training workloads…

PyTorch team integrates SGLang with IBM Spyre in less than a week
For developersNovember 12, 2026, 15:45

PyTorch team integrates SGLang with IBM Spyre in less than a week

A two-person team integrated SGLang with IBM Spyre in less than a week, using AI coding agents to build an open source plugin of roughly 1,500 lines. PyTorch says SGLang…