AWS adds workload recovery and deeper checks to SageMaker HyperPod

AWS has introduced workload recovery and expanded infrastructure checks for Amazon SageMaker HyperPod, targeting training failures that can escape conventional node health tests. The new capabilities are designed to detect incorrect GPU results, Kubernetes failures and stalled Slurm jobs sooner.
- AWS added workload recovery and expanded infrastructure checks to Amazon SageMaker HyperPod.
- MultiNodeTrainingValidation tests whether GPUs train correctly across one or more instances.
- A Kubernetes monitoring agent reports failed pods and nodes involved in training failures.
- Hung job detection can cancel stalled Slurm jobs so they can be requeued.
- AWS estimates a sample stalled job cost could fall from about $16,900 to about $530.
Why node health checks can miss training failures
SageMaker HyperPod already uses a health monitoring agent to watch for GPU, Elastic Fabric Adapter and kernel errors. Its deep health checks test hardware during provisioning, while automatic node recovery can reboot or replace faulty hardware.
AWS says those checks assess one node at a time. They do not establish whether GPUs are producing correct training results, whether a job is still making progress, or whether Kubernetes components required by a job are functioning correctly.
That leaves several failure modes. A GPU may pass stress tests but produce incorrect values, with NaN or infinite loss appearing only during a training run. An NCCL timeout, stalled data pipeline or deadlock can leave a job running in Slurm while it continues to hold GPUs and memory. Kubernetes failures can also involve crash-looping device plugins, missing DaemonSets or tainted nodes.
AWS cites Meta’s report that silent data corruption accounted for 6 of the 419 unexpected interruptions during its 54-day Llama 3 405B pre-training run.
Three capabilities target different failure points
MultiNodeTrainingValidation is a new deep health check for HyperPod clusters orchestrated by Amazon Elastic Kubernetes Service, or Amazon EKS. It runs a fixed workload first on each instance and then across instances of the same type. The test fails an instance if its loss becomes NaN or infinite, or if training does not finish within the time limit.
The check uses a short Tiny Megatron test on each instance and a Megatron test across instances communicating over EFA. When an instance fails, HyperPod labels it as failed and, when automatic node recovery is enabled, reboots or replaces it. The check can run when instances are provisioned or updated, and on demand. AWS says it is available on EKS clusters with supported EFA-enabled NVIDIA GPU instances.
The HyperPod Kubernetes Resource Monitoring Agent continuously watches the Kubernetes layer. It detects issues including node condition changes, pod and DaemonSet failures, and failed training jobs. When a job fails, it reports the involved pods and nodes through Kubernetes events and Amazon CloudWatch logs. For jobs that opt in, it also reports NCCL communication state.
Hung job detection runs alongside Slurm jobs and looks for a lack of progress through job output, GPU activity or log patterns defined by the user. It writes an event to CloudWatch and can cancel the job so Slurm requeues it. On EKS, HyperPod already detects hung jobs run through the HyperPod training operator, while Ray Train jobs receive hang detection by default, according to AWS.
Reducing wasted compute and improving recovery
AWS describes the three features as addressing different forms of goodput. MultiNodeTrainingValidation can prevent jobs from starting on hardware that produces invalid output. Kubernetes monitoring can help identify the pods and nodes behind a failed job, while hung job detection can reduce the time that resources remain tied up by a stalled run.
In an AWS example, a pre-training job uses 32 ml.p5.48xlarge instances, or 256 NVIDIA H100 GPUs. If the job hangs at 1:00 AM and is not noticed until 9:00 AM, AWS says the incident wastes 256 instance-hours or 2,048 GPU-hours. At the SageMaker HyperPod on-demand price of approximately $66 per instance-hour in US East (N. Virginia), the cost is about $16,900.
With a 15-minute hung job detection rule, HyperPod cancels the stalled job and Slurm requeues it. AWS estimates that the hang would cost about $530 instead, saving approximately $16,400. The example illustrates the operational goal of the update: identify failures earlier, release or replace affected resources, and restart workloads with less manual intervention.


