Artificial intelligence news.
Straight from the original announcements.

AWS Adds SDC Detection and Recovery to SageMaker HyperPod

November 12, 2026, 00:30By the AI News Bites Editorial TeamSource: AWS
AWS Adds SDC Detection and Recovery to SageMaker HyperPod
Image: AWS

AWS has detailed a method for detecting and recovering from silent data corruption in Amazon SageMaker HyperPod training jobs. The approach adds three checks to a PyTorch DistributedDataParallel training loop, then maps suspicious GPU ranks to nodes that HyperPod can replace.

Key points
  • AWS adds three SDC checks to a PyTorch DDP training loop.
  • The checks inspect non-finite gradients, replayed micro-batches, and weight fingerprints.
  • A suspect rank can be mapped to a HyperPod node for containment and replacement.
  • Testing used 4 ml.p5.48xlarge instances with 32 NVIDIA H100 GPUs.
  • The method requires at least three GPU nodes and shared checkpoint storage.

Why silent corruption is difficult to find

Silent data corruption occurs when hardware such as a GPU returns a wrong but plausible number without raising an error. In distributed training, that value can enter gradients and weights across every replica, while the loss continues to look normal for hours.

AWS cites Meta's 54-day pre-training run for Llama 3 405B on up to 16K H100 GPUs. Meta attributed 6 of 419 unexpected job interruptions to silent data corruption, according to AWS. Those figures covered only cases that interrupted the job and were traced to SDC.

Existing HyperPod protections watch for reported GPU errors, run deep health checks, and replace hardware that fails. AWS says SDC can evade those measures because the hardware reports no fault. Common protections also cover different boundaries: ECC protects memory, transport checks verify that bytes arrived as sent, and checkpoint checksums can validate a poisoned but consistent checkpoint.

Three checks inspect the training computation

The first check looks for non-finite gradients, including NaN, Inf, and overflow in a GPU's forward or backward pass. It also detects NaN or Inf introduced by gradient all-reduce. This check can identify the problem on the step where it happens.

The second is a ring replay audit. On audit steps, each GPU recomputes a micro-batch from another node and compares the loss bit for bit and the gradients by fingerprint. AWS says this can catch finite, wrong values in the compute path when they change the loss or a gradient.

The third compares weight fingerprints after the optimizer update. It can identify a replica whose weights differ from a strict majority, helping detect corruption in the optimizer step or stored weights. AWS tested the checks using software-injected faults on a SageMaker HyperPod cluster with 4 ml.p5.48xlarge instances, or 32 NVIDIA H100 GPUs, and repeated every test on three instances, the minimum required by the checks.

From an alarm to node replacement

AWS describes a four-step response. The training loop detects the issue and names a suspect rank. On a SageMaker HyperPod cluster orchestrated by Amazon Elastic Kubernetes Service, the Kubernetes node name contains the instance ID, allowing the rank to be mapped to a node.

The job can then be stopped and the node cordoned while operators review GPU metrics in HyperPod observability dashboards. AWS recommends confirming ambiguous alarms, replacing the node after review, and resuming from a checkpoint saved before the suspect window. The company says replacing a node destroys its root and secondary volumes, including local NVMe, so checkpoints should be stored on shared storage such as Amazon FSx for Lustre or Amazon S3.

The method requires an EKS-orchestrated HyperPod cluster with at least three GPU nodes, a PyTorch DDP job whose replicas perform their own optimizer steps, and a data pipeline able to rebuild a rank's micro-batch for a given step. AWS says the full implementation is available in the accompanying GitHub repository. For users running large distributed training jobs, the proposal moves SDC detection into the computation itself, where standard hardware and transport checks may not see it.

Related news

Meta Introduces TLX, a Hardware-Native Compiler for GPU Workloads
For developersNovember 10, 2026, 17:59

Meta Introduces TLX, a Hardware-Native Compiler for GPU Workloads

Meta has detailed Triton Low-Level Extensions, or TLX, an extension to Triton designed to improve GPU performance in production machine learning workloads. TLX gives…

PyTorch serves Meta’s 816B Muse Spark Open on one TPU v7 host
For developersNovember 10, 2026, 17:45

PyTorch serves Meta’s 816B Muse Spark Open on one TPU v7 host

PyTorch says it can serve Meta’s 816B-parameter Muse Spark Open mixture-of-experts model on a single TPU v7 host containing four chips. The system uses SGLang-TorchTPU…

AWS SageMaker adds GPU partitioning for more efficient model hosting
Products and toolsNovember 6, 2026, 19:01

AWS SageMaker adds GPU partitioning for more efficient model hosting

Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU partitioning, allowing multiple models to share a physical GPU through hardware-isolated partitions.…