Keep Distributed Training Running with Automated Workload Recovery

NVIDIA has introduced NVIDIA Recovery, an open-source, workload-level service within the DSX OS portfolio that automates cluster-wide failure recovery for distributed AI training. The tool detects disruptions such as NCCL timeouts and GPU hangs, uses attribution to identify faulty resources, and coordinates isolation and checkpoint restarts across workloads. In production monitoring on a training cluster, the service recorded thousands of anomaly detections with mean detection and recovery-action latencies of 69 and 94 seconds. Open article →