PyTorch 2.15 rebuilds its NCCL backend so one failing GPU no longer sinks a training run

PyTorch 2.15 has shipped with a rewritten backend for Nvidia's NCCL communication library, built so that a single misbehaving GPU does not force a large training job to start over from scratch.

Tristan Rice, a PyTorch core developer and Meta engineer, said during a livestream on November 5 that the old networking interface had piled up technical debt. The legacy ProcessGroupNCCL could create dozens of communicators per process group, which led to occasional deadlocks and erratic performance.

The new backend creates one communicator per process group at initialization, removing interference between communicators. Teams that want more parallelism need to create more process groups. If a GPU crashes or slows down, an outside orchestrator can drop it and assign a replacement worker.

A flight recorder logs every operation as a series of snapshots that administrators can inspect after a failure. Before, stalls caused by timeouts or deadlocks often left only a generic error.

The legacy backend remains in 2.15 as a fallback. If the new one performs well, it is due to be removed in PyTorch 2.16, planned for early 2027.

The release also brings a new Python transport API for asynchronous one-way data movement, better parallel processing on Apple hardware, and support for Nvidia's Blackwell GPUs. It contains 3,117 commits from 583 contributors.

Source: The Register

Raon Community wrote this article from the facts reported in the original. The original belongs to its author; see the source for full details.