Skip to main content
A segfault during checkpoint saving followed by an NCCL collective timeout points to a driver-level problem, not a fabric fault. On GB200 nodes with CDMM enabled, call torch.cuda.synchronize() before checkpointing, or move checkpoint writes to object storage.

​
What you see

The signature on GB200 nodes with CDMM enabled is specific:
  • The segfault lands in crc32_16bytes, a CRC-32 checksum routine that runs during the checkpoint save.
  • The segfault appears in the log before the NCCL timeout. The timeout is the surviving ranks waiting on the ranks the segfault killed.
  • It’s scale-dependent, more likely at eight racks or more, and it can follow the workload to another cloud provider after an upgrade from driver 570 to 580.

​
Why it happens

The likely cause is Coherent Driver-Based Memory Management (CDMM), a memory mode that NVIDIA introduced in the R580 driver for GB200 and that CoreWeave’s GB200 node images enable by default. CDMM changes the timing of GPU-to-CPU transfers, and the leading theory is that the checkpoint code reads host memory before the transfer finishes. Driver 570 predates CDMM and isn’t affected.

​
Work around the checkpoint segfault

If you see a segfault before an NCCL timeout in a checkpointing job, don’t investigate the IB fabric, NVLink, or NCCL configuration. They’ll all read healthy. Work these options instead, lightest first:
  1. Call torch.cuda.synchronize() before checkpointing, so all in-flight GPU work completes before the save begins.
  2. Move checkpoint writes to object storage, which changes the data path.
If neither works, contact support with the stack trace and the node’s driver and ncore versions. Server Errors Nodes
Last modified on October 13, 2026