> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Why does my GB200 job segfault during checkpointing?

A segfault during checkpoint saving followed by an NCCL collective timeout points to a driver-level problem, not a fabric fault. On GB200 nodes with CDMM enabled, call `torch.cuda.synchronize()` before checkpointing, or move checkpoint writes to object storage.

## What you see

The signature on GB200 nodes with CDMM enabled is specific:

* The segfault lands in `crc32_16bytes`, a CRC-32 checksum routine that runs during the checkpoint save.
* The segfault appears in the log **before** the NCCL timeout. The timeout is the surviving ranks waiting on the ranks the segfault killed.
* It's scale-dependent, more likely at eight racks or more, and it can follow the workload to another cloud provider after an upgrade from driver 570 to 580.

## Why it happens

The likely cause is Coherent Driver-Based Memory Management (CDMM), a memory mode that NVIDIA introduced in the R580 driver for GB200 and that CoreWeave's GB200 node images enable by default. CDMM changes the timing of GPU-to-CPU transfers, and the leading theory is that the checkpoint code reads host memory before the transfer finishes. Driver 570 predates CDMM and isn't affected.

## Work around the checkpoint segfault

If you see a segfault before an NCCL timeout in a checkpointing job, don't investigate the IB fabric, NVLink, or NCCL configuration. They'll all read healthy. Work these options instead, lightest first:

1. Call `torch.cuda.synchronize()` before checkpointing, so all in-flight GPU work completes before the save begins.
2. Move checkpoint writes to object storage, which changes the data path.

If neither works, [contact support](/support/contact) with the stack trace and the node's driver and ncore versions.

## Related pages

* [What do NCCL errors mean and how do I diagnose them?](/support/sunk/articles/what-do-nccl-errors-mean): the WorkNCCL timeout diagnostic order.
* [What do training job exit codes 137, 139, and 143 mean?](/support/sunk/articles/what-do-training-job-exit-codes-mean): telling a driver segfault from an application or hardware one.
* [Update the GPU driver](/products/cks/nodes/gpu-driver-management/update-gpu-driver)

[Server Errors](/support/sunk/tags/server-errors)

[Nodes](/support/sunk/tags/nodes)