How do I fix torch.distributed rendezvous and TCPStore failures?
Your torch.distributed job fails during startup or rendezvous with a c10d or TCPStore error such as sendBytes failed … …
How do I set up Slurm accounting for priority scheduling?
SUNK enables Slurm accounting by default: accounting.enabled is true and SUNK deploys a local slurmdbd with MySQL automa …
What do NCCL errors mean and how do I diagnose them?
Most NCCL errors aren’t NCCL bugs. The message you see usually names the layer where NCCL gave up, not the layer that fa …
What do training job exit codes 137, 139, and 143 mean?
Training job exit codes above 128 mean a signal killed the process, where the exit code is 128 + signal number. The most …
Why does Pyxis report File already exists after a job is preempted?
After a job is preempted or requeued, Pyxis can fail to start with ERROR File already exists if a leftover container dir …
Why does my GB200 job segfault during checkpointing?
A segfault during checkpoint saving followed by an NCCL collective timeout points to a driver-level problem, not a fabri …
Why does my job stop responding in NCCL communicator init after a previous crash?
If the job stops responding during NCCL communicator initialization specifically after a previous run crashed, the cause …
Why does my node keep draining for a missing InfiniBand device?
A node that drains with NHC: check_file_test: -e /dev/infiniband returned 1 has lost its InfiniBand device mount inside …
Why is my node drained or cordoned?
A Slurm node enters drained state when it stops accepting new jobs. The drain reason (visible in scontrol show node or t …