How do I fix torch.distributed rendezvous and TCPStore failures?
Your torch.distributed job fails during startup or rendezvous with a c10d or TCPStore error such as sendBytes failed … …
How do I set up Slurm accounting for priority scheduling?
SUNK enables Slurm accounting by default: accounting.enabled is true and SUNK deploys a local slurmdbd with MySQL automa …
What do NCCL errors mean and how do I diagnose them?
Most NCCL errors aren’t NCCL bugs. The message you see usually names the layer where NCCL gave up, not the layer that fa …
What do training job exit codes 137, 139, and 143 mean?
Training job exit codes above 128 mean a signal killed the process, where the exit code is 128 + signal number. The most …
Why does Pyxis report File already exists after a job is preempted?
After a job is preempted or requeued, Pyxis can fail to start with ERROR File already exists if a leftover container dir …
Why is my node drained or cordoned?
A Slurm node enters drained state when it stops accepting new jobs. The drain reason (visible in scontrol show node or t …