Skip to main content
If the job stops responding during NCCL communicator initialization specifically after a previous run crashed, the cause is usually a corrupted on-disk compile cache, not NCCL or the fabric. Clear the PyTorch Inductor and Triton caches, or disable the Inductor caches for the next run.

​
What you see

Your job shows all of the following:
  • It stops responding during NCCL communicator initialization.
  • A previous run of the job crashed.
  • The previous run’s log has a destroy_process_group ... was not called warning. That’s the strong signal. It means the previous process exited without tearing down cleanly.

​
Why it happens

A previous run that died without cleaning up can leave both the PyTorch Inductor and Triton caches in a bad state. The two caches default to different storage, which determines how far the corruption spreads:
  • Inductor defaults to a torchinductor_[USERNAME] directory under the system temporary directory, normally /tmp/torchinductor_$USER. That’s node-local, so corruption stays on the nodes the crashed run used.
  • Triton’s own default is $HOME/.triton/cache, on shared storage. When Inductor compiles the kernels and TRITON_CACHE_DIR isn’t set, Inductor points Triton at a triton/ directory inside the Inductor cache, so those kernels follow the Inductor cache. Triton kernels compiled outside Inductor, such as custom kernels in a library, still use the home-directory cache, and corruption there follows your jobs to any node.
On SUNK 8.1 and later, Slurm gives each job outside a container its own /tmp and deletes it when the job ends, unless your administrator turned that off. An Inductor cache at the default path doesn’t survive into the next job.
Two things move the Inductor cache, and both change how the corruption behaves. Setting TORCHINDUCTOR_CACHE_DIR to a path on shared storage is the obvious one, and it makes the Inductor cache behave like the Triton case. The subtler one is TMPDIR: Inductor resolves its default through Python’s temporary directory, which honors TMPDIR, so wherever TMPDIR points is where the cache lands. Check TMPDIR before you assume the cache is on the node. Under Pyxis or Enroot, /tmp inside the container is tmpfs (RAM) rather than the node’s NVMe, so an Inductor cache left at the default doesn’t survive the container at all and can’t be the stale cache you’re chasing. It also consumes memory while the job runs. Jobs that set TMPDIR to a fixed NVMe-backed path keep the cache across runs on that node. The per-job path that Node-local storage and /tmp on Slurm nodes recommends is new for each job ID, so a new job doesn’t reuse a cache that an earlier job left there. If your job runs in a container and doesn’t set TMPDIR, suspect a cache on shared storage instead, such as the Triton cache in your home directory.

​
Clear or disable the caches

Fix it by clearing the caches or disabling them for the next run:
Don’t rely on a node reboot to fix this. Replacing or rebooting a node clears a node-local Inductor cache, but it leaves caches on shared storage untouched, such as the Triton cache in your home directory, so corruption there follows the job to its next node. Clear the caches explicitly instead. To prevent cross-job corruption, point the cache paths at a per-job-id directory so a crash in one job can’t corrupt the cache another job reuses. Server Errors Storage
Last modified on October 13, 2026