> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Why does my job stop responding in NCCL communicator init after a previous crash?

If the job stops responding during NCCL communicator initialization specifically after a previous run crashed, the cause is usually a corrupted on-disk compile cache, not NCCL or the fabric. Clear the PyTorch Inductor and Triton caches, or disable the Inductor caches for the next run.

## What you see

Your job shows all of the following:

* It stops responding during NCCL communicator initialization.
* A previous run of the job crashed.
* The previous run's log has a `destroy_process_group ... was not called` warning. That's the strong signal. It means the previous process exited without tearing down cleanly.

## Why it happens

A previous run that died without cleaning up can leave both the PyTorch Inductor and Triton caches in a bad state.

The two caches default to different storage, which determines how far the corruption spreads:

* **Inductor** defaults to a `torchinductor_[USERNAME]` directory under the system temporary directory, normally `/tmp/torchinductor_$USER`. That's node-local, so corruption stays on the nodes the crashed run used.
* **Triton**'s own default is `$HOME/.triton/cache`, on shared storage. When Inductor compiles the kernels and `TRITON_CACHE_DIR` isn't set, Inductor points Triton at a `triton/` directory inside the Inductor cache, so those kernels follow the Inductor cache. Triton kernels compiled outside Inductor, such as custom kernels in a library, still use the home-directory cache, and corruption there follows your jobs to any node.

> [!NOTE]
>
> On SUNK 8.1 and later, Slurm gives each job outside a container its own `/tmp` and deletes it when the job ends, unless your administrator turned that off. An Inductor cache at the default path doesn't survive into the next job.

Two things move the Inductor cache, and both change how the corruption behaves. Setting `TORCHINDUCTOR_CACHE_DIR` to a path on shared storage is the obvious one, and it makes the Inductor cache behave like the Triton case. The subtler one is `TMPDIR`: Inductor resolves its default through Python's temporary directory, which honors `TMPDIR`, so wherever `TMPDIR` points is where the cache lands. Check `TMPDIR` before you assume the cache is on the node.

Under Pyxis or Enroot, `/tmp` inside the container is `tmpfs` (RAM) rather than the node's NVMe, so an Inductor cache left at the default doesn't survive the container at all and can't be the stale cache you're chasing. It also consumes memory while the job runs. Jobs that set `TMPDIR` to a fixed NVMe-backed path keep the cache across runs on that node. The per-job path that [Node-local storage and `/tmp` on Slurm nodes](/products/sunk/manage_sunk/node-local-storage-and-tmp) recommends is new for each job ID, so a new job doesn't reuse a cache that an earlier job left there. If your job runs in a container and doesn't set `TMPDIR`, suspect a cache on shared storage instead, such as the Triton cache in your home directory.

## Clear or disable the caches

Fix it by clearing the caches or disabling them for the next run:

```bash theme={"system"}
# Clear the on-disk compile caches left by a crashed run.
# Safe to delete; they are regenerated on next launch.
# Resolves both the explicit overrides and a relocated TMPDIR.
rm -rf "${TORCHINDUCTOR_CACHE_DIR:-${TMPDIR:-/tmp}/torchinductor_$USER}" "${TRITON_CACHE_DIR:-$HOME/.triton/cache}"
```

```bash theme={"system"}
# Or disable the Inductor caches so a corrupt cache cannot be reused.
export TORCHINDUCTOR_FORCE_DISABLE_CACHES=1
```

Don't rely on a node reboot to fix this. Replacing or rebooting a node clears a node-local Inductor cache, but it leaves caches on shared storage untouched, such as the Triton cache in your home directory, so corruption there follows the job to its next node. Clear the caches explicitly instead. To prevent cross-job corruption, point the cache paths at a per-job-id directory so a crash in one job can't corrupt the cache another job reuses.

## Related pages

* [What do NCCL errors mean and how do I diagnose them?](/support/sunk/articles/what-do-nccl-errors-mean): NCCL error strings and timeouts.
* [How do I fix torch.distributed rendezvous and TCPStore failures?](/support/sunk/articles/how-do-i-fix-torch-distributed-rendezvous-failures): failures during rendezvous, before NCCL init.
* [Node-local storage and `/tmp` on Slurm nodes](/products/sunk/manage_sunk/node-local-storage-and-tmp): where `TMPDIR` should point.

[Server Errors](/support/sunk/tags/server-errors)

[Storage](/support/sunk/tags/storage)