> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Why are CUDA toolkit libraries missing in a SUNK container?

SUNK compute nodes provide NVIDIA GPU drivers to Pyxis containers. They don't install the full CUDA toolkit. Libraries such as cuSPARSELt, cuDNN, cuBLAS, and NCCL must come from your container image or from a virtual environment you create on a compute node. Use an image that already bundles the toolkit, such as an NVIDIA NGC PyTorch image or one of the [ML container images](/products/sunk/discover_sunk/ml-containers).

## Symptoms

A job can detect GPUs (`nvidia-smi` succeeds) and still fail when Python or the application loads a toolkit library:

```text theme={"system"}
ImportError: libcusparseLt.so.0: cannot open shared object file
```

The same pattern appears for `libcudnn`, `libcublas`, or `libnccl` when those libraries aren't in the image.

This is a different problem from choosing an image whose CUDA version is newer than the node driver. For details, see [Match the CUDA version to your GPU driver](/products/sunk/discover_sunk/ml-containers#match-the-cuda-version-to-your-gpu-driver).

## Confirm what the node injected

Inside the container, list the driver mounts and the CUDA libraries on the library path:

```bash theme={"system"}
mount | grep -E 'nvidia|nv|cuda'
ls /usr/lib/x86_64-linux-gnu/libcuda* 2>/dev/null
ls /usr/lib/aarch64-linux-gnu/libcuda* 2>/dev/null
```

Driver libraries such as `libcuda.so` are present when the node injected the driver. Missing `libcusparseLt`, `libcudnn`, or similar files means the image doesn't include the toolkit.

GB200, GH200, and GB300 compute nodes are aarch64, because they use NVIDIA Grace CPUs. The login node on those clusters is typically x86\_64, so packages installed from the login node can be built for the wrong architecture. Compare `uname -m` on the login node and inside a compute allocation, then install Python packages from the compute allocation so the wheels match the GPU nodes.

## Use an image that includes the toolkit

Replace `[TAG]` with a tag from the image registry, and replace `[GPU-PARTITION]` with a GPU partition on your cluster.

```bash theme={"system"}
srun --container-image=nvcr.io#nvidia/pytorch:[TAG] \
  --container-mounts=/mnt/home:/mnt/home \
  --gres=gpu:1 \
  --pty bash
```

You can also start from `ghcr.io#coreweave/ml-containers/torch-extras:[TAG]`. For details, see [ML container images](/products/sunk/discover_sunk/ml-containers).

If you install PyTorch and CUDA libraries into a virtual environment on shared storage instead of using a container, create that environment on a GPU compute node:

```bash theme={"system"}
srun --nodes=1 --partition=[GPU-PARTITION] --gres=gpu:1 --pty bash
```

## Related

* [ML container images](/products/sunk/discover_sunk/ml-containers)
* [How do I run containers with Pyxis and Enroot in Slurm?](/support/sunk/articles/how-do-i-run-containers-with-pyxis-and-enroot-in-slurm)
* [Install software as containers using Pyxis and enroot](/products/sunk/tutorials/train-on-sunk/1-set-up-slurm-cluster#install-software-as-containers-using-pyxis-and-enroot)

***

<Badge stroke shape="pill" color="blue" size="md">[Workload Scheduling](/support/sunk/tags/workload-scheduling)</Badge>


## Related topics

- [ML container images](/products/sunk/discover_sunk/ml-containers.md)