Why are CUDA toolkit libraries missing in a SUNK container?

SUNK compute nodes provide NVIDIA GPU drivers to Pyxis containers. They don’t install the full CUDA toolkit. Libraries such as cuSPARSELt, cuDNN, cuBLAS, and NCCL must come from your container image or from a virtual environment you create on a compute node. Use an image that already bundles the toolkit, such as an NVIDIA NGC PyTorch image or one of the ML container images.

Symptoms

A job can detect GPUs (nvidia-smi succeeds) and still fail when Python or the application loads a toolkit library:

ImportError: libcusparseLt.so.0: cannot open shared object file

The same pattern appears for libcudnn, libcublas, or libnccl when those libraries aren’t in the image. This is a different problem from choosing an image whose CUDA version is newer than the node driver. For details, see Match the CUDA version to your GPU driver.

Confirm what the node injected

Inside the container, list the driver mounts and the CUDA libraries on the library path:

mount | grep -E 'nvidia|nv|cuda'
ls /usr/lib/x86_64-linux-gnu/libcuda* 2>/dev/null
ls /usr/lib/aarch64-linux-gnu/libcuda* 2>/dev/null

Driver libraries such as libcuda.so are present when the node injected the driver. Missing libcusparseLt, libcudnn, or similar files means the image doesn’t include the toolkit. GB200, GH200, and GB300 compute nodes are aarch64, because they use NVIDIA Grace CPUs. The login node on those clusters is typically x86_64, so packages installed from the login node can be built for the wrong architecture. Compare uname -m on the login node and inside a compute allocation, then install Python packages from the compute allocation so the wheels match the GPU nodes.

Use an image that includes the toolkit

Replace [TAG] with a tag from the image registry, and replace [GPU-PARTITION] with a GPU partition on your cluster.

srun --container-image=nvcr.io#nvidia/pytorch:[TAG] \
  --container-mounts=/mnt/home:/mnt/home \
  --gres=gpu:1 \
  --pty bash

You can also start from ghcr.io#coreweave/ml-containers/torch-extras:[TAG]. For details, see ML container images. If you install PyTorch and CUDA libraries into a virtual environment on shared storage instead of using a container, create that environment on a GPU compute node:

srun --nodes=1 --partition=[GPU-PARTITION] --gres=gpu:1 --pty bash

Workload Scheduling

​Symptoms

​Confirm what the node injected

​Use an image that includes the toolkit

​Related

Symptoms

Confirm what the node injected

Use an image that includes the toolkit

Related