> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# NVSHMEM and GDRCopy support

> Use NVSHMEM and GDRCopy for high-performance GPU-to-GPU communication

With the release of `ncore-image v.2.10.1`, you can use NVSHMEM and GDRCopy in your container image for high-performance GPU-to-GPU communication. This page is for cluster operators and workload engineers who need to enable direct GPU-to-GPU memory access on SUNK or CKS nodes.

## Overview

NVSHMEM (NVIDIA SHMEM) and GDRCopy (GPU Direct RDMA Copy) enable direct memory access between GPUs without involving the CPU, reducing latency and increasing throughput for certain workloads.

The following sections describe how to obtain the supported image, what modifications it includes, and how to configure your containers to use NVSHMEM and GDRCopy.

## Access the image

To gain access to `ncore-image v.2.10.1`, [contact CoreWeave Support](/support).

## Image modifications

`ncore-image v.2.10.1` contains the following modifications to support NVSHMEM usage with `ibgda`. These modifications are applied at the node level and do not require changes from workload authors.

The image includes the following NVIDIA driver options:

* `nvidia.NVreg_EnableStreamMemOPs=1`
* `nvidia.NVreg_RegistryDwords="PeerMappingOverride=1;"`

The image also includes the GDRCopy driver `gdrdrv-dkms_2.5-1`.

## Use the image

When you use this image, you must complete the following steps so that your containers can access GDRCopy and NVSHMEM correctly.

### Enable the GDRCopy environment variable

Set the environment variable in the container to enable GDRCopy. This lets the container access `gdrdrv`:

```yaml theme={"system"}
env:
  - name: NVIDIA_GDRCOPY
    value: enabled
```

<Note>
  If you're using Slurm, this environment variable is already set.
</Note>

### Patch NVSHMEM ibgda

In NVSHMEM version 3.2.5, you must patch `ibgda` in one or more of your containers so that NVSHMEM recognizes the InfiniBand devices presented on CoreWeave nodes. Download NVSHMEM version 3.2.5.

In `src/modules/transport/ibgda/ibgda.cpp`, change line 3659 from `mlx5` to `ibp` to work in SUNK and CKS.

**Original code:**

```cpp highlight={1} theme={"system"}
if (!strstr(name, "mlx5")) {
    ftable.close_device(device->context);
    device->context = NULL;
    NVSHMEMI_WARN_PRINT("device %s is not enumerated as an mlx5 device. Skipping...\n",
                        name);
    continue;
}
```

**Modified code:**

```cpp highlight={1} theme={"system"}
if (!strstr(name, "ibp")) {
    ftable.close_device(device->context);
    device->context = NULL;
    NVSHMEMI_WARN_PRINT("device %s is not enumerated as an mlx5 device. Skipping...\n",
                        name);
    continue;
}
```

## Additional resources

* [Use GPUDirect RDMA with InfiniBand](/products/networking/hpc-interconnect/use-gpudirect-rdma)
* [Configure compute Nodes](/products/sunk/deploy_sunk/configure-compute-nodes)
* [Slurm images reference](/products/sunk/reference/slurm-images)