Skip to main content
In this guide, learn how to use GPUDirect Remote Direct Memory Access (RDMA) with InfiniBand at CoreWeave, and how to test it with the NVIDIA Collective Communications Library (NCCL). GPUDirect RDMA enables direct memory access between GPUs across Nodes over InfiniBand, which reduces latency and frees CPU cycles for distributed training and other multi-Node GPU workloads. This guide is for users running multi-Node GPU workloads on CKS or Slurm who need high-bandwidth, low-latency communication between GPUs.

​
Prerequisites

CoreWeave supports GPUDirect RDMA over InfiniBand for some GPU instance types. To use this feature, you must:
  • Select a Node Pool with InfiniBand support.
  • Install NCCL and the OpenFabrics Enterprise Distribution (OFED) driver in the Pod image.
  • Configure the Pods to use GPUDirect RDMA.

​
Select a Node Pool with InfiniBand support

To use GPUDirect RDMA, make sure the Node Pool has Nodes with InfiniBand, as shown in our list of GPU instance types. All Nodes with InfiniBand have the required kernel drivers pre-installed. CKS manages all the required driver and operator dependencies. To avoid Node instability, you should not install other driver management tools. Once you have a Node Pool that supports InfiniBand, the next section covers the Pod-level configuration that opts workloads into GPUDirect RDMA.

​
Configure the Pods

Configure your Pods to use GPUDirect RDMA over InfiniBand by following these steps:
  1. Set the value of spec.containers.resources.requests.rdma/ib to 1. This value doesn’t indicate the number of InfiniBand devices requested. It works as a boolean to schedule Pods onto servers with InfiniBand support. Kubernetes schedules resources through requests and limits. When you specify only limits, Kubernetes sets requests to the same amount as the limit. For more information, see Resource management for Pods and containers in the Kubernetes documentation. For a full YAML example showing how to set the rdma/ib value in the Pod spec for both requests and limits, see Kubernetes example.
  2. Configure the Pods to use GPUDirect RDMA by setting these environment variables:
    • NCCL_SOCKET_IFNAME: The network interface name to use for NCCL communication. Set this to the InfiniBand interface name.
    • NCCL_IB_HCA: The InfiniBand host channel adapter (HCA) to use for NCCL communication.
    • UCX_NET_DEVICES: The network devices to use for Unified Communication X (UCX) communication. Set this to the InfiniBand interface name.
For Kubernetes and Slurm examples, see Kubernetes example and Slurm example.
  1. Optional: Enable extended logging with the NCCL_DEBUG environment variable. To increase the verbosity of NCCL’s logging, set the NCCL_DEBUG environment variable to INFO for extra debug information. This helps diagnose issues with RDMA support, but it increases the log file size, so disable it when testing is complete. See NCCL_DEBUG in the NCCL documentation for more logging options.

​
Kubernetes example

When you deploy a Kubernetes Pod in the cluster, use the highlighted lines in this example to set the rdma/ib value in the Pod spec for both requests and limits, and to set the required environment variables.
Kubernetes example with debug logging
# [...]
spec:
  containers:
  - name: example
    resources:
      requests:
        cpu: 10
        memory: 10Gi
        rdma/ib: 1
        nvidia.com/gpu: 8
      limits:
        cpu: 10
        memory: 10Gi
        rdma/ib: 1
        nvidia.com/gpu: 8
  env:
    - name: NCCL_SOCKET_IFNAME
      value: eth0
    - name: NCCL_IB_HCA
      value: ibp
    - name: UCX_TLS
      value: tcp
    - name: UCX_NET_DEVICES
      value: eth0
    - name: NCCL_DEBUG
      value: INFO
# [...]
Setting NCCL_DEBUG to INFO enables extended logging. Remove it if you don’t need extended logging.

​
Slurm example

When you deploy a Slurm job, use the highlighted lines in this example to set the required environment variables. Remove NCCL_DEBUG unless you need extended logging.
Example Slurm sbatch script
#!/bin/bash

#SBATCH --partition h100
#SBATCH --nodes 16
#SBATCH --ntasks-per-node 8
#SBATCH --gpus-per-node 8
# [...] other SBATCH options, as needed

export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_HCA=ibp
export UCX_TLS=tcp
export UCX_NET_DEVICES=eth0
export NCCL_DEBUG=INFO # Remove NCCL_DEBUG unless debug logging

​
Test with NCCL

After you configure your Pods or Slurm jobs, verify that GPUDirect RDMA works as expected by running NCCL tests across multiple Nodes. CoreWeave provides several sample NCCL test jobs designed for use with Message Passing Interface (MPI) Operator or Slurm. To test GPUDirect RDMA support with InfiniBand, see the nccl-tests repository for the test jobs and instructions for running them.
Last modified on May 29, 2026