> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# NCCL configuration reference for CoreWeave fabrics

> Canonical NCCL and UCX environment variables for InfiniBand and RoCE backend fabrics on CoreWeave

This page is the canonical reference for NCCL and UCX environment variables on CoreWeave high-performance backend fabrics. It covers the variables you set for InfiniBand clusters, the variables that differ on RoCE (Spectrum-X) clusters such as GB300, and the log lines that look alarming but are normal. Use it as the lookup target when a training or inference job needs fabric-specific tuning.

For the Pod scheduling and Multus configuration that exposes RDMA devices to your containers, see [Use GPUDirect RDMA with InfiniBand](/products/networking/hpc-interconnect/use-gpudirect-rdma) and [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce). For node placement using fabric labels, see [Read IB and RoCE node labels for placement](/products/networking/hpc-interconnect/infiniband-roce-labels).

## What this reference is and is not

This page covers fabric-level NCCL and UCX settings: which interfaces NCCL uses, how to select host channel adapters, and how the transport differs between InfiniBand and RoCE on CoreWeave.

This page isn't a guide to collective algorithms, topology-aware ranks, or debugging distributed training hangs. Those belong to training documentation. This page also doesn't tune performance baselines or publish expected bandwidth numbers.

## InfiniBand cluster recipe

On InfiniBand clusters, NCCL uses its built-in InfiniBand verbs transport. Set the following environment variables in your Pod spec or Slurm job. These values match the examples in [Use GPUDirect RDMA with InfiniBand](/products/networking/hpc-interconnect/use-gpudirect-rdma).

```bash theme={"system"}
# Front-end interface for NCCL control and out-of-band traffic.
# Safe to set; selects the primary Pod interface, not the RDMA path.
export NCCL_SOCKET_IFNAME=eth0

# InfiniBand host channel adapters NCCL uses for GPUDirect RDMA collectives.
# "ibp" matches the InfiniBand device naming on CoreWeave IB clusters.
export NCCL_IB_HCA=ibp

# UCX devices and transports, if your stack uses UCX (for example, MPI workloads).
export UCX_NET_DEVICES=eth0
export UCX_TLS=tcp
```

| Variable             | Purpose                                                                                                                         |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `NCCL_SOCKET_IFNAME` | Selects the front-end interface for NCCL's TCP control and bootstrap traffic. Set to the primary Pod interface, usually `eth0`. |
| `NCCL_IB_HCA`        | Selects the RDMA host channel adapters for collectives. Use `ibp` on CoreWeave InfiniBand clusters.                             |
| `UCX_NET_DEVICES`    | Devices UCX uses, for stacks that run UCX alongside NCCL.                                                                       |
| `UCX_TLS`            | UCX transports to enable.                                                                                                       |

## RoCE (Spectrum-X) cluster recipe

RoCE clusters such as GB300 use Ethernet-based RDMA. Several NCCL variables differ from InfiniBand because the address family is IPv6 and the traffic class is set explicitly. The interface attachment (Multus NADs) and the full Pod spec live in [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce).

```bash theme={"system"}
# Front-end interface for control traffic, same as InfiniBand.
export NCCL_SOCKET_IFNAME=eth0

# RoCE RDMA devices. On GB300 RoCE clusters this is the "ibp" interface.
# On other RoCE clusters, match the interface names that Multus creates
# (for example, a comma-separated list such as net1,net2).
export NCCL_IB_HCA=ibp

# RoCE uses an IPv6 address family for RDMA, unlike InfiniBand.
export NCCL_IB_ADDR_FAMILY=AF_INET6

# Traffic class for RoCE. 96 is the value validated on GB300 RoCE.
export NCCL_IB_TC=96

# NCCL network plugin. On RoCE, the default built-in path is "none".
# The Spectrum-X plugin is only used for very large jobs.
export NCCL_NET_PLUGIN=none
```

### Variables that differ from InfiniBand

| Variable              | InfiniBand               | RoCE (Spectrum-X)                                     |
| --------------------- | ------------------------ | ----------------------------------------------------- |
| `NCCL_IB_ADDR_FAMILY` | Not required             | `AF_INET6` (RoCE addressing on CoreWeave is IPv6)     |
| `NCCL_IB_TC`          | Not required             | Set the RoCE traffic class, for example `96` on GB300 |
| `NCCL_NET_PLUGIN`     | Not required             | `none` for typical jobs                               |
| Interface attachment  | Devices exposed directly | Attached through Multus NADs (see RoCE guide)         |

## NCCL network plugin guidance

On CoreWeave RoCE fabrics, NCCL defaults to `NCCL_NET_PLUGIN=none`, which uses the built-in InfiniBand verbs path. This is correct for most jobs.

The NVIDIA Spectrum-X NCCL plugin provides NIC-level hardware-accelerated collectives, but it's only relevant for large jobs at the scale of roughly 10,000 or more GPUs. For typical multi-Node training and inference jobs, leave the plugin at `none`. If you believe your workload qualifies for the Spectrum-X plugin, contact your CoreWeave representative.

## NCCL\_IB\_RETRY\_CNT has a hard maximum of 7

`NCCL_IB_RETRY_CNT` is stored in a 3-bit register, so its maximum valid value is 7. Setting it to 8 or higher doesn't increase retries: the value wraps and is effectively treated as a small or zero count, which makes the fabric appear far less reliable than it is. If you want more retry tolerance, 7 is the ceiling. Don't set it above 7.

## Segment-size and channel knobs

Most workloads don't need to tune these. Change them only when you have a measured reason to, and change one at a time so you can attribute the effect.

| Variable                                    | What it controls                                                                          | When to consider changing it                                                                                                          |
| ------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `NCCL_BUFFSIZE`                             | Size of the per-connection staging buffer                                                 | Large-message collectives where the default underfills the link                                                                       |
| `NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS` | Lower and upper bounds on the number of parallel channels (CUDA blocks) NCCL uses         | Raise the minimum when throughput plateaus below the link rate on large messages; lower the maximum when NCCL consumes too many SMs   |
| `NCCL_P2P_NET_CHUNKSIZE`                    | Chunk size for point-to-point network transfers, which back collectives like `all_to_all` | Lower it for small-message latency; raise it for large-message point-to-point bandwidth (CoreWeave GB300 RoCE baselines use `524288`) |

Tuning these without a baseline measurement usually makes performance worse, not better. Capture a baseline first, change one knob, re-measure, and keep the change only if it helps.

## UCX coexistence and GPUDirect RDMA

NCCL and UCX use different RDMA data paths, so a Node where NCCL tests pass doesn't guarantee that UCX-based workloads (NIXL, MPI) work. The UCX transport settings, the `rdma-core` version requirement for GB300 RoCE, and the UCX diagnostics live in [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce#ucx-and-gpudirect-rdma-on-gb300-roce).

For MPI workloads (for example, Intel MPI or Ansys), the relevant variables differ from NCCL. A common configuration is:

```bash theme={"system"}
# MPI fabric selection for shared memory plus OpenFabrics interfaces.
export I_MPI_FABRICS=shm:ofi

# OpenFabrics provider for NVIDIA fabrics.
export FI_PROVIDER=mlx
```

## Log lines that look alarming but are normal

When you enable `NCCL_DEBUG=INFO`, you see verbose output. The following messages don't indicate a failure:

* `No Aggregation Manager sharp_am detected`. This means SHARP isn't available, so NCCL falls back to point-to-point collectives. SHARP is an InfiniBand-only switch-side reduction technology and isn't available on RoCE (Spectrum-X) fabrics. On a RoCE cluster, this fallback is expected and not an error.
* Transport fallback notices during initialization. NCCL probes available transports and reports the ones it selects. These are informational.

## Enable debug logging

To diagnose RDMA or transport issues, increase NCCL verbosity. Debug logging increases log volume, so disable it after you finish.

```bash theme={"system"}
# Verbose NCCL logging. Increases log size; remove after debugging.
export NCCL_DEBUG=INFO

# Optional: restrict verbose output to the network subsystem.
export NCCL_DEBUG_SUBSYS=NET
```

For the full list of NCCL variables, see [`NCCL_DEBUG` in the NVIDIA NCCL documentation](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-debug).

## When NCCL reports a vendor error

NCCL sometimes reports a vendor error code from the underlying RDMA stack. Use the onset pattern to distinguish a fabric fault from a configuration problem. A sudden, multi-Node onset points to a fabric-level event. A consistent failure on a single Pod points to a container or configuration problem such as a UCX or `rdma-core` mismatch. For the RoCE diagnostics and the link-health check, see [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce#ucx-and-gpudirect-rdma-on-gb300-roce) and [Read IB and RoCE node labels for placement](/products/networking/hpc-interconnect/infiniband-roce-labels).

## Related pages

* [Use GPUDirect RDMA with InfiniBand](/products/networking/hpc-interconnect/use-gpudirect-rdma): Pod scheduling and NCCL setup for IB.
* [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce): Multus attachment, UCX requirements, and NCCL setup for RoCE.
* [Read IB and RoCE node labels for placement](/products/networking/hpc-interconnect/infiniband-roce-labels): topology-aware placement and link-health checks.
* [About CoreWeave HPC interconnects](/products/networking/hpc-interconnect/about-hpc-interconnect): fabric overview.