NCCL configuration reference for CoreWeave fabrics

This page is the canonical reference for NCCL and UCX environment variables on CoreWeave high-performance backend fabrics. It covers the variables you set for InfiniBand clusters, the variables that differ on RoCE (Spectrum-X) clusters such as GB300, how NCCL chooses a network plugin, and the log lines that look alarming but are normal. Use it as the lookup target when a training or inference job needs fabric-specific tuning. For the Pod scheduling and Multus configuration that exposes RDMA devices to your containers, see Use GPUDirect RDMA with InfiniBand and Use GPUDirect RDMA with RoCE. For node placement using fabric labels, see Read IB and RoCE node labels for placement.

What this reference is and is not

This page covers fabric-level NCCL and UCX settings: which interfaces NCCL uses, how to select host channel adapters, how NCCL chooses a network plugin, and how the transport differs between InfiniBand and RoCE on CoreWeave. This page isn’t a guide to collective algorithms, topology-aware ranks, or debugging distributed training hangs. Those belong to training documentation. This page also doesn’t tune performance baselines or publish expected bandwidth numbers.

InfiniBand cluster recipe

On InfiniBand clusters, set the following environment variables in your Pod spec or Slurm job. The settings select NCCL’s built-in InfiniBand verbs transport. The values match the examples in Use GPUDirect RDMA with InfiniBand and the Slurm reference jobs in the nccl-tests repository. Some jobs in that repository point UCX_NET_DEVICES at the InfiniBand devices instead, for stacks that run UCX over RDMA.

# Front-end interface for NCCL control and out-of-band traffic.
# Safe to set; selects the primary Pod interface, not the RDMA path.
export NCCL_SOCKET_IFNAME=eth0

# InfiniBand host channel adapters NCCL uses for GPUDirect RDMA collectives.
# "ibp" matches the InfiniBand device naming on CoreWeave IB clusters.
export NCCL_IB_HCA=ibp

# Use NCCL's built-in InfiniBand verbs transport. Omit this line if you're
# enabling SHARP, which needs the HPC-X plugin. Every rank must match.
# See "NCCL network plugin guidance".
export NCCL_NET_PLUGIN=none

# UCX devices and transports, if your stack uses UCX (for example, MPI workloads).
# NCCL doesn't use UCX. These values keep UCX off the InfiniBand devices.
export UCX_NET_DEVICES=eth0
export UCX_TLS=tcp

Variable	Purpose
`NCCL_SOCKET_IFNAME`	Selects the front-end interface for NCCL’s TCP control and bootstrap traffic. Set to the primary Pod interface, usually `eth0`.
`NCCL_IB_HCA`	Selects the RDMA host channel adapters for collectives. Use `ibp` on CoreWeave InfiniBand clusters.
`NCCL_NET_PLUGIN`	Selects the NCCL network plugin. Set to `none` to use the built-in InfiniBand verbs transport, or leave it unset to load the HPC-X network plugin, which SHARP requires. Every rank in a job must use the same value.
`UCX_NET_DEVICES`	Devices UCX uses, for stacks that run UCX alongside NCCL.
`UCX_TLS`	Transports UCX is allowed to use, for stacks that run UCX alongside NCCL.

RoCE (Spectrum-X) cluster recipe

RoCE clusters such as GB300 use Ethernet-based RDMA. Several NCCL variables differ from InfiniBand because the address family is IPv6 and the traffic class is set explicitly. The interface attachment (Multus NADs) and the full Pod spec live in Use GPUDirect RDMA with RoCE.

# Front-end interface for control traffic, same as InfiniBand.
export NCCL_SOCKET_IFNAME=eth0

# RoCE RDMA devices. On GB300 RoCE clusters this is the "ibp" interface.
# On other RoCE clusters, match the interface names that Multus creates
# (for example, a comma-separated list such as net1,net2).
export NCCL_IB_HCA=ibp

# RoCE uses an IPv6 address family for RDMA, unlike InfiniBand.
export NCCL_IB_ADDR_FAMILY=AF_INET6

# Traffic class for RoCE. 96 is the value validated on GB300 RoCE.
export NCCL_IB_TC=96

# Load the Spectrum-X network plugin, which needs HPC-X 2.50 or newer.
# Do not set NCCL_NET_PLUGIN. Leave it unset so NCCL finds the plugin.
# Every rank must match. See "NCCL network plugin guidance".
export LD_LIBRARY_PATH=/opt/hpcx/nccl_spectrum-x_plugin/lib:$LD_LIBRARY_PATH

Variables that differ from InfiniBand

Variable	InfiniBand	RoCE (Spectrum-X)
`NCCL_IB_ADDR_FAMILY`	Not required	`AF_INET6` (RoCE addressing on CoreWeave is IPv6)
`NCCL_IB_TC`	Not required	Set the RoCE traffic class, for example `96` on GB300
`NCCL_NET_PLUGIN`	`none` for typical jobs	Leave unset so NCCL loads the Spectrum-X plugin
`LD_LIBRARY_PATH`	No change needed	Prepend `/opt/hpcx/nccl_spectrum-x_plugin/lib` so NCCL finds the plugin
Interface attachment	Devices exposed directly	Attached through Multus NADs (see RoCE guide)

For plugin selection and SHARP requirements, see NCCL network plugin guidance.

NCCL network plugin guidance

NCCL_NET_PLUGIN selects the network plugin NCCL uses for communication between Nodes. A network plugin replaces NCCL’s built-in transport, so every rank in a job must agree on the setting. The following sections describe how NCCL chooses a plugin, what the InfiniBand and RoCE plugins provide, and how to decide whether to use the InfiniBand plugin.

How NCCL chooses a network plugin

When NCCL_NET_PLUGIN is unset, NCCL searches the library path for libnccl-net.so and uses that plugin if it finds one. NCCL falls back to its built-in transport when it finds no plugin it can use. Setting NCCL_NET_PLUGIN=none skips the search and forces the built-in transport. Many CoreWeave compute environments include NVIDIA HPC-X, which installs the nccl-rdma-sharp-plugins build of libnccl-net.so. CoreWeave SUNK compute container images and the images built from the nccl-tests repository both bundle it. Where HPC-X is on the library path, NCCL loads the plugin unless you set NCCL_NET_PLUGIN=none. This is why the reference jobs in that repository set the variable explicitly instead of relying on a default. Because the plugin ships in the container image rather than on the Node, whether a rank loads it depends on the image that rank runs.

What the InfiniBand network plugin provides

The nccl-rdma-sharp-plugins network plugin is NVIDIA’s plugin as bundled with HPC-X. It isn’t a CoreWeave fork, and CoreWeave applies no source patches to it. The plugin replaces NCCL’s built-in transport between Nodes and can also use CollNet and SHARP for the collectives that support them. NCCL’s built-in InfiniBand verbs transport, which NCCL_NET_PLUGIN=none selects, has no SHARP path. Loading the plugin doesn’t enable SHARP on its own. NCCL_COLLNET_ENABLE defaults to 0, and CoreWeave doesn’t enable SHARP on its fabrics by default, so jobs fall back to point-to-point collectives. Loading the plugin and setting NCCL_COLLNET_ENABLE=1 still aren’t sufficient. SHARP also has to be enabled on the fabric itself, which CoreWeave does for you. To find out whether SHARP is enabled on your cluster, or to ask for it, contact your CoreWeave representative. The No Aggregation Manager sharp_am detected line reports that fallback on InfiniBand as well as RoCE. For more information, see Log lines that look alarming but are normal. Plugin versions differ between images, and the network plugin API version the plugin exposes must match what your NCCL build expects. A plugin your NCCL build can’t use isn’t a startup failure. NCCL logs that the external network plugin is unsupported, unloads it, and falls back to its built-in transport. A rank that falls back this way while other ranks keep the plugin creates the mixed configuration described in Every rank must load the same network plugin. Instead of assuming a single fleet-wide version, check your container image tag to find the version your job uses.

Every rank must load the same network plugin

Every rank in a job must load the same network plugin. The plugin transport and NCCL’s built-in transport don’t interoperate, so you can’t run some ranks on one and some ranks on the other.

A mixed configuration, where some ranks use a plugin transport and other ranks use NCCL’s built-in transport, isn’t supported. It can cause faults during collectives, including memory faults during broadcast.

Two configurations are uniform, and either one is supported:

Set NCCL_NET_PLUGIN=none on every rank. All ranks use the built-in InfiniBand verbs transport. This is the recommended setting on InfiniBand unless you’re deliberately enabling SHARP.
Run every rank from the same container image and leave NCCL_NET_PLUGIN unset. All ranks load the same plugin from that image.

A common cause of a mixed configuration is that different ranks run different container images in one job. Building your own image on top of a CoreWeave base image can also change whether HPC-X is on the library path, for example when the new image replaces LD_LIBRARY_PATH instead of appending to it. A rank that used to load the plugin might stop loading it. To confirm that ranks agree, set NCCL_DEBUG=INFO and compare the Using network line across ranks. Every rank must report the same network name. The NET/Plugin lines in the same output show which plugin libraries each rank searched for, and report the ones it couldn’t find.

Decide whether to use the InfiniBand plugin

Use NCCL_NET_PLUGIN=none unless you’re deliberately enabling SHARP. NCCL’s built-in transport performs well on CoreWeave InfiniBand fabrics, and CoreWeave publishes no custom NCCL tuner. CoreWeave publishes no GPU count at which the HPC-X plugin becomes faster than the built-in transport. If you want to compare without enabling SHARP, measure with the paired reference jobs in the nccl-tests repository. Those paired jobs differ as follows:

The standard jobs set NCCL_NET_PLUGIN=none.
The SHARP variants, whose filenames end in -sharp, leave the plugin loaded and set NCCL_COLLNET_ENABLE=1 and SHARP_COLL_ENABLE_PCI_RELAXED_ORDERING=1.

Compare a pair while keeping the image, Nodes, placement, message sizes, and iteration count identical. If the plugin doesn’t produce a repeatable improvement in your workload, use NCCL_NET_PLUGIN=none.

The RoCE Spectrum-X plugin

The NVIDIA Spectrum-X NCCL plugin provides NIC-level hardware-accelerated collectives. Use it on RoCE clusters for jobs of any size. The plugin requires HPC-X 2.50 or newer, and CoreWeave images that bundle HPC-X ship it at /opt/hpcx/nccl_spectrum-x_plugin. To load this plugin, prepend /opt/hpcx/nccl_spectrum-x_plugin/lib to LD_LIBRARY_PATH and leave NCCL_NET_PLUGIN unset, as shown in the RoCE cluster recipe on this page. As on InfiniBand, every rank in a job must load the same network plugin. If your image ships an HPC-X version older than 2.50, upgrade the image or contact your CoreWeave representative. SHARP is an InfiniBand-only technology and isn’t available on RoCE fabrics.

NCCL_IB_RETRY_CNT has a hard maximum of 7

NCCL_IB_RETRY_CNT is stored in a 3-bit register, so its maximum valid value is 7. Setting it to 8 or higher doesn’t increase retries: the value wraps and is effectively treated as a small or zero count, which makes the fabric appear far less reliable than it is. If you want more retry tolerance, 7 is the ceiling. Don’t set it above 7.

Segment-size and channel knobs

Most workloads don’t need to tune these. Change them only when you have a measured reason to, and change one at a time so you can attribute the effect.

Variable	What it controls	When to consider changing it
`NCCL_BUFFSIZE`	Size of the per-connection staging buffer	Large-message collectives where the default underfills the link
`NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`	Lower and upper bounds on the number of parallel channels (CUDA blocks) NCCL uses	Raise the minimum when throughput plateaus below the link rate on large messages; lower the maximum when NCCL consumes too many SMs
`NCCL_P2P_NET_CHUNKSIZE`	Chunk size for point-to-point network transfers, which back collectives like `all_to_all`	Lower it for small-message latency; raise it for large-message point-to-point bandwidth (CoreWeave GB300 RoCE baselines use `524288`)

Tuning these without a baseline measurement usually makes performance worse, not better. Capture a baseline first, change one knob, re-measure, and keep the change only if it helps.

UCX coexistence and GPUDirect RDMA

NCCL and UCX use different RDMA data paths, so a Node where NCCL tests pass doesn’t guarantee that UCX-based workloads (NIXL, MPI) work. The UCX transport settings, the rdma-core version requirement for GB300 RoCE, and the UCX diagnostics live in Use GPUDirect RDMA with RoCE. For MPI workloads (for example, Intel MPI or Ansys), the relevant variables differ from NCCL. A common configuration is:

# MPI fabric selection for shared memory plus OpenFabrics interfaces.
export I_MPI_FABRICS=shm:ofi

# OpenFabrics provider for NVIDIA fabrics.
export FI_PROVIDER=mlx

Log lines that look alarming but are normal

When you enable NCCL_DEBUG=INFO, you see verbose output. The following messages don’t indicate a failure:

No Aggregation Manager sharp_am detected. This means SHARP isn’t available, so NCCL falls back to point-to-point collectives. SHARP is an InfiniBand-only switch-side reduction technology and isn’t available on RoCE (Spectrum-X) fabrics. On a RoCE cluster, this fallback is expected and not an error.
Transport fallback notices during initialization. NCCL probes available transports and reports the ones it selects. These are informational.

Enable debug logging

To diagnose RDMA or transport issues, increase NCCL verbosity. Debug logging increases log volume, so disable it after you finish.

# Verbose NCCL logging. Increases log size; remove after debugging.
export NCCL_DEBUG=INFO

# Optional: restrict verbose output to the network subsystem.
export NCCL_DEBUG_SUBSYS=NET

For the full list of NCCL variables, see NCCL_DEBUG in the NVIDIA NCCL documentation.

When NCCL reports a vendor error

NCCL sometimes reports a vendor error code from the underlying RDMA stack. Use the onset pattern to distinguish a fabric fault from a configuration problem. A sudden, multi-Node onset points to a fabric-level event. A consistent failure on a single Pod points to a container or configuration problem such as a UCX or rdma-core mismatch. For the RoCE diagnostics and the link-health check, see Use GPUDirect RDMA with RoCE and Read IB and RoCE node labels for placement.

Use GPUDirect RDMA with InfiniBand: Pod scheduling and NCCL setup for IB.
Use GPUDirect RDMA with RoCE: Multus attachment, UCX requirements, and NCCL setup for RoCE.
Read IB and RoCE node labels for placement: topology-aware placement and link-health checks.
About CoreWeave HPC interconnects: fabric overview.

​What this reference is and is not

​InfiniBand cluster recipe

​RoCE (Spectrum-X) cluster recipe

​Variables that differ from InfiniBand

​NCCL network plugin guidance

​How NCCL chooses a network plugin

​What the InfiniBand network plugin provides

​Every rank must load the same network plugin

​Decide whether to use the InfiniBand plugin

​The RoCE Spectrum-X plugin

​NCCL_IB_RETRY_CNT has a hard maximum of 7

​Segment-size and channel knobs

​UCX coexistence and GPUDirect RDMA

​Log lines that look alarming but are normal

​Enable debug logging

​When NCCL reports a vendor error

​Related pages

What this reference is and is not

InfiniBand cluster recipe

RoCE (Spectrum-X) cluster recipe

Variables that differ from InfiniBand

NCCL network plugin guidance

How NCCL chooses a network plugin

What the InfiniBand network plugin provides

Every rank must load the same network plugin

Decide whether to use the InfiniBand plugin

The RoCE Spectrum-X plugin

NCCL_IB_RETRY_CNT has a hard maximum of 7

Segment-size and channel knobs

UCX coexistence and GPUDirect RDMA

Log lines that look alarming but are normal

Enable debug logging

When NCCL reports a vendor error

Related pages