NCCL configuration reference for CoreWeave fabrics

This page is the canonical reference for NCCL and UCX environment variables on CoreWeave high-performance backend fabrics. It covers the variables you set for InfiniBand clusters, the variables that differ on RoCE (Spectrum-X) clusters such as GB300, and the log lines that look alarming but are normal. Use it as the lookup target when a training or inference job needs fabric-specific tuning. For the Pod scheduling and Multus configuration that exposes RDMA devices to your containers, see Use GPUDirect RDMA with InfiniBand and Use GPUDirect RDMA with RoCE. For node placement using fabric labels, see Read IB and RoCE node labels for placement.

What this reference is and is not

This page covers fabric-level NCCL and UCX settings: which interfaces NCCL uses, how to select host channel adapters, and how the transport differs between InfiniBand and RoCE on CoreWeave. This page isn’t a guide to collective algorithms, topology-aware ranks, or debugging distributed training hangs. Those belong to training documentation. This page also doesn’t tune performance baselines or publish expected bandwidth numbers.

InfiniBand cluster recipe

On InfiniBand clusters, NCCL uses its built-in InfiniBand verbs transport. Set the following environment variables in your Pod spec or Slurm job. These values match the examples in Use GPUDirect RDMA with InfiniBand.

# Front-end interface for NCCL control and out-of-band traffic.
# Safe to set; selects the primary Pod interface, not the RDMA path.
export NCCL_SOCKET_IFNAME=eth0

# InfiniBand host channel adapters NCCL uses for GPUDirect RDMA collectives.
# "ibp" matches the InfiniBand device naming on CoreWeave IB clusters.
export NCCL_IB_HCA=ibp

# UCX devices and transports, if your stack uses UCX (for example, MPI workloads).
export UCX_NET_DEVICES=eth0
export UCX_TLS=tcp

Variable	Purpose
`NCCL_SOCKET_IFNAME`	Selects the front-end interface for NCCL’s TCP control and bootstrap traffic. Set to the primary Pod interface, usually `eth0`.
`NCCL_IB_HCA`	Selects the RDMA host channel adapters for collectives. Use `ibp` on CoreWeave InfiniBand clusters.
`UCX_NET_DEVICES`	Devices UCX uses, for stacks that run UCX alongside NCCL.
`UCX_TLS`	UCX transports to enable.

RoCE (Spectrum-X) cluster recipe

RoCE clusters such as GB300 use Ethernet-based RDMA. Several NCCL variables differ from InfiniBand because the address family is IPv6 and the traffic class is set explicitly. The interface attachment (Multus NADs) and the full Pod spec live in Use GPUDirect RDMA with RoCE.

# Front-end interface for control traffic, same as InfiniBand.
export NCCL_SOCKET_IFNAME=eth0

# RoCE RDMA devices. On GB300 RoCE clusters this is the "ibp" interface.
# On other RoCE clusters, match the interface names that Multus creates
# (for example, a comma-separated list such as net1,net2).
export NCCL_IB_HCA=ibp

# RoCE uses an IPv6 address family for RDMA, unlike InfiniBand.
export NCCL_IB_ADDR_FAMILY=AF_INET6

# Traffic class for RoCE. 96 is the value validated on GB300 RoCE.
export NCCL_IB_TC=96

# NCCL network plugin. On RoCE, the default built-in path is "none".
# The Spectrum-X plugin is only used for very large jobs.
export NCCL_NET_PLUGIN=none

Variables that differ from InfiniBand

Variable	InfiniBand	RoCE (Spectrum-X)
`NCCL_IB_ADDR_FAMILY`	Not required	`AF_INET6` (RoCE addressing on CoreWeave is IPv6)
`NCCL_IB_TC`	Not required	Set the RoCE traffic class, for example `96` on GB300
`NCCL_NET_PLUGIN`	Not required	`none` for typical jobs
Interface attachment	Devices exposed directly	Attached through Multus NADs (see RoCE guide)

NCCL network plugin guidance

On CoreWeave RoCE fabrics, NCCL defaults to NCCL_NET_PLUGIN=none, which uses the built-in InfiniBand verbs path. This is correct for most jobs. The NVIDIA Spectrum-X NCCL plugin provides NIC-level hardware-accelerated collectives, but it’s only relevant for large jobs at the scale of roughly 10,000 or more GPUs. For typical multi-Node training and inference jobs, leave the plugin at none. If you believe your workload qualifies for the Spectrum-X plugin, contact your CoreWeave representative.

NCCL_IB_RETRY_CNT has a hard maximum of 7

NCCL_IB_RETRY_CNT is stored in a 3-bit register, so its maximum valid value is 7. Setting it to 8 or higher doesn’t increase retries: the value wraps and is effectively treated as a small or zero count, which makes the fabric appear far less reliable than it is. If you want more retry tolerance, 7 is the ceiling. Don’t set it above 7.

Segment-size and channel knobs

Most workloads don’t need to tune these. Change them only when you have a measured reason to, and change one at a time so you can attribute the effect.

Variable	What it controls	When to consider changing it
`NCCL_BUFFSIZE`	Size of the per-connection staging buffer	Large-message collectives where the default underfills the link
`NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`	Lower and upper bounds on the number of parallel channels (CUDA blocks) NCCL uses	Raise the minimum when throughput plateaus below the link rate on large messages; lower the maximum when NCCL consumes too many SMs
`NCCL_P2P_NET_CHUNKSIZE`	Chunk size for point-to-point network transfers, which back collectives like `all_to_all`	Lower it for small-message latency; raise it for large-message point-to-point bandwidth (CoreWeave GB300 RoCE baselines use `524288`)

Tuning these without a baseline measurement usually makes performance worse, not better. Capture a baseline first, change one knob, re-measure, and keep the change only if it helps.

UCX coexistence and GPUDirect RDMA

NCCL and UCX use different RDMA data paths, so a Node where NCCL tests pass doesn’t guarantee that UCX-based workloads (NIXL, MPI) work. The UCX transport settings, the rdma-core version requirement for GB300 RoCE, and the UCX diagnostics live in Use GPUDirect RDMA with RoCE. For MPI workloads (for example, Intel MPI or Ansys), the relevant variables differ from NCCL. A common configuration is:

# MPI fabric selection for shared memory plus OpenFabrics interfaces.
export I_MPI_FABRICS=shm:ofi

# OpenFabrics provider for NVIDIA fabrics.
export FI_PROVIDER=mlx

Log lines that look alarming but are normal

When you enable NCCL_DEBUG=INFO, you see verbose output. The following messages don’t indicate a failure:

No Aggregation Manager sharp_am detected. This means SHARP isn’t available, so NCCL falls back to point-to-point collectives. SHARP is an InfiniBand-only switch-side reduction technology and isn’t available on RoCE (Spectrum-X) fabrics. On a RoCE cluster, this fallback is expected and not an error.
Transport fallback notices during initialization. NCCL probes available transports and reports the ones it selects. These are informational.

Enable debug logging

To diagnose RDMA or transport issues, increase NCCL verbosity. Debug logging increases log volume, so disable it after you finish.

# Verbose NCCL logging. Increases log size; remove after debugging.
export NCCL_DEBUG=INFO

# Optional: restrict verbose output to the network subsystem.
export NCCL_DEBUG_SUBSYS=NET

For the full list of NCCL variables, see NCCL_DEBUG in the NVIDIA NCCL documentation.

When NCCL reports a vendor error

NCCL sometimes reports a vendor error code from the underlying RDMA stack. Use the onset pattern to distinguish a fabric fault from a configuration problem. A sudden, multi-Node onset points to a fabric-level event. A consistent failure on a single Pod points to a container or configuration problem such as a UCX or rdma-core mismatch. For the RoCE diagnostics and the link-health check, see Use GPUDirect RDMA with RoCE and Read IB and RoCE node labels for placement.

Use GPUDirect RDMA with InfiniBand: Pod scheduling and NCCL setup for IB.
Use GPUDirect RDMA with RoCE: Multus attachment, UCX requirements, and NCCL setup for RoCE.
Read IB and RoCE node labels for placement: topology-aware placement and link-health checks.
About CoreWeave HPC interconnects: fabric overview.

​What this reference is and is not

​InfiniBand cluster recipe

​RoCE (Spectrum-X) cluster recipe

​Variables that differ from InfiniBand

​NCCL network plugin guidance

​NCCL_IB_RETRY_CNT has a hard maximum of 7

​Segment-size and channel knobs

​UCX coexistence and GPUDirect RDMA

​Log lines that look alarming but are normal

​Enable debug logging

​When NCCL reports a vendor error

​Related pages