> ## Documentation Index > Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt > Use this file to discover all available pages before exploring further. # NCCL configuration reference for CoreWeave fabrics > Canonical NCCL and UCX environment variables for InfiniBand and RoCE backend fabrics on CoreWeave This page is the canonical reference for NCCL and UCX environment variables on CoreWeave high-performance backend fabrics. It covers the variables you set for InfiniBand clusters, the variables that differ on RoCE (Spectrum-X) clusters such as GB300, how NCCL chooses a network plugin, and the log lines that look alarming but are normal. Use it as the lookup target when a training or inference job needs fabric-specific tuning. For the Pod scheduling and Multus configuration that exposes RDMA devices to your containers, see [Use GPUDirect RDMA with InfiniBand](/products/networking/hpc-interconnect/use-gpudirect-rdma) and [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce). For node placement using fabric labels, see [Read IB and RoCE node labels for placement](/products/networking/hpc-interconnect/infiniband-roce-labels). ## What this reference is and is not This page covers fabric-level NCCL and UCX settings: which interfaces NCCL uses, how to select host channel adapters, how NCCL chooses a network plugin, and how the transport differs between InfiniBand and RoCE on CoreWeave. This page isn't a guide to collective algorithms, topology-aware ranks, or debugging distributed training hangs. Those belong to training documentation. This page also doesn't tune performance baselines or publish expected bandwidth numbers. ## InfiniBand cluster recipe On InfiniBand clusters, set the following environment variables in your Pod spec or Slurm job. The settings select NCCL's built-in InfiniBand verbs transport. The values match the examples in [Use GPUDirect RDMA with InfiniBand](/products/networking/hpc-interconnect/use-gpudirect-rdma) and the Slurm reference jobs in the [`nccl-tests` repository](https://github.com/coreweave/nccl-tests). Some jobs in that repository point `UCX_NET_DEVICES` at the InfiniBand devices instead, for stacks that run UCX over RDMA. ```bash theme={"system"} # Front-end interface for NCCL control and out-of-band traffic. # Safe to set; selects the primary Pod interface, not the RDMA path. export NCCL_SOCKET_IFNAME=eth0 # InfiniBand host channel adapters NCCL uses for GPUDirect RDMA collectives. # "ibp" matches the InfiniBand device naming on CoreWeave IB clusters. export NCCL_IB_HCA=ibp # Use NCCL's built-in InfiniBand verbs transport. Omit this line if you're # enabling SHARP, which needs the HPC-X plugin. Every rank must match. # See "NCCL network plugin guidance". export NCCL_NET_PLUGIN=none # UCX devices and transports, if your stack uses UCX (for example, MPI workloads). # NCCL doesn't use UCX. These values keep UCX off the InfiniBand devices. export UCX_NET_DEVICES=eth0 export UCX_TLS=tcp ``` | Variable | Purpose | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NCCL_SOCKET_IFNAME` | Selects the front-end interface for NCCL's TCP control and bootstrap traffic. Set to the primary Pod interface, usually `eth0`. | | `NCCL_IB_HCA` | Selects the RDMA host channel adapters for collectives. Use `ibp` on CoreWeave InfiniBand clusters. | | `NCCL_NET_PLUGIN` | Selects the NCCL network plugin. Set to `none` to use the built-in InfiniBand verbs transport, or leave it unset to load the HPC-X network plugin, which SHARP requires. Every rank in a job must use the same value. | | `UCX_NET_DEVICES` | Devices UCX uses, for stacks that run UCX alongside NCCL. | | `UCX_TLS` | Transports UCX is allowed to use, for stacks that run UCX alongside NCCL. | ## RoCE (Spectrum-X) cluster recipe RoCE clusters such as GB300 use Ethernet-based RDMA. Several NCCL variables differ from InfiniBand because the address family is IPv6 and the traffic class is set explicitly. The interface attachment (Multus NADs) and the full Pod spec live in [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce). ```bash theme={"system"} # Front-end interface for control traffic, same as InfiniBand. export NCCL_SOCKET_IFNAME=eth0 # RoCE RDMA devices. On GB300 RoCE clusters this is the "ibp" interface. # On other RoCE clusters, match the interface names that Multus creates # (for example, a comma-separated list such as net1,net2). export NCCL_IB_HCA=ibp # RoCE uses an IPv6 address family for RDMA, unlike InfiniBand. export NCCL_IB_ADDR_FAMILY=AF_INET6 # Traffic class for RoCE. 96 is the value validated on GB300 RoCE. export NCCL_IB_TC=96 # Load the Spectrum-X network plugin, which needs HPC-X 2.50 or newer. # Do not set NCCL_NET_PLUGIN. Leave it unset so NCCL finds the plugin. # Every rank must match. See "NCCL network plugin guidance". export LD_LIBRARY_PATH=/opt/hpcx/nccl_spectrum-x_plugin/lib:$LD_LIBRARY_PATH ``` ### Variables that differ from InfiniBand | Variable | InfiniBand | RoCE (Spectrum-X) | | --------------------- | ------------------------ | ----------------------------------------------------------------------- | | `NCCL_IB_ADDR_FAMILY` | Not required | `AF_INET6` (RoCE addressing on CoreWeave is IPv6) | | `NCCL_IB_TC` | Not required | Set the RoCE traffic class, for example `96` on GB300 | | `NCCL_NET_PLUGIN` | `none` for typical jobs | Leave unset so NCCL loads the Spectrum-X plugin | | `LD_LIBRARY_PATH` | No change needed | Prepend `/opt/hpcx/nccl_spectrum-x_plugin/lib` so NCCL finds the plugin | | Interface attachment | Devices exposed directly | Attached through Multus NADs (see RoCE guide) | For plugin selection and SHARP requirements, see [NCCL network plugin guidance](#nccl-network-plugin-guidance). ## NCCL network plugin guidance `NCCL_NET_PLUGIN` selects the network plugin NCCL uses for communication between Nodes. A network plugin replaces NCCL's built-in transport, so every rank in a job must agree on the setting. The following sections describe how NCCL chooses a plugin, what the InfiniBand and RoCE plugins provide, and how to decide whether to use the InfiniBand plugin. ### How NCCL chooses a network plugin When `NCCL_NET_PLUGIN` is unset, NCCL searches the library path for `libnccl-net.so` and uses that plugin if it finds one. NCCL falls back to its built-in transport when it finds no plugin it can use. Setting `NCCL_NET_PLUGIN=none` skips the search and forces the built-in transport. Many CoreWeave compute environments include NVIDIA HPC-X, which installs the `nccl-rdma-sharp-plugins` build of `libnccl-net.so`. CoreWeave SUNK compute container images and the images built from the `nccl-tests` repository both bundle it. Where HPC-X is on the library path, NCCL loads the plugin unless you set `NCCL_NET_PLUGIN=none`. This is why the reference jobs in that repository set the variable explicitly instead of relying on a default. Because the plugin ships in the container image rather than on the Node, whether a rank loads it depends on the image that rank runs. ### What the InfiniBand network plugin provides The `nccl-rdma-sharp-plugins` network plugin is NVIDIA's plugin as bundled with HPC-X. It isn't a CoreWeave fork, and CoreWeave applies no source patches to it. The plugin replaces NCCL's built-in transport between Nodes and can also use CollNet and SHARP for the collectives that support them. NCCL's built-in InfiniBand verbs transport, which `NCCL_NET_PLUGIN=none` selects, has no SHARP path. Loading the plugin doesn't enable SHARP on its own. `NCCL_COLLNET_ENABLE` defaults to `0`, and CoreWeave doesn't enable SHARP on its fabrics by default, so jobs fall back to point-to-point collectives. Loading the plugin and setting `NCCL_COLLNET_ENABLE=1` still aren't sufficient. SHARP also has to be enabled on the fabric itself, which CoreWeave does for you. To find out whether SHARP is enabled on your cluster, or to ask for it, contact your CoreWeave representative. The `No Aggregation Manager sharp_am detected` line reports that fallback on InfiniBand as well as RoCE. For more information, see [Log lines that look alarming but are normal](#log-lines-that-look-alarming-but-are-normal). Plugin versions differ between images, and the network plugin API version the plugin exposes must match what your NCCL build expects. A plugin your NCCL build can't use isn't a startup failure. NCCL logs that the external network plugin is unsupported, unloads it, and falls back to its built-in transport. A rank that falls back this way while other ranks keep the plugin creates the mixed configuration described in [Every rank must load the same network plugin](#every-rank-must-load-the-same-network-plugin). Instead of assuming a single fleet-wide version, check your container image tag to find the version your job uses. ### Every rank must load the same network plugin Every rank in a job must load the same network plugin. The plugin transport and NCCL's built-in transport don't interoperate, so you can't run some ranks on one and some ranks on the other. A mixed configuration, where some ranks use a plugin transport and other ranks use NCCL's built-in transport, isn't supported. It can cause faults during collectives, including memory faults during `broadcast`. Two configurations are uniform, and either one is supported: * Set `NCCL_NET_PLUGIN=none` on every rank. All ranks use the built-in InfiniBand verbs transport. This is the recommended setting on InfiniBand unless you're deliberately enabling SHARP. * Run every rank from the same container image and leave `NCCL_NET_PLUGIN` unset. All ranks load the same plugin from that image. A common cause of a mixed configuration is that different ranks run different container images in one job. Building your own image on top of a CoreWeave base image can also change whether HPC-X is on the library path, for example when the new image replaces `LD_LIBRARY_PATH` instead of appending to it. A rank that used to load the plugin might stop loading it. To confirm that ranks agree, set `NCCL_DEBUG=INFO` and compare the `Using network` line across ranks. Every rank must report the same network name. The `NET/Plugin` lines in the same output show which plugin libraries each rank searched for, and report the ones it couldn't find. ### Decide whether to use the InfiniBand plugin Use `NCCL_NET_PLUGIN=none` unless you're deliberately enabling SHARP. NCCL's built-in transport performs well on CoreWeave InfiniBand fabrics, and CoreWeave publishes no custom NCCL tuner. CoreWeave publishes no GPU count at which the HPC-X plugin becomes faster than the built-in transport. If you want to compare without enabling SHARP, measure with the paired reference jobs in the [`nccl-tests` repository](https://github.com/coreweave/nccl-tests). Those paired jobs differ as follows: * The standard jobs set `NCCL_NET_PLUGIN=none`. * The SHARP variants, whose filenames end in `-sharp`, leave the plugin loaded and set `NCCL_COLLNET_ENABLE=1` and `SHARP_COLL_ENABLE_PCI_RELAXED_ORDERING=1`. Compare a pair while keeping the image, Nodes, placement, message sizes, and iteration count identical. If the plugin doesn't produce a repeatable improvement in your workload, use `NCCL_NET_PLUGIN=none`. ### The RoCE Spectrum-X plugin The NVIDIA Spectrum-X NCCL plugin provides NIC-level hardware-accelerated collectives. Use it on RoCE clusters for jobs of any size. The plugin requires HPC-X 2.50 or newer, and CoreWeave images that bundle HPC-X ship it at `/opt/hpcx/nccl_spectrum-x_plugin`. To load this plugin, prepend `/opt/hpcx/nccl_spectrum-x_plugin/lib` to `LD_LIBRARY_PATH` and leave `NCCL_NET_PLUGIN` unset, as shown in the RoCE cluster recipe on this page. As on InfiniBand, every rank in a job must load the same network plugin. If your image ships an HPC-X version older than 2.50, upgrade the image or contact your CoreWeave representative. SHARP is an InfiniBand-only technology and isn't available on RoCE fabrics. ## NCCL\_IB\_RETRY\_CNT has a hard maximum of 7 `NCCL_IB_RETRY_CNT` is stored in a 3-bit register, so its maximum valid value is 7. Setting it to 8 or higher doesn't increase retries: the value wraps and is effectively treated as a small or zero count, which makes the fabric appear far less reliable than it is. If you want more retry tolerance, 7 is the ceiling. Don't set it above 7. ## Segment-size and channel knobs Most workloads don't need to tune these. Change them only when you have a measured reason to, and change one at a time so you can attribute the effect. | Variable | What it controls | When to consider changing it | | ------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `NCCL_BUFFSIZE` | Size of the per-connection staging buffer | Large-message collectives where the default underfills the link | | `NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS` | Lower and upper bounds on the number of parallel channels (CUDA blocks) NCCL uses | Raise the minimum when throughput plateaus below the link rate on large messages; lower the maximum when NCCL consumes too many SMs | | `NCCL_P2P_NET_CHUNKSIZE` | Chunk size for point-to-point network transfers, which back collectives like `all_to_all` | Lower it for small-message latency; raise it for large-message point-to-point bandwidth (CoreWeave GB300 RoCE baselines use `524288`) | Tuning these without a baseline measurement usually makes performance worse, not better. Capture a baseline first, change one knob, re-measure, and keep the change only if it helps. ## UCX coexistence and GPUDirect RDMA NCCL and UCX use different RDMA data paths, so a Node where NCCL tests pass doesn't guarantee that UCX-based workloads (NIXL, MPI) work. The UCX transport settings, the `rdma-core` version requirement for GB300 RoCE, and the UCX diagnostics live in [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce#ucx-and-gpudirect-rdma-on-gb300-roce). For MPI workloads (for example, Intel MPI or Ansys), the relevant variables differ from NCCL. A common configuration is: ```bash theme={"system"} # MPI fabric selection for shared memory plus OpenFabrics interfaces. export I_MPI_FABRICS=shm:ofi # OpenFabrics provider for NVIDIA fabrics. export FI_PROVIDER=mlx ``` ## Log lines that look alarming but are normal When you enable `NCCL_DEBUG=INFO`, you see verbose output. The following messages don't indicate a failure: * `No Aggregation Manager sharp_am detected`. This means SHARP isn't available, so NCCL falls back to point-to-point collectives. SHARP is an InfiniBand-only switch-side reduction technology and isn't available on RoCE (Spectrum-X) fabrics. On a RoCE cluster, this fallback is expected and not an error. * Transport fallback notices during initialization. NCCL probes available transports and reports the ones it selects. These are informational. ## Enable debug logging To diagnose RDMA or transport issues, increase NCCL verbosity. Debug logging increases log volume, so disable it after you finish. ```bash theme={"system"} # Verbose NCCL logging. Increases log size; remove after debugging. export NCCL_DEBUG=INFO # Optional: restrict verbose output to the network subsystem. export NCCL_DEBUG_SUBSYS=NET ``` For the full list of NCCL variables, see [`NCCL_DEBUG` in the NVIDIA NCCL documentation](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-debug). ## When NCCL reports a vendor error NCCL sometimes reports a vendor error code from the underlying RDMA stack. Use the onset pattern to distinguish a fabric fault from a configuration problem. A sudden, multi-Node onset points to a fabric-level event. A consistent failure on a single Pod points to a container or configuration problem such as a UCX or `rdma-core` mismatch. For the RoCE diagnostics and the link-health check, see [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce#ucx-and-gpudirect-rdma-on-gb300-roce) and [Read IB and RoCE node labels for placement](/products/networking/hpc-interconnect/infiniband-roce-labels). ## Related pages * [Use GPUDirect RDMA with InfiniBand](/products/networking/hpc-interconnect/use-gpudirect-rdma): Pod scheduling and NCCL setup for IB. * [Use GPUDirect RDMA with RoCE](/products/networking/hpc-interconnect/use-gpudirect-rdma-roce): Multus attachment, UCX requirements, and NCCL setup for RoCE. * [Read IB and RoCE node labels for placement](/products/networking/hpc-interconnect/infiniband-roce-labels): topology-aware placement and link-health checks. * [About CoreWeave HPC interconnects](/products/networking/hpc-interconnect/about-hpc-interconnect): fabric overview.