Skip to main content
When the job runs but throughput is far below expectations, that is usually NCCL falling back from InfiniBand to TCP at startup. Job failures from mid-run port drops (QP errors and NCCL timeouts) are a separate problem. On rack-scale, NVLink-connected systems, Pod placement relative to the NVLink domain is also a common cause, covered in NVLink domain placement on rack-scale systems below. Use the following checks for InfiniBand clusters. For RoCE clusters such as GB300, use the RoCE cluster recipe and verify the interface attachment in Use GPUDirect RDMA with RoCE. Quantify the slowdown. “Far below expectations” depends on your GPUs and interconnect, so compare against a measured baseline rather than a feeling. Run nccl-tests all_reduce on a small, known-good group of Nodes and record the bus bandwidth (busbw) it reports. That is your healthy baseline for the same message sizes: a job running correctly over InfiniBand lands close to it, while a job that has fallen back to TCP usually reports busbw an order of magnitude lower. If your cluster has GPU straggler detection enabled, the Slurm Job Metrics dashboard reports bus bandwidth (BusBW) per collective continuously from live jobs, so you can compare against the baseline without stopping the job to run nccl-tests. When InfiniBand is the cause, the slowness is usually one of these: InfiniBand is not actually being used, the InfiniBand interfaces are down, or NCCL is missing the environment variables that point it at the IB HCA. Quick diagnostic. From inside your training Pod, check InfiniBand port status. If a port intended for the job’s InfiniBand traffic shows Down or Disabled, contact support with the Node names and ibstat output. Common causes:
  • Pod spec is missing rdma/ib: 1. Check the RDMA resource request against Configure the Pods.
  • NCCL environment variables are missing or incorrect. Use the NCCL configuration reference for your fabric’s settings. If your stack needs UCX on the RDMA fabric, follow the reference’s UCX exception.
  • Node Pool does not have InfiniBand. See InfiniBand and RoCE labels for the labels to look for.
For the complete Pod YAML and NCCL configuration, see Use GPUDirect RDMA with InfiniBand. To inspect which transport NCCL chose, set NCCL_DEBUG=INFO and look for NET/IB (good) compared to NET/Socket (TCP fallback). On rack-scale, NVLink-connected instances such as NVL72, GPUs communicate over NVLink within an NVLink domain and over the scaleout fabric (InfiniBand or RoCE) beyond it. NCCL is fastest when collectives stay inside the NVLink domain, so a job whose Pods are spread across domains can route traffic that should run over NVLink onto the slower scaleout fabric instead. When this happens, throughput drops even though InfiniBand is healthy and every cluster-wide check passes. Align scheduling with physical connectivity so related Pods land in the same NVLink domain. Use the nvidia.com/gpu.clique label as a Pod affinity topologyKey, and see IMEX overview for how NVLink domains, partitions, and placement work on CoreWeave.

​
Isolating a bad Node on large jobs

If you are training across hundreds or thousands of GPUs and the cluster-wide checks above pass, the cause may be a small group of Nodes with degraded HCAs rather than a configuration issue. Split your nodelist in half, run nccl-tests on each half, and keep bisecting the half that performs worse until you identify the rack or Nodes contributing to the slowdown. Then contact support with the Node names.
Administrator
Last modified on September 17, 2026