Prerequisites
Before you begin, confirm your environment meets the following requirements:- On SUNK Standard, SUNK v7.4.0 or later.
- NCCL 2.28.2 or later loaded by the job at runtime. For container jobs, that’s the NCCL in your container image. For jobs without a container, it’s the NCCL in the compute node image. A Python virtual environment that bundles its own NCCL, as
pip install torchdoes, takes precedence over the image’s. - At least 2 CPUs allocated per task.
Enable the plugin
- SUNK Standard
- SUNK Self-Service
On SUNK v8.0.0 and later, the plugin is on by default. The Slurm Helm chart leaves
compute.gpusd.enabled unset, and the operator defaults it to true. To opt out, set compute.gpusd.enabled to false in your cluster’s values.yaml.On SUNK v7.4.x, the plugin is off until you enable it. In your cluster’s values.yaml, set compute.gpusd.enabled to true. See Slurm parameter reference for more information.- Downloads and installs the GPUSD package on compute nodes at startup.
- Exposes ports
10400-10407onslurmdPods for metrics collection. - Exposes port
10500onslurmdPods on SUNK Self-Service and on SUNK Standard v8.2.0 and later, when the GPUSD package is version 1.1.0 or later, which is the default on those versions. - Deploys a VMPodScrape resource to scrape NCCL plugin metrics.
--container or --container-image, an Enroot hook automatically mounts the plugin into the container and sets NCCL_PROFILER_PLUGIN in the job environment. Container-based jobs don’t need additional configuration.
For jobs running without a container, add the following environment variable to your batch job script:
Enable debugging information
To confirm the plugin loaded successfully and to capture useful logs during initial validation, enable NCCL debug output. On first run, enableexport NCCL_DEBUG=INFO to print debugging information. If you’ve correctly loaded the plugin, the output includes a line resembling the following:
View metrics in Grafana
The Slurm Job Metrics dashboard in CoreWeave Grafana surfaces Straggler Detection data across several panels:
All of these panels are in the NCCL Metrics row, below Node Resources. The exceptions are the GPU Straggler and Slow NCCL Operations annotations, which you toggle from the top of the dashboard.
With the plugin enabled and metrics flowing, you can use these dashboards to identify straggler and slow GPUs during training runs and drain affected nodes before they impact job completion.