Skip to main content
SUNK Standard SUNK Self-Service GPU Straggler Detection tracks each GPU in a distributed training job and flags the rank that falls out of sync, so you can drain the offending node before it degrades job performance further. It also provides continuous NVIDIA Collective Communications Library (NCCL) metrics on latency, bandwidth, and message sizes, plus rank-level attribution through the straggler tables. Dedicated Grafana dashboards surface all of this. The plugin is on by default on SUNK Self-Service and on SUNK Standard v8.0.0 and later. This guide shows how to check its status, turn it on or off, and verify that it loads in your jobs. After you complete this guide, your cluster surfaces NCCL performance data in Grafana and reports GPU stragglers, including ranks that stall outright.

Prerequisites

Before you begin, confirm your environment meets the following requirements:
  • On SUNK Standard, SUNK v7.4.0 or later.
  • NCCL 2.28.2 or later loaded by the job at runtime. For container jobs, that’s the NCCL in your container image. For jobs without a container, it’s the NCCL in the compute node image. A Python virtual environment that bundles its own NCCL, as pip install torch does, takes precedence over the image’s.
  • At least 2 CPUs allocated per task.

Enable the plugin

On SUNK v8.0.0 and later, the plugin is on by default. The Slurm Helm chart leaves compute.gpusd.enabled unset, and the operator defaults it to true. To opt out, set compute.gpusd.enabled to false in your cluster’s values.yaml.On SUNK v7.4.x, the plugin is off until you enable it. In your cluster’s values.yaml, set compute.gpusd.enabled to true. See Slurm parameter reference for more information.
When the plugin is enabled, SUNK automatically:
  • Downloads and installs the GPUSD package on compute nodes at startup.
  • Exposes ports 10400-10407 on slurmd Pods for metrics collection.
  • Exposes port 10500 on slurmd Pods on SUNK Self-Service and on SUNK Standard v8.2.0 and later, when the GPUSD package is version 1.1.0 or later, which is the default on those versions.
  • Deploys a VMPodScrape resource to scrape NCCL plugin metrics.
For jobs launched with --container or --container-image, an Enroot hook automatically mounts the plugin into the container and sets NCCL_PROFILER_PLUGIN in the job environment. Container-based jobs don’t need additional configuration. For jobs running without a container, add the following environment variable to your batch job script:

Enable debugging information

To confirm the plugin loaded successfully and to capture useful logs during initial validation, enable NCCL debug output. On first run, enable export NCCL_DEBUG=INFO to print debugging information. If you’ve correctly loaded the plugin, the output includes a line resembling the following:
You can set the following optional environment variables to control performance metrics collection and logging verbosity:

View metrics in Grafana

The Slurm Job Metrics dashboard in CoreWeave Grafana surfaces Straggler Detection data across several panels: All of these panels are in the NCCL Metrics row, below Node Resources. The exceptions are the GPU Straggler and Slow NCCL Operations annotations, which you toggle from the top of the dashboard. With the plugin enabled and metrics flowing, you can use these dashboards to identify straggler and slow GPUs during training runs and drain affected nodes before they impact job completion.
Last modified on November 5, 2026