Enable GPU straggler detection

SUNK Standard SUNK Self-Service CoreWeave Straggler Detection tracks each GPU in a distributed training job and flags the rank that falls out of sync, so you can drain the offending node before it degrades job performance further. It also provides continuous NVIDIA Collective Communications Library (NCCL) metrics on latency, bandwidth, and message sizes, plus rank-level attribution through the straggler tables. Dedicated Grafana dashboards surface all of this. This guide details how to install and configure the Straggler Detection plugin within a SUNK cluster. After you complete this guide, your cluster flags GPU stragglers, including ranks that stall outright, and surfaces NCCL performance data in Grafana.

Prerequisites

Before you begin, confirm your environment meets the following requirements:

SUNK v7.4.0 or later.
NCCL 2.28.2 or later loaded by the job at runtime. For container jobs, that’s the NCCL in your container image. For jobs without a container, it’s the NCCL in the compute node image. A Python virtual environment that bundles its own NCCL, as pip install torch does, takes precedence over the image’s.
At least 2 CPUs allocated per task.

Enable the plugin

On SUNK v8.0.0 and later, the plugin is on by default. The Slurm Helm chart leaves compute.gpusd.enabled unset, and the operator defaults it to true, as it does the SunkCluster key spec.features.gpusd.enabled. Set one to false to opt out. On SUNK v7.4.x, the plugin is off until you enable it. In your cluster’s values.yaml, set compute.gpusd.enabled to true. See Slurm parameter reference for more information.

compute:
  gpusd:
    enabled: true

This setting automatically:

Downloads and installs the GPUSD package on compute nodes at startup.
Exposes ports 10400-10407 on slurmd Pods for metrics collection.
Deploys a VMPodScrape resource to scrape NCCL plugin metrics.

For jobs launched with --container or --container-image, an Enroot hook automatically mounts the plugin into the container and sets NCCL_PROFILER_PLUGIN in the job environment. Container-based jobs don’t need additional configuration. For jobs running without a container, add the following environment variable to your batch job script:

export NCCL_PROFILER_PLUGIN=/usr/lib/libnccl-profiler-gpusd.so

Enable debugging information

To confirm the plugin loaded successfully and to capture useful logs during initial validation, enable NCCL debug output. On first run, enable export NCCL_DEBUG=INFO to print debugging information. If you’ve correctly loaded the plugin, the output includes a line resembling the following:

h200-205-187:1189647:1189647 [0] NCCL INFO Successfully loaded external profiler plugin /usr/lib/libnccl-profiler-gpusd.so

You can set the following optional environment variables to control performance metrics collection and logging verbosity:

Variable	Purpose
`GPUSD_PERF_DEBUG=0`	Disable performance metrics (straggler detection only)
`GPUSD_PERF_DEBUG=1`	Enable performance metrics (always on)
`GPUSD_PERF_DEBUG=2`	Toggle metrics with `SIGUSR1` (on) or `SIGUSR2` (off)
`GPUSD_DEBUG=VERSION`	Minimal logging
`GPUSD_DEBUG=INFO`	Standard logging
`GPUSD_DEBUG=TRACE`	Verbose logging

View metrics in Grafana

The Slurm Job Metrics dashboard in CoreWeave Grafana surfaces Straggler Detection data across several panels:

Panel	Description
GPU Straggler	Table that identifies the rank and node that fell behind the rest of the job
Slow NCCL Operations	Table that identifies ranks whose algorithmic bandwidth (AlgoBW) dropped, and the collective affected
NCCL BusBW and NCCL Latency	Bandwidth and collective latency, one panel per process-group size
NCCL Message Size	Average payload size per collective operation
GPU Straggler annotation	Marks the intervals where a straggler was flagged on the Job State Timeline and Current FP8 FLOPS panels
Slow NCCL Operations annotation	Overlays slow collective operations onto the NCCL BusBW panels

All of these panels are in the NCCL Metrics row, below Node Resources. The exceptions are the GPU Straggler and Slow NCCL Operations annotations, which you toggle from the top of the dashboard. With the plugin enabled and metrics flowing, you can use these dashboards to identify straggler and slow GPUs during training runs and drain affected nodes before they impact job completion.

​Prerequisites

​Enable the plugin

​Enable debugging information

​View metrics in Grafana

Prerequisites

Enable the plugin

Enable debugging information

View metrics in Grafana