Slurm Job Metrics

To view the dashboard, go to the Slurm Job Metrics dashboard. The Slurm Job Metrics dashboard displays detailed information about the performance and hardware utilization of a selected Slurm job. You can use this dashboard to:

• Monitor the GPU and CPU utilization of a given Slurm job
• Track the rate of filesystem operations related to the Slurm job
• View Node conditions and alerts that may impact the performance of the Slurm job

​

Filters and parameters

Use these filters at the top of the page to choose the data you want to view: The dashboard also includes buttons with links to the SLURM / Namespace dashboard and slurmctld logs. Set the time range and refresh rate parameters at the top-right of the page. The default time range is 5 minutes, and the default refresh rate is 1 minute.

​

Panel descriptions

​

Job Info

The Job Info section displays identifying information about the selected Slurm job, including:

​

Estimate model flop utilization (MFU) from tensor core utilization

Use the Tensor Core Utilization Running Jobs panel in the GPU Metrics section to get a rough estimate of MFU. Although tensor core utilization is a hardware-level metric and MFU is an application-level metric, the Tensor Core Utilization Running Jobs panel acts as an upper bound for MFU. Your model’s actual mathematical efficiency cannot exceed the percentage of time the hardware’s tensor cores are actively processing instructions. Use tensor core utilization as a hard upper bound and diagnostic resource. Note that the Current FP8 FLOPS panel is not a good indicator of actual model FLOPS. It provides an idealized estimate based on tensor core utilization across all GPUs in the job. This metric assumes a specific scenario: an H100 GPU running FP8 operations with structured sparsity, where 100% tensor core utilization corresponds to 1979 TFLOPS per GPU.

​

Job Info: Job State Timeline and Last State

The Job Info section contains panels that display information about the state of the selected Slurm job. Last State displays the most recent reported state, while the Job State Timeline displays the job’s status over time. A Slurm job can be in the following states:

​

Job Info: Node alerts

Job alerts in the Slurm Job / Metrics dashboard are generated by CoreWeave’s Mission Control, an automated system that continuously monitors and manages the underlying compute infrastructure to maintain high cluster reliability and availability. These alerts target hardware and system-level issues, such as GPU errors, networking failures, and endpoint timeouts - conditions that are not typically observable through application-layer metrics such as training metrics or standard logs.

​

Understanding Node alerts and conditions

The following image shows an interruption caused by a Node alert, namely GPUContainedECCError.

• Blue lines: Indicate Node conditions
• Red lines: Indicate Node alerts

The Node alert indicated by the red line, GPUContainedECCError, appears prior to drop in compute, while the Node conditions indicated by the blue lines delineate the window the drop in compute occurred within. The image shows that the H100 cluster experienced a hard fault around 15:15. Throughput dropped from 500 PFLOPS to 0 because the Slurm scheduler pulled a Node out of the pool due to a GPUContainedECCError. Understanding how Node alerts and conditions are overlaid on Node metrics can help diagnose and troubleshoot problems.

​

GPU Metrics

The GPU Metrics section displays detailed information related to hardware utilization. In this section, red lines correspond with higher temperature or utilization of the measured value, while green lines indicate a lower value or idle state. Whether these values suggest “good” or “bad” performance depends on the expected behavior and resource utilization of the job.

​

GPU Metrics: Color coding

This example shows a high (red) value in the GPU Core Utilization panel, a medium (yellow) value in the GPU Temperatures panel, and low (green) values in the GPU Mem Copy Utilization panel. This indicates that the tracked Slurm job had high utilization of GPU Cores and low utilization of GPU memory, which may be expected for a small model size. Comparing the fluctuations in GPU Temperature with the charts in the Filesystem section reveals that drops in the GPU temperature may correspond with spikes in NFS write operations.

​

Filesystem

The Filesystem section includes information about read and write operations on the Network File System (NFS) and local files.

​

Filesystem: NFS Average Response and Request

The NFS Average Response and Request graphs describe the performance of the filesystem. A slowdown or spike could indicate that the storage is too slow, and that the job might perform better with faster or a different type of storage, such as object storage.

​

Filesystem: NFS Total Read / Write

The NFS Total Read / Write graphs typically display a large red spike when a job starts, as the model and data are read in. While the job runs, the graph shows smaller write spikes at regular intervals, which occur as the checkpoints are written out. Comparing these graphs with the panels in the GPU Metrics section may help to confirm that running jobs are behaving as expected.

​

Node Resources

The Node Resources section includes the CPU Allocation panel, which displays the total number of CPU cores utilized over the job runtime.