kubelet, which measures the cgroup of the slurmd container. Slurm places job processes in cgroups that it manages itself, outside that hierarchy, so cAdvisor never attributes job CPU and memory to the slurmd container. Pod metrics report only the slurmd daemon’s own overhead, which stays near zero no matter what the job is doing. Slurm still accounts for the usage and enforces the job’s limits through its own cgroup configuration. See Compute and login nodes and the slurmConfig.cgroupConfig defaults in Slurm parameters.
To see what Slurm has allocated on a node or to a job, use the Slurm metrics the Syncer exports: slurm_node_cpu_alloc, slurm_node_mem_alloc, and slurm_node_gpu_alloc for per-node allocation, and slurm_job_cpus_allocated and slurm_job_gpus_allocated for per-job allocation. For the full list, see Syncer metrics.
Allocation is not the same as consumption. To see what the hardware is actually doing, use these CoreWeave Managed Grafana dashboards:
- Slurm Job Metrics for the GPU and CPU utilization of a single job.
- Node details for CPU, GPU, and memory utilization on one compute node. Expand the Resources panel group.
- Slurm Cluster for job resource usage across a namespace, broken down by user and partition.
Nodes