Skip to main content
The Syncer bidirectionally synchronizes the state between Slurm nodes and the respective Kubernetes Pods. It translates state information into formats understood by each side, such as Ready, Running, or Drain, and lets operations made on either side update the state of the other. The Syncer is deployed along with each Slurm cluster.

​
Information flow and reconcile operations

The Syncer supports several flows of information and operations. The following sections describe each flow and the conditions that trigger it.

​
Slurm drains from Kubernetes

When certain conditions happen on the Kubernetes side that make the respective Slurm node either inoperable or undesired for continued Slurm job scheduling, the Syncer propagates these conditions as a drain on the Slurm node. When the condition clears, the Syncer then removes the drain. A drain from Kubernetes uses the k8s: prefix to indicate within Slurm that the drain originated on the Kubernetes side.
The Syncer only removes or updates drain reasons that are prefixed with k8s:. A non-prefixed drain is left as is.
Some of the possible conditions that apply a drain are:
  • The Kubernetes Pod associated with this Slurm node is not ready.
  • The Kubernetes Pod associated with this Slurm node has been deleted.
  • The Kubernetes Pod associated with this Slurm node is pending deletion. See NodeSet Controller.
  • The Kubernetes Pod associated with this Slurm node is Cordoned. See Pod Controller.
Many Kubernetes-side drains are removed automatically when the originating condition is resolved on the Kubernetes side. When the drain is removed in Slurm before the originating condition is resolved, the Syncer reapplies the drain.

​
Slurm downed nodes

Nodes in Slurm can be set down by three routes:
  • Automatically by the Slurm Controller.
  • Manually by the user from within Slurm.
  • Upon Pod deletion in Kubernetes.
When a node is downed, the Syncer doesn’t automatically resume or drain the node. This allows for more flexibility when managing nodes in Slurm. The Slurm controller transitions nodes out of down per the ReturnToService configuration value.
The default (and recommended) configuration of the Slurm chart uses ReturnToService=2, which automatically resumes any down node that starts communicating with the Slurm controller. To change this behavior, adjust this value. A non-default value requires the user to take action within Slurm after a Pod is updated in Kubernetes, before the node is usable in Slurm.

​
Slurm node deletion

The Syncer updates the state within Slurm following NodeSlice changes. For example, when nodes are removed from NodeSets, these changes are reflected in the underlying NodeSlice(s). After detecting removed NodeSlice entries, the Syncer requests deletion of the corresponding Slurm nodes. Enable this functionality using the Slurm chart option .syncer.config.syncer.slurmNodeCleanUp.

​
Slurm node status

The Syncer converts the current running, responding, and drain Slurm states into conditions and labels on the Pod. The conditions SlurmDrain, SlurmRunning, and SlurmNotResponding mirror the state within Slurm. The Syncer propagates the reason for the drain into the Message for the SlurmDrain condition. The labels sunk.coreweave.com/running, sunk.coreweave.com/drain, and sunk.coreweave.com/not_responding aid dashboards that use metrics from kube-state-metrics, and aren’t used for any logic within the Operator or Syncer. When a Slurm node is drained from within Slurm, that drain propagates up to the Node as well. For more information, see NodeController.

​
NHC drain and HPC verification

Although this feature was implemented for CoreWeave’s particular environment, it can be used for similar workflows in other environments.
NHC (Node Health Check) can be used within Slurm, and is often useful to run within prolog or epilog scripts to verify node functionality. CoreWeave Kubernetes HPC verification workflows run similar checks that trigger un-drain of older NHC failures. The Syncer identifies drains that can be undrained through the presence of verify-undrain anywhere in the node’s drain reason. When this string is present, the Syncer checks the HPCVerification condition of the Pod to see if a newer verification pass has happened since the drain.

​
Extra field

Nodes in Slurm have an Extra field that can be used to store user-specified information. SUNK uses this to store information that provides visibility within Slurm to conditions on the Kubernetes side. The Syncer manages updates to the Extra field to reflect the information. The information is stored as JSON in the Extra field to allow for parsing and manipulation.
Users can add more information into the Extra field in Slurm. The Extra field must contain valid JSON, or the contents are cleared and replaced at the next synchronization. When there are conflicts, JSON fields set by the Syncer overwrite those set by the user.

​
Hook API

The hook API provided by the Syncer lets events in Slurm directly trigger operations within Kubernetes. Some of these hooks facilitate blocking synchronization or immediate actions. The Syncer provides several hooks for node objects, described in the following sections.

​
Pre-hook

The pre-hook endpoint ensures that other jobs running outside Slurm on the Node are removed before the Slurm jobs start. It also begins the state propagation that triggers the Pod Controller and Node Controller to perform further actions.

​
Reboot

This endpoint is only available when the Syncer has permissions to perform operations on the Nodes. The Syncer node permissions are set with the Slurm chart option .syncer.nodePermissions.enabled.
The reboot endpoint reboots the Kubernetes Node associated with a Slurm node. By default, this endpoint sets the PhaseState condition on the Node along with the associated reason production-powerreset, which then triggers other Node management tooling to reboot the Node. To modify the condition type and associated reason, use .syncer.hooksAPI.nodeRebootCondition and .syncer.hooksAPI.nodeRebootReason.

​
Metrics

The Syncer provides a scrapeable metrics endpoint, which exposes metrics for the nodes, jobs, and the overall Slurm cluster. The PodMonitor deployed with the SUNK chart labels all metrics with their associated Slurm cluster using the slurm_cluster label. The Syncer also exports additional metrics for the standard Go runtime and controller runtime. Labels added by the code are shown in the following list. More labels can be added by the scrape configuration. The Syncer applies the following labels:
  • account: Slurm account name
  • id: Slurm job ID
  • name: Slurm job name
  • node: Slurm node name
  • partition: Slurm partition
  • state: Slurm job current state
  • user: Slurm user name
  • message_type: Slurm RPC message type
MetricUnitLabelsDescription
slurm_controller_rpc_countcountmessage_typeRPC count per message type.
slurm_controller_rpc_mean_duration_secondssecondsmessage_typeRPC mean duration per message type.
slurm_job_state—partition,
account,
user,
id,
name,
state		Current state of the job represented in the state label.
slurm_job_cpus_allocatedcountpartition,
account,
user,
id,
node		The number of CPUs allocated to a job by Slurm node. Only present for running jobs.
slurm_job_gpus_allocatedcountpartition,
account,
user,
id,
node		The number of GPUs allocated to a job by Slurm node. Only present for running jobs.
slurm_job_uptime_secondssecondspartition,
account,
user,
id,
node		The number of seconds a job has been running. Only present for running jobs.
slurm_jobs_pendingcountpartition,
account,
user		The number of pending jobs in the Slurm cluster.
slurm_jobs_runningcountpartition,
account,
user		The number of running jobs in the Slurm cluster.
slurm_jobs_suspendedcountpartition,
account,
user		The number of suspended jobs in the Slurm cluster.
slurm_node_state—node
,partition
,state		Current state of the node represented in the state label.
slurm_node_cpu_alloccountnodeThe number of CPUs allocated per node.
slurm_node_cpu_idlecountnodeThe number of CPUs idle per node.
slurm_node_cpu_totalcountnodeThe total number of CPUs per node.
slurm_node_mem_allocMBnodeThe amount of allocated memory per node.
slurm_node_mem_totalMBnodeThe total amount of memory per node.
slurm_node_gpu_alloccountnodeThe number of GPUs allocated per node.
slurm_node_gpu_idlecountnodeThe number of GPUs idle per node.
slurm_node_gpu_totalcountnodeThe total number of GPUs per node.
slurm_nodes_alloccount—The number of nodes with state allocated.
slurm_nodes_compcount—The number of nodes with state completing.
slurm_nodes_downcount—The number of nodes with state down.
slurm_nodes_draincount—The number of nodes with state drain.
slurm_nodes_errcount—The number of nodes with state error.
slurm_nodes_failcount—The number of nodes with state fail.
slurm_nodes_idlecount—The number of nodes with state idle.
slurm_nodes_maintcount—The number of nodes with state maintenance.
slurm_nodes_mixcount—The number of nodes with state mix.
slurm_nodes_resvcount—The number of nodes with state reserved.
slurm_nodes_totalcount—The total number of nodes.
slurm_nodes_not_respondingcount—The number of nodes with state not_responding.
slurm_partition_cpu_alloccountpartitionThe number of CPUs allocated in a partition.
slurm_partition_cpu_idlecountpartitionThe number of CPUs idle in a partition.
slurm_partition_cpu_totalcountpartitionThe total number of CPUs in a partition.
slurm_partition_mem_allocMBpartitionThe amount of allocated memory in a partition.
slurm_partition_mem_totalMBpartitionThe total memory in a partition.
slurm_partition_gpu_alloccountpartitionThe number of GPUs allocated in a partition.
slurm_partition_gpu_idlecountpartitionThe number of GPUs idle in a partition.
slurm_partition_gpu_totalcountpartitionThe total number of GPUs in a partition.
slurm_queue_canceledcount—The number of canceled jobs in the Slurm cluster (only those still tracked by slurmctld).
slurm_queue_completedcount—The number of completed jobs in the Slurm cluster (only those still tracked by slurmctld).
slurm_queue_completingcount—The number of completing jobs in the Slurm cluster.
slurm_queue_configuringcount—The number of configuring jobs in the Slurm cluster.
slurm_queue_failedcount—The number of failed jobs in the Slurm cluster (only those still tracked by slurmctld).
slurm_queue_node_failcount—The number of jobs stopped due to node failure in the cluster (only those still tracked by slurmctld).
slurm_queue_pendingcount—The number of pending jobs in the Slurm scheduler queue.
slurm_queue_pending_dependencycount—The number of pending jobs in the Slurm scheduler queue with unsatisfied dependencies.
slurm_queue_preemptedcount—The number of preempted jobs in the Slurm cluster (only those still tracked by slurmctld).
slurm_queue_runningcount—The number of running jobs in the Slurm cluster.
slurm_queue_suspendedcount—The number of suspended jobs in the Slurm cluster.
slurm_queue_timeoutcount—The number of timed out jobs in the Slurm cluster (only those still tracked by slurmctld).
slurm_scheduler_backfill_cycle_last_secondsseconds—The duration of the last scheduler backfill cycle.
slurm_scheduler_backfill_cycle_mean_secondsseconds—The mean duration of the scheduler backfill cycles.
slurm_scheduler_backfill_depth_meancount—The mean depth of the scheduler backfill.
slurm_scheduler_backfilled_jobs_totalcount—The number of jobs started due to backfilling since last Slurm start.
slurm_scheduler_backfilled_jobs_cycle_totalcount—The number of jobs started due to backfilling since last time stats were reset.
slurm_scheduler_backfilled_jobs_heterogeneous_totalcount—The number of heterogeneous jobs started due to backfilling since last Slurm start.
slurm_scheduler_cycle_last_secondsseconds—The duration of the last scheduler cycle.
slurm_scheduler_cycle_mean_secondsseconds—The mean duration of the scheduler cycles.
slurm_scheduler_cycles_per_minuteopm—The number of scheduler cycles per minute.
slurm_scheduler_dbd_queuecount—The number of items in the scheduler dbd agent queue.
slurm_scheduler_jobs_submittedcount—The number of submitted jobs reported by the scheduler.
slurm_scheduler_jobs_startedcount—The number of jobs started by the scheduler.
slurm_scheduler_jobs_completedcount—The number of jobs completed by the scheduler.
slurm_scheduler_jobs_failedcount—The number of jobs failed by the scheduler.
slurm_scheduler_jobs_cancelledcount—The number of jobs canceled by the scheduler.
slurm_scheduler_jobs_pendingcount—The number of jobs pending in the scheduler queue.
slurm_scheduler_jobs_runningcount—The number of jobs currently running in the scheduler.
slurm_scheduler_queuecount—The number of items in the scheduler queue.
slurm_scheduler_threadscount—The number of scheduler threads.
slurm_scheduler_cycle_mean_depthcount—The mean depth of the scheduler cycles.
Last modified on May 27, 2026