Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt

Use this file to discover all available pages before exploring further.

The SlurmCluster CRD is responsible for configuring and synchronizing Slurm’s cluster-wide information. Currently, this creates the topology files used by Slurm, if network topology is enabled. In the future, this may consolidate some of the Slurm cluster’s control-plane deployment to simplify the Helm chart and provide some cluster status information via the CRD.

​
Topology ConfigMap generation

Slurm supports using a topology plugin, which helps the Slurm Scheduler run related jobs on nodes that are closer for better performance. The topology is defined in topology.conf. The controller generates this file automatically based on Node labels. It’s saved into a ConfigMap and mounted into containers, so that it can be updated without restarting the Pods. Only scontrol reload is needed to apply the changes to the Slurm cluster. Because the topology file is cluster-wide, duplicate entries are not possible. As a result, the topology cannot be generated on a per-NodeSet basis. The SlurmCluster Controller generates the topology based on Node labels that describe the network topology. SUNK supports two topology plugins: tree and block.

​
Changing the topology plugin

The topology.conf can contain both topology definitions at the same time (if labels for both are present on Nodes), but only one topology plugin can be active at a time. The default is tree topology. To use block topology, add TopologyPlugin=topology/block to slurmConfig.extraConfig in the Helm chart values.

​
Tree topology

The current implementation expects an InfiniBand-style network structure. The Node labels used are as follows:
  • ib.coreweave.cloud/fabric - Indicates the top level IB fabric the node belongs to in order to support multiple fabrics
  • ib.coreweave.cloud/superpod - The superpod within an IB fabric the node is a part of
  • ib.coreweave.cloud/leafgroup - The leafgroup within the IB superpod the node is a part of
  • node.coreweave.cloud/datahall - The datahall the node is located in, used to make the leafgroup value unique. This label is optional.
The fabric and superpod labels are expected to be unique within the cluster. If datahall is not used, then leafgroup value must also be unique within the cluster. If datahall is used, the combined datahall + leafgroup value must be unique within the cluster. For example, this is appropriate:
  • F1 -> SP1 -> LG1, LG2
  • F1 -> SP2 -> LG3, LG4
  • F1 -> SP3 -> DH1-LG3, DH2-LG3
While this is not:
  • F1 -> SP1 -> LG1, LG2
  • F1 -> SP2 -> LG1, DH1-LG2
  • F1 -> SP3 -> DH1-LG2, DH2-LG3
Only Nodes that have all the required labels can be pulled in correctly. If a Node is missing the required topology labels, it will be omitted from the generated topology.

​
Block topology

The block topology implementation is designed for NVlink systems, with one block per NVlink domain. One block is generated for each unique ds.coreweave.com/nvlink.domain label value on the Nodes. The block size is set to the size of the largest block. If a block has fewer than 2 nodes, no block topology configuration is emitted.
Last modified on March 24, 2026