Skip to main content

Use SUNK to Deploy Slurm on CoreWeave Cloud

Quickstart guide

This quickstart guide explains the basic steps to deploy a Slurm cluster with SUNK on CoreWeave Cloud with Helm.


This guide does not include steps to configure directory services or other advanced options, and is not intended for production deployment. See the Best Practices section to learn more about using Slurm in production.


First, verify your CoreWeave account has full Kubernetes access and the local workstation is configured to use the desired namespace.

Next, verify SUNK is installed in the cluster by checking for the presence of the SUNK CRDs with kubectl:

$ kubectl get crds | grep sunk 2023-03-22T17:20:40Z 2023-08-09T00:01:23Z 2023-09-15T18:12:27Z

If the CRDs are not present, contact CoreWeave support to enable SUNK in the cluster.

Next, create the PVCs for Shared Storage volumes. Copy the following YAML to example-pvcs.yaml on the local workstation.

# High-performance NVMe volume is shared between the Slurm login and compute nodes
apiVersion: v1
kind: PersistentVolumeClaim
name: data-nvme
- ReadWriteMany
storage: 64Gi
storageClassName: shared-nvme
# Home directory for the root user is useful if not using LDAP
apiVersion: v1
kind: PersistentVolumeClaim
name: root-nvme
- ReadWriteMany
storage: 64Gi
storageClassName: shared-nvme
## Optional: a bulk storage volume using the `shared-hdd`.
# ---
#apiVersion: v1
#kind: PersistentVolumeClaim
# name: data-hdd
# accessModes:
# - ReadWriteMany
# resources:
# requests:
# storage: 256Gi
# storageClassName: shared-hdd

This manifest creates two Shared Storage volumes, data-nvme and root-nvme, with 64Gi of storage each. Some workloads may benefit from other storage classes, such as HDD bulk storage volumes, as shown in the commented lines of the example.

Next, deploy the PVCs with kubectl.

$ kubectl apply -f example-pvcs.yaml
persistentvolumeclaim/data-nvme created
persistentvolumeclaim/root-nvme created

Adjust the Helm values

Create a values-cw.yaml file that configures the Slurm cluster with a minimal configuration.

enabled: true
replicas: 2
- epyc
- standard
# Create a small node with 1cpu and 1g memory
memory: 1Gi
cpu: 1
memory: 1Gi
cpu: 1
- name: /mnt/nvme
pvc: data-nvme
- name: /mnt/hdd
pvc: data-hdd

The compute.nodes section creates two CPU-only Compute nodes with one CPU and 1Gi of memory each.

The compute.mounts section mounts the Shared Storage volumes created earlier. The data-nvme volume is mounted to /mnt/nvme to create a shared area for sbatch scripts and other utilities. The root-nvme volume is mounted to /root for the root user's home directory, which is useful for Slurm deployments that don't use LDAP. These are also automatically mounted to the Login nodes.

Deploy Slurm with Helm

Ensure that Helm is installed on the local workstation, then add the CoreWeave repository. Bitnami is required for the MySQL database used by Slurm.

$ helm repo add coreweave

Install Slurm with Helm, supplying your configuration using the values-cw.yaml file.

$ helm install slurm coreweave/slurm --values values-cw.yaml

Verify Slurm is ready

Open a second terminal to view the Pods and check the status with kubectl get pods. The Slurm resources should be ready within 5 to 10 minutes.

$ kubectl get pods
slurm-accounting-6854d9694f-fzl7x 4/4 Running 0 5m30s
slurm-controller-8667dd8675-9rspz 5/5 Running 0 5m29s
slurm-login-0 4/4 Running 0 5m30s
slurm-mysql-0 2/2 Running 0 5m30s
slurm-rest-5759666cc6-wbd62 4/4 Running 0 5m30s
slurm-simple-048-41 4/4 Running 0 5m30s
slurm-simple-052-48 4/4 Running 0 5m30s
slurm-syncer-69d57785bf-4rcd2 1/1 Running 0 5m30s

The slurm-syncer Pod has dependencies on other Pods in the Slurm cluster. It's normal to see these Kubernetes Pods restart several times while Slurm deploys, as can be seen in the "Ready" column above.

The Pods are:

  • slurm-accounting-* - The Slurm accounting daemon.
  • slurm-controller-* - The Slurm controller daemon.
  • slurm-login-0 - The Slurm login node.
  • slurm-mysql-0 - The MySQL database used by Slurm.
  • slurm-rest-* - The Slurm REST API.
  • slurm-simple-* - The Slurm compute nodes.
  • slurm-syncer-* - Syncs the cluster state between Slurm and Kubernetes.

See the Slurm documentation to learn more about how to use these components. Use kubectl exec to access the Login node as root.

$ kubectl exec -it slurm-login-0 -c sshd -- bash

Best Practices

This guide describes how to deploy a minimal Slurm cluster on CoreWeave Cloud. In a production environment, you should adhere to these best practices.

Authentication services

An authentication service is necessary for production deployments. SUNK supports LDAP services such as Authentik, OpenLDAP, Okta, and Active Directory. You can deploy OpenLDAP on CoreWeave Cloud by navigating to the Applications -> Catalog tab and searching OpenLDAP.

GPU Quotas

Usually, the GPU quota should equal eight times the number of requested compute nodes: (compute nodes * 8). Request a quota increase by navigating to the Upgrade Quotas tab in CoreWeave Cloud. For further help contact CoreWeave support.