Skip to main content

Use SUNK to Deploy Slurm on CoreWeave Cloud

Quickstart guide

This quickstart guide explains the basic steps to deploy a Slurm cluster with SUNK on CoreWeave Cloud with Helm.

note

This guide does not include steps to configure directory services or other advanced options, and is not intended for production deployment. See the Best Practices section to learn more about using Slurm in production.

Prerequisites​

First, verify your CoreWeave account has full Kubernetes access and the local workstation is configured to use the desired namespace.

Next, verify SUNK is installed in the cluster by checking for the presence of the SUNK CRDs with kubectl:

$ kubectl get crds | grep sunk
nodesets.sunk.coreweave.com                                 2023-03-22T17:20:40Z
nodeslices.sunk.internal.coreweave.com                      2023-08-09T00:01:23Z
slurmclusters.sunk.coreweave.com                            2023-09-15T18:12:27Z

If the CRDs are not present, contact CoreWeave support to enable SUNK in the cluster.

Next, create the PVCs for Shared Storage volumes. Copy the following YAML to example-pvcs.yaml on the local workstation.

example-pvcs.yaml
# High-performance NVMe volume is shared between the Slurm login and compute nodes
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-nvme
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 64Gi
  storageClassName: shared-nvme
---
# Home directory for the root user is useful if not using LDAP
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: root-nvme
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 64Gi
  storageClassName: shared-nvme
## Optional: a bulk storage volume using the `shared-hdd`.
# ---
#apiVersion: v1
#kind: PersistentVolumeClaim
#metadata:
#  name: data-hdd
#spec:
#  accessModes:
#    - ReadWriteMany
#  resources:
#    requests:
#      storage: 256Gi
#  storageClassName: shared-hdd

This manifest creates two Shared Storage volumes, data-nvme and root-nvme, with 64Gi of storage each. Some workloads may benefit from other storage classes, such as HDD bulk storage volumes, as shown in the commented lines of the example.

Next, deploy the PVCs with kubectl.

$ kubectl apply -f example-pvcs.yaml
persistentvolumeclaim/data-nvme created
persistentvolumeclaim/root-nvme created

Adjust the Helm values​

Create a values-cw.yaml file that configures the Slurm cluster with a minimal configuration.

values-cw.yaml
compute:
  nodes:
    simple:
      enabled: true
      replicas: 2
      definitions:
        - epyc
        - standard
      # Create a small node with 1cpu and 1g memory
      resources:
        limits:
          memory: 1Gi
          cpu: 1
        requests:
          memory: 1Gi
          cpu: 1
  mounts:
    - name: /mnt/nvme
      pvc: data-nvme
    - name: /mnt/hdd
      pvc: data-hdd

The compute.nodes section creates two CPU-only Compute nodes with one CPU and 1Gi of memory each.

The compute.mounts section mounts the Shared Storage volumes created earlier. The data-nvme volume is mounted to /mnt/nvme to create a shared area for sbatch scripts and other utilities. The root-nvme volume is mounted to /root for the root user's home directory, which is useful for Slurm deployments that don't use LDAP. These are also automatically mounted to the Login nodes.

Deploy Slurm with Helm​

Ensure that Helm is installed on the local workstation, then add the CoreWeave repository. Bitnami is required for the MySQL database used by Slurm.

$ helm repo add coreweave http://helm.corp.ingress.ord1.coreweave.com

Install Slurm with Helm, supplying your configuration using the values-cw.yaml file.

$ helm install slurm coreweave/slurm --values values-cw.yaml

Verify Slurm is ready​

Open a second terminal to view the Pods and check the status with kubectl get pods. The Slurm resources should be ready within 5 to 10 minutes.

$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
slurm-accounting-6854d9694f-fzl7x   4/4     Running   0          5m30s
slurm-controller-8667dd8675-9rspz   5/5     Running   0          5m29s
slurm-login-0                       4/4     Running   0          5m30s
slurm-mysql-0                       2/2     Running   0          5m30s
slurm-rest-5759666cc6-wbd62         4/4     Running   0          5m30s
slurm-simple-048-41                 4/4     Running   0          5m30s
slurm-simple-052-48                 4/4     Running   0          5m30s
slurm-syncer-69d57785bf-4rcd2       1/1     Running   0          5m30s
note

The slurm-syncer Pod has dependencies on other Pods in the Slurm cluster. It's normal to see these Kubernetes Pods restart several times while Slurm deploys, as can be seen in the "Ready" column above.

The Pods are:

  • slurm-accounting-* - The Slurm accounting daemon.
  • slurm-controller-* - The Slurm controller daemon.
  • slurm-login-0 - The Slurm login node.
  • slurm-mysql-0 - The MySQL database used by Slurm.
  • slurm-rest-* - The Slurm REST API.
  • slurm-simple-* - The Slurm compute nodes.
  • slurm-syncer-* - Syncs the cluster state between Slurm and Kubernetes.

See the Slurm documentation to learn more about how to use these components. Use kubectl exec to access the Login node as root.

$ kubectl exec -it slurm-login-0 -c sshd -- bash

Best Practices​

This guide describes how to deploy a minimal Slurm cluster on CoreWeave Cloud. In a production environment, you should adhere to these best practices.

Authentication services​

An authentication service is necessary for production deployments. SUNK supports LDAP services such as Authentik, OpenLDAP, Okta, and Active Directory. You can deploy OpenLDAP on CoreWeave Cloud by navigating to the Applications -> Catalog tab and searching OpenLDAP.

GPU Quotas​

Usually, the GPU quota should equal eight times the number of requested compute nodes: (compute nodes * 8). Request a quota increase by navigating to the Upgrade Quotas tab in CoreWeave Cloud. For further help contact CoreWeave support.