SUNK Pod Scheduler integration

Documentation Index

Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt

Use this file to discover all available pages before exploring further.

​

Prerequisites

• A CKS cluster with SUNK deployed and the SUNK Pod Scheduler enabled.
• A CoreWeave Sandbox Runner deployed in your CKS cluster.
• Version requirements:
  • Runner image: >=v0.31.0
  • cwsandbox-client: >=0.10.0

​

Step 1: Verify the SUNK Pod Scheduler

1. Verify the scheduler is running:
   kubectl get pods -n tenant-slurm -l app.kubernetes.io/name=sunk-scheduler
   
   If no pods are returned, enable the scheduler in your Slurm Helm values by setting scheduler.enabled: true. See Schedule Kubernetes pods with the SUNK Pod Scheduler for details.
2. Verify the scheduler name. The name is tenant-slurm-slurm-scheduler in most deployments. Confirm by running:
   kubectl get pods -n tenant-slurm -l app.kubernetes.io/name=sunk-scheduler -o yaml \
     | grep -o "\-\-scheduler-name=[^ ]*"
   
   Note the value after the = sign.
3. Verify the scheduler scope. The SUNK Pod Scheduler must be able to watch the namespaces where sandbox pods are created. Check your scheduler.scope.type setting in the Slurm Helm values. See Check the scheduler scope for configuration details. If the scope is set to cluster, the scheduler watches all namespaces and no additional configuration is needed. If the scope is set to namespace, the scheduler only watches specific namespaces. In that case, either add the sandbox namespaces to scheduler.scope.namespaces, or set the profile’s namespace strategy to static with a fixed namespace that the scheduler already watches (for example, slurm-sandboxes). Using a static namespace also simplifies sandbox pod and placeholder job discoverability.
4. Lower slurmd resource requests. The default NodeSet resource requests consume most of the node’s allocatable capacity in Kubernetes, leaving no room for sandbox pods. Add a low-requests compute definition to the NodeSets you want to use with sandboxes so that Kubernetes has capacity to schedule sandbox pods alongside Slurm jobs. Without this change, sandbox pods are rejected with OutOfMemory or OutOfcpu errors. See Manage resources for configuration details.

​

Step 2: Add a SUNK profile to the Runner

profileConfig:
  # ... other profiles ...
  slurm:
    tags:
      - sunk
    pod:
      spec:
        schedulerName: "tenant-slurm-slurm-scheduler"
        terminationGracePeriodSeconds: 24
    namespace:
      strategy: per-user
      autoCreate: true
    network:
      egress:
        default: internet
        modes:
          internet:
            type: internet
          none:
            type: none

• schedulerName tells Kubernetes to hand the pod to the SUNK Pod Scheduler instead of the default scheduler.
• terminationGracePeriodSeconds must be less than Slurm’s KillWait setting minus 5 seconds. The default KillWait is 30 seconds, so 24 is a safe value. To check your cluster’s KillWait value:
  kubectl get configmap -n tenant-slurm tenant-slurm-slurm-conf -o yaml | grep KillWait
  
  Alternatively, check the live Slurm configuration from a Slurm login node:
  # After SSH'ing into your Slurm cluster
  scontrol show config | grep KillWait
  
• tags add the sunk label to sandbox pods on this profile.
• namespace creates a separate namespace per user for sandbox pods.
• network defines the available egress modes. Sandboxes default to internet egress.

​

Add the profile to your values

tags:
  - sunk
pod:
  spec:
    schedulerName: "tenant-slurm-slurm-scheduler"
    terminationGracePeriodSeconds: 24
namespace:
  strategy: per-user
  autoCreate: true
network:
  egress:
    default: internet
    modes:
      internet:
        type: internet
      none:
        type: none

yq -i '.profileConfig.slurm = load("slurm-profile.yaml")' sandbox-tower-values.yaml

​

Deploy the updated Runner

helm upgrade sandbox-tower coreweave/sandbox-tower \
  --namespace sandbox-system \
  -f sandbox-tower-values.yaml

kubectl logs -n sandbox-system -l app.kubernetes.io/name=sandbox-tower --tail=5

​

Step 3: Create sandboxes

from cwsandbox import Sandbox

with Sandbox.run(profile_ids=["slurm"]) as sb:
    print(f"Sandbox ID: {sb.sandbox_id}")
    result = sb.exec(["hostname"]).result()
    print(result.stdout)

# After SSH'ing into your Slurm cluster
sacct --format=JobID,JobName%60,State,NodeList -X | grep [SANDBOX-ID]

from cwsandbox import Sandbox, SandboxDefaults

defaults = SandboxDefaults(profile_ids=("slurm",))

with Sandbox.session(defaults) as session:
    sb1 = session.sandbox()
    sb2 = session.sandbox()

    r1 = sb1.exec(["hostname"]).result()
    r2 = sb2.exec(["hostname"]).result()
    print(f"sb1: {r1.stdout.strip()}, sb2: {r2.stdout.strip()}")

​

Resource requests and Slurm accounting

​

Control placement with Slurm annotations

sb = Sandbox.run(
    profile_ids=["slurm"],
    annotations={
        "sunk.coreweave.com/partition": "hpc-prod",
    },
)

​

Common annotations

​

Enforce annotations in the profile configuration

pod:
  metadata:
    annotations:
      sunk.coreweave.com/partition: "sandboxes"

​

Match the Slurm user from a training job

import os
from cwsandbox import Sandbox, SandboxDefaults

slurm_uid = os.environ["SLURM_JOB_UID"]

defaults = SandboxDefaults(
    profile_ids=("slurm",),
    annotations={
        "sunk.coreweave.com/user-id": slurm_uid,
    },
)

with Sandbox.session(defaults) as session:
    sb = session.sandbox()
    result = sb.exec(["hostname"]).result()
    print(result.stdout)

​

Troubleshooting

​

Nodes temporarily in completing state

​

Sandboxes not landing where expected

# After SSH'ing into your Slurm cluster
sacct --format=JobID,JobName%60,Partition,Account,AllocCPUS,ReqMem,State,NodeList,Start,End -X \
  | grep [SANDBOX-ID]

​

See also

• About CoreWeave sandboxes
• Deploy CoreWeave Sandbox Runner
• Schedule Kubernetes pods with the SUNK Pod Scheduler
• RL training with sandboxes
• Sandbox configuration