Integrate sandboxes with the SUNK Pod Scheduler

Documentation Index

Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt

Use this file to discover all available pages before exploring further.

​

Prerequisites

• A CKS cluster with SUNK deployed and the SUNK Pod Scheduler enabled.
• A CoreWeave sandbox runner deployed on the same cluster.
• The CoreWeave Intelligent CLI (cwic), installed and authenticated. See Deploy and manage a runner for setup details.
• cwsandbox-client >= 0.10.0 for client-side annotation support.

​

Step 1: Verify the SUNK Pod Scheduler

1. Verify the scheduler is running:
   kubectl get pods -n tenant-slurm -l app.kubernetes.io/name=sunk-scheduler
   
   If no pods are returned, enable the scheduler in your Slurm Helm values by setting scheduler.enabled: true. See Enable the scheduler for details.
2. Look up the scheduler name and KillWait value. You will reuse both when you author the profile in Step 2:
   kubectl get pods -n tenant-slurm -l app.kubernetes.io/name=sunk-scheduler -o yaml \
     | yq '.items[0].spec.containers[] | select(.name == "scheduler").args'
   
   Look for --scheduler-name and --slurm-kill-wait in the output:
   - --scheduler-name=tenant-slurm-slurm-scheduler
   - --slurm-kill-wait=30s
   
   See Look up the scheduler configuration for the underlying behavior.
3. Confirm the scheduler scope covers the sandbox namespaces. The SUNK Pod Scheduler must be able to watch the namespaces where sandbox pods are created. Check your scheduler.scope.type setting in the Slurm Helm values. If the scope is cluster, the scheduler watches all namespaces and no additional configuration is needed. If the scope is namespace, the scheduler only watches the namespaces you list. You have two ways forward:
   • Pin sandboxes to a known namespace. Set the profile’s namespace strategy to static with a fixed namespace (for example, sandbox-slurm) and add that namespace to scheduler.scope.namespaces. This is the simplest path and makes sandbox pods and placeholder jobs easy to find. See Choose a namespace strategy.
   • Let the profile create per-user namespaces. Pick a per-user or per-profile strategy with a recognizable namespacePrefix (for example, sb-), then list the namespaces that match the prefix after the first sandbox runs:
     kubectl get namespaces -l sandbox.coreweave.com/profile-id
     
     Add those namespaces (or the prefix pattern) to scheduler.scope.namespaces and re-roll Slurm.
4. Lower slurmd resource requests on NodeSets you want to share with sandboxes. This is a change on the Slurm side, in your Slurm Helm values, not in any sandbox configuration. The default NodeSet resource requests consume most of the node’s allocatable capacity in Kubernetes, leaving no room for sandbox pods. Without this change, sandbox pods are rejected with OutOfMemory or OutOfcpu errors. See Manage resources with the SUNK Pod Scheduler for configuration details.

​

Step 2: Create a SUNK-aware profile

display_name: slurm
description: SUNK-scheduled sandboxes that run as Slurm jobs
spec:
  namespace:
    strategy: per-user
  network:
    egress:
      default: internet
      modes:
        internet:
          type: internet
        none:
          type: none
  pod:
    spec:
      schedulerName: "tenant-slurm-slurm-scheduler"
      terminationGracePeriodSeconds: 20

• spec.pod.spec.schedulerName tells Kubernetes to hand the pod to the SUNK Pod Scheduler instead of the default Kubernetes scheduler.
• spec.pod.spec.terminationGracePeriodSeconds must be strictly less than --slurm-kill-wait minus 5 seconds. The Kubernetes default of 30 seconds exceeds the default 25-second threshold, so you must set this explicitly. See Set the termination grace period for the underlying rule.
• spec.namespace chooses how sandbox pods are grouped into namespaces. per-user gives each user their own namespace; static pins every sandbox to a fixed namespace. See Choose a namespace strategy.
• spec.network declares the outbound modes sandboxes can pick. The example above exposes internet (default) and none. See Configure egress.

cwic sandbox profile create -f slurm-profile.yaml

curl -X POST https://api.coreweave.com/v1beta2/sandbox/profile-templates \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "profileTemplate": {
      "displayName": "slurm",
      "description": "SUNK-scheduled sandboxes that run as Slurm jobs",
      "spec": {
        "namespace": { "strategy": "per-user" },
        "network": {
          "egress": {
            "default": "internet",
            "modes": {
              "internet": { "type": "internet" },
              "none":     { "type": "none" }
            }
          }
        },
        "pod": {
          "spec": {
            "schedulerName": "tenant-slurm-slurm-scheduler",
            "terminationGracePeriodSeconds": 20
          }
        }
      }
    }
  }'

profile_bindings:
  - profile_template_id: "[EXISTING-DEFAULT-PROFILE-ID]"
    profile_name: default
    is_default: true
  - profile_template_id: "[SLURM-PROFILE-ID]"
    profile_name: slurm

cwic sandbox runner edit [RUNNER-ID] -f runner-bindings.yaml

curl -X PATCH https://api.coreweave.com/v1beta2/sandbox/managed-runners/[RUNNER-ID] \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "runner": {
      "id": "[RUNNER-ID]",
      "profileBindings": [
        { "profileTemplateId": "[EXISTING-DEFAULT-PROFILE-ID]", "profileName": "default", "isDefault": true },
        { "profileTemplateId": "[SLURM-PROFILE-ID]",            "profileName": "slurm" }
      ]
    },
    "updateMask": "profileBindings"
  }'

​

Step 3: Create sandboxes

from cwsandbox import Sandbox

with Sandbox.run(runway_ids=["slurm"]) as sb:
    print(f"Sandbox ID: {sb.sandbox_id}")
    result = sb.exec(["hostname"]).result()
    print(result.stdout)

sacct --format=JobID,JobName%60,State,NodeList -X | grep [SANDBOX-ID]

from cwsandbox import Sandbox, SandboxDefaults

defaults = SandboxDefaults(runway_ids=("slurm",))

with Sandbox.session(defaults) as session:
    sb1 = session.sandbox()
    sb2 = session.sandbox()

    r1 = sb1.exec(["hostname"]).result()
    r2 = sb2.exec(["hostname"]).result()
    print(f"sb1: {r1.stdout.strip()}, sb2: {r2.stdout.strip()}")

​

Resource requests and Slurm accounting

requests={"cpu": "500m", "memory": "512Mi"},
limits={"cpu": "2",     "memory": "2Gi"},

​

Step 4: Control placement with Slurm annotations

sb = Sandbox.run(
    runway_ids=["slurm"],
    annotations={
        "sunk.coreweave.com/partition": "hpc-prod",
    },
)

​

Common annotations

id -u           # your own UID
id -u [USERNAME]  # someone else's UID
whoami          # confirm your username first if needed

​

Enforce annotations in the profile

spec:
  pod:
    metadata:
      annotations:
        sunk.coreweave.com/partition: "sandboxes"

​

Match the Slurm user from a training job

import os
from cwsandbox import Sandbox, SandboxDefaults

slurm_uid = os.environ["SLURM_JOB_UID"]

defaults = SandboxDefaults(
    runway_ids=("slurm",),
    annotations={
        "sunk.coreweave.com/user-id": slurm_uid,
    },
)

with Sandbox.session(defaults) as session:
    sb = session.sandbox()
    result = sb.exec(["hostname"]).result()
    print(result.stdout)

​

Troubleshooting

​

Placeholder jobs temporarily in completing state

​

Sandboxes not landing where expected

sacct --format=JobID,JobName%60,Partition,Account,AllocCPUS,ReqMem,State,NodeList,Start,End -X \
  | grep [SANDBOX-ID]

​

See also

• About CoreWeave sandboxes
• Deploy and manage a runner
• Configure a sandbox profile
• Schedule Kubernetes pods with the SUNK Pod Scheduler
• RL training with sandboxes
• Sandbox configuration