Schedule Kubernetes Pods with the SUNK Pod Scheduler

​

Enable the scheduler

scheduler:
  enabled: true

• To monitor the entire cluster, set scheduler.scope.type to cluster.
• To monitor specific namespaces, set scheduler.scope.type to namespace and set scheduler.scope.namespaces to a comma-separated list of namespaces. If scheduler.scope.namespaces is blank, the scheduler defaults to the namespace where it’s deployed.

​

Configure a Pod for scheduling

​

Set the scheduler name

​

Set resource requests

​

Set the termination grace period

​

Look up the scheduler configuration

kubectl get pods -l app.kubernetes.io/name=sunk-scheduler -oyaml | yq '.items[0].spec.containers[] | select(.name == "scheduler").args'

- --scheduler-name=tenant-slurm-staging-slurm-scheduler
- --slurm-kill-wait=30s

​

Example Pod

apiVersion: v1
kind: Pod
metadata:
  name: slurm-test
  namespace: tenant-slurm-staging
  annotations:
    sunk.coreweave.com/account: root
    sunk.coreweave.com/comment: "A Kubernetes job"
    sunk.coreweave.com/exclusive: "none"
spec:
  schedulerName: tenant-slurm-staging-slurm-scheduler
  containers:
    - name: main
      image: ubuntu
      command:
        - /bin/bash
      args:
        - -c
        - sleep infinity
      resources:
        limits:
          memory: 960Gi
          rdma/ib: "1"
          nvidia.com/gpu: "8"
        requests:
          cpu: "10"
          memory: 10Gi
          nvidia.com/gpu: "8"
  terminationGracePeriodSeconds: 5
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: gpu.nvidia.com/model
                operator: In
                values:
                  - H100_NVLINK_80GB

​

Choose a node sharing strategy

​

Annotations reference

​

Time limits

​

Known limitations

• No gang scheduling. The scheduler schedules each Pod as a separate Slurm job. Multi-node PodGroups aren’t supported. The scheduler is best suited for single-node workloads such as inference.
• No bin-packing. Slurm doesn’t fill partially used nodes before moving to idle ones. This can spread Pods across nodes and lead to GPU fragmentation, especially during scaling.
• Kubernetes scheduling features aren’t supported. podAffinity, podAntiAffinity, and topologySpreadConstraints have no effect. Slurm makes all node placement decisions. Node-level affinity (such as gpu.nvidia.com/model) is supported.
• Static CPU allocation causes node drains. A Pod triggers Kubernetes static CPU pinning only when it has Guaranteed QoS (CPU requests equal to limits) and a whole-number CPU request. Pinning conflicts with Slurm’s CPU accounting and creates resource contention, which leads to node drains in Slurm. Breaking either condition prevents it. For details and prevention steps, see Static CPU allocation and the SUNK Pod Scheduler.
• Non-SUNK Pods are invisible to Slurm. Slurm can’t see Pods scheduled through the standard Kubernetes scheduler. Running non-DaemonSet Pods on Slurm nodes without the SUNK Pod Scheduler can cause resource conflicts and unexpected node drains.
• Taints may conflict with Slurm placement. The SUNK Pod Scheduler doesn’t evaluate Kubernetes taints. If Slurm places a Pod on a node with a conflicting taint, the kubelet rejects the Pod. Check Pod events with kubectl describe pod if a Pod is stuck.
• Scale-down reconfigure delay. When nodes are removed from the cluster, there’s about a one-minute delay before Slurm’s configuration updates. During this window, Slurm may try to schedule work onto nodes that are being removed.
• Only a fixed set of exclusive values is valid. On SUNK v5.7.0 and later, the sunk.coreweave.com/exclusive annotation accepts only none, ok, user, mcs, or topo. Any other non-empty value makes the placeholder Slurm job submission fail, so the Pod appears briefly and then vanishes. See Exclusive annotation values.
• GPU Pods can collide with Slurm GRES under open sharing. Setting exclusive: "ok" on a GPU Pod opens the node to any job and can collide with Slurm’s GRES accounting. Use exclusive: "none" for full-node GPU Pods or exclusive: "user" with a dedicated user-id for partial-GPU Pods.
• Admission mismatch causes OutOfcpu or OutOfmemory. Slurm and Kubernetes account for resources independently. System Pods, DaemonSets, and the slurmd container consume Kubernetes-allocatable capacity that Slurm doesn’t see, so the kubelet can reject a Pod that Slurm placed. Right-size Pod requests below node allocatable. See Reserve resources for system Pods.
• Duplicate volume declarations break mounts. Declaring the same PersistentVolumeClaim (PVC) as two separate volume entries in one Pod spec, a common mistake with LeaderWorkerSet and templated workloads, can silently break subPath mounts. Declare each PVC once in volumes and reference it with multiple volumeMounts.

​

Troubleshoot a SUNK-scheduled Pod

​

Next steps

• Manage resources with the SUNK Pod Scheduler for node sharing configuration, resource tuning, and GPU allocation guidance.