Responsibilities for third-party schedulers
CoreWeave manages platform components in thecw-* and other Control Plane namespaces, including the GPU device plugin, the Container Network Interface (CNI), Node health automation, and HPC Verification. CoreWeave also manages SUNK and the cks-kueue chart.
You own each third-party scheduler integration. That mean you install, configure, operate, and monitor it. When you use a third-party scheduler, you must:
- Deploy and maintain the scheduler.
- Pin scheduler container images to a specific tag or digest. Don’t use
latest. - Configure multi-node GPU workloads for InfiniBand. See Use GPUDirect RDMA.
- Use instance-type and CoreWeave topology labels for GPU and topology-aware placement. See Node type selection labels.
- Install and operate the observability stack that the scheduler requires in your namespaces.
default-scheduler continues to run because CoreWeave platform components depend on it.
Third-party schedulers and HPC Verification
CoreWeave runs HPC Verification health checks on idle Nodes at the top of each hour. Verification Pods use thecw-hpc-verification PriorityClass, which has a value of -1, so the Kubernetes default-scheduler preempts them as soon as one of your workloads schedules onto the Node.
That preemption works only when the scheduler that placed the verification Pods is the same scheduler that places your workloads. Because verification Pods are submitted through the default-scheduler, a third-party scheduler doesn’t treat them as candidates for preemption. This affects Volcano and the KAI Scheduler (including Run:ai).
This conflict produces the following symptoms:
- Verification Pods that linger on Nodes you expect to use.
- Workloads that stay Pending with insufficient-GPU messages during the hourly verification window.
- Topology fit failures, because verification Pods appear to consume capacity your scheduler can’t reclaim.
Make HPC Verification Pods preemptible
Scheduler-side configuration alone doesn’t resolve this. CoreWeave applies a per-cluster change to how verification Pods are submitted and labeled, so that your scheduler can preempt them. The change depends on which scheduler you run and on how you configured it, so CoreWeave builds it for your cluster rather than applying a standard patch. To run a third-party scheduler on a production cluster, do the following:- Install and configure your scheduler, including any preemption settings it requires. See the following scheduler-specific sections.
- To have verification Pods made preemptible by your scheduler, contact CoreWeave Support with your cluster ID, the scheduler name and version, and its configuration.
- After CoreWeave confirms the change, verify that one of your workloads preempts a running
hpc-verification-*Pod on an otherwise idle Node.
Avoid .spec.nodeName
Setting .spec.nodeName bypasses the scheduler entirely, including any custom scheduler you install. The Pod is then invisible to Kueue and to topology-aware scheduling. Use a nodeSelector or nodeAffinity to target Nodes instead. See Workload scheduling for more information about avoiding .spec.nodeName.
Volcano on CKS
Volcano’spreempt action lets it preempt lower-priority Pods, and it isn’t enabled by default. To turn it on, edit the existing volcano-scheduler-configmap ConfigMap in the namespace where you installed Volcano. Add preempt to the actions list in the volcano-scheduler.conf key:
default-scheduler. After you update the ConfigMap, contact CoreWeave Support to have verification Pods made preemptible by Volcano, then verify that one of your workloads preempts a running hpc-verification-* Pod on an otherwise idle Node.
Run:ai and the KAI Scheduler on CKS
The KAI Scheduler is NVIDIA’s open source scheduler derived from Run:ai. Like Volcano, it doesn’t preempt Pods placed by thedefault-scheduler, so your workloads can’t reclaim Nodes from verification Pods. Workloads submitted during the verification window report that the Node has insufficient GPU resources.
No scheduler-side configuration resolves this. As with Volcano, CoreWeave applies the per-cluster change described in Third-party schedulers and HPC Verification. For the KAI Scheduler, that change references the queue and the PriorityClass that your installation uses, so CoreWeave builds it from your configuration instead of applying a standard remedy. Contact CoreWeave Support before you run the KAI Scheduler on a production cluster, and include your cluster ID, the scheduler version, and your queue configuration.
SkyPilot on CKS
SkyPilot orchestrates jobs but doesn’t provide its own Kubernetes scheduler. The cluster’s scheduler places the Pods that SkyPilot submits, which on CKS is thedefault-scheduler unless you also run Kueue. For setup and worked examples, see Run SkyPilot on CKS.
A common SkyPilot misdiagnosis is reporting that Nodes disappeared when SkyPilot redeployed the cluster. Before you suspect a hardware or capacity fault, determine whether SkyPilot tore down and recreated the resources, and check the Node Pool status to confirm capacity is intact. See Node Pool status.
For multi-node jobs, configure InfiniBand as described in Run SkyPilot on CKS. Without it, multi-node jobs can fall back to Ethernet and run slowly.
Anyscale on CKS
To run Ray and Anyscale workloads on CKS, see Anyscale and Run Ray with Kueue.When to open a support ticket
Open a ticket with CoreWeave Support in the following situations:- You plan to run a third-party scheduler on a production cluster.
- Your workloads don’t preempt verification Pods after CoreWeave applies the per-cluster change.
- A third-party scheduler interacts with CoreWeave platform automation in a way you can’t resolve from your configuration.
Related pages
- Workload scheduling on CKS: labels, taints, and the default scheduler.
- Scheduling profiles: the CKS scheduling profiles.
- Kueue: CoreWeave’s supported Kueue distribution.
- HPC Verification: how health checks interact with your scheduler.