Kueue
Run Kueue on CKS
Chart reference | Description |
---|---|
coreweave/cks-kueue | CoreWeave's Helm chart for deploying Kueue on CKS clusters |
About Kueue
Kueue is a Kubernetes-native system that manages jobs using quotas. Kueue makes job decisions based on resource availability, job priorities, and the quota policies defined in your cluster queues. Kueue can determine when a job should wait for available resources, when a job should start (Pods created), and when a job should be preempted (active Pods deleted).
CoreWeave's cks-kueue
Chart includes a kueue
subchart, which is used to deploy Kueue to your CKS (CoreWeave Kubernetes Service) cluster.
When you install Kueue through our Helm chart, Kueue metrics are automatically scraped and ingested into Grafana.
Usage
Add the CoreWeave Helm repo.
$helm repo add coreweave https://charts.core-services.ingress.coreweave.com
Then, install Kueue on your CKS cluster.
$helm install kueue coreweave/cks-kueue --namespace=kueue-system --create-namespace
Sample Kueue configuration
After installing the cks-kueue
chart, use the following sample configuration to set up a basic Kueue environment for CKS. This configuration includes several key Kueue components:
ResourceFlavor
: Defines the characteristics of compute resources (CPU, memory, GPUs) available in your clusterClusterQueue
: Establishes resource quotas and admission policies across your entire clusterLocalQueue
: Creates namespaced queues that reference a ClusterQueue for job submissionWorkloadPriorityClass
: Defines priority levels for jobs to determine scheduling order and preemption behavior
The configuration also defines two priority classes for different job types: production jobs with high priority and development jobs with lower priority.
# ResourceFlavor defines the compute resources available in your cluster# This flavor represents the standard CKS node configuration---apiVersion: kueue.x-k8s.io/v1beta1kind: ResourceFlavormetadata:name: default-flavor---# ClusterQueue establishes resource quotas and admission policies# This queue allows jobs to consume up to the specified resource limitsapiVersion: kueue.x-k8s.io/v1beta1kind: ClusterQueuemetadata:name: "cluster-queue"spec:# Enable preemption of lower priority jobs when higher priority jobs need resourcespreemption:withinClusterQueue: LowerPriority# Allow jobs from all namespaces to use this queuenamespaceSelector: {} # Match all namespaces.resourceGroups:- coveredResources: ["cpu", "memory", "nvidia.com/gpu", "rdma/ib"]flavors:- name: "default-flavor"resources:- name: "cpu"nominalQuota: 254 # Total CPU cores available- name: "memory"nominalQuota: 2110335488Ki # Total memory available (~2TB)- name: "nvidia.com/gpu"nominalQuota: 16 # Total GPUs available- name: "rdma/ib"nominalQuota: 12800 # Total RDMA bandwidth available---# LocalQueue creates a namespaced queue for job submission# Jobs submitted to this queue will use the cluster-queue resourcesapiVersion: kueue.x-k8s.io/v1beta1kind: LocalQueuemetadata:namespace: "default"name: "default"spec:clusterQueue: "cluster-queue"---# WorkloadPriorityClass defines priority levels for job scheduling# Higher values = higher priority (jobs with higher priority can preempt lower priority jobs)apiVersion: kueue.x-k8s.io/v1beta1kind: WorkloadPriorityClassmetadata:name: prod-priorityvalue: 1000description: "Priority class for prod jobs"---apiVersion: kueue.x-k8s.io/v1beta1kind: WorkloadPriorityClassmetadata:name: dev-priorityvalue: 100description: "Priority class for development jobs"---