Configure Node allocation budgets

Node allocation budgets control how much of each Node’s allocatable CPU and memory sandboxes can request. This guide shows cluster administrators how to configure and enable budgets for all Nodes, selected Node Pools, or groups of Nodes. For example, a Node has 32 allocatable CPUs and 128 GiB of allocatable memory. A 50% budget allows aggregate sandbox requests of 16 CPUs and 64 GiB. Percentages apply to resource requests, rather than measured utilization. Existing sandboxes aren’t evicted when a budget decreases.

CoreWeave sandboxes are in public preview. For access, contact your CoreWeave account team, CoreWeave Support, or email support@coreweave.com.

Before you begin

Before you configure budgets, complete the following prerequisites:

Confirm with CoreWeave that Node allocation budget support and its required permissions are enabled for your cluster.
Apply any required managed runner update and wait for the runner to become ready and connected.
Use CoreWeave Intelligent CLI v1.44.0 or later, authenticated to the correct organization with Sandbox administrator permissions.
Install jq.
Have Kubernetes access to inspect Node labels and published budgets. Before you decrease a budget, you also need permission to cordon and uncordon the affected Nodes.
Use the default Kubernetes scheduler, or confirm that your scheduler enforces the budget extended resources.
If sandboxes select a RuntimeClass, confirm with your cluster administrator that it declares a nonempty overhead.podFixed map. A runtime with no overhead must declare explicit zero values. Sandboxes without a RuntimeClass don’t require this setting.

CoreWeave Intelligent CLI v1.44.0 supports the environment-only workflow in this guide. A runner update doesn’t enable budgets until you apply the configuration. All logical runners sharing the same physical Kubernetes cluster must use the same budget configuration and mode. Separate clusters can use different configurations.

Identify the runner and Node labels

Identify the runner to update and the Node labels to use in your budget selectors. List available runners:

cwic sandbox runner get

Optional: Filter by cluster. Replace [CLUSTER-ID] with your CoreWeave Kubernetes Service (CKS) cluster ID:

cwic sandbox runner get --cluster "[CLUSTER-ID]"

Choose the runner and Kubernetes context. Replace [RUNNER-ID] with the runner ID and [KUBE-CONTEXT] with its Kubernetes context:

export RUNNER_ID="[RUNNER-ID]"
export KUBE_CONTEXT="[KUBE-CONTEXT]"

Inspect Node Pool and hostname labels:

kubectl --context "$KUBE_CONTEXT" get nodes \
  -L compute.coreweave.com/node-pool,kubernetes.io/hostname

CKS Nodes can expose these labels:

Label	Purpose
`compute.coreweave.com/node-pool`	Select Nodes by Node Pool name
`compute.coreweave.com/node-pool-uid`	Select a particular Node Pool identity
`kubernetes.io/hostname`	Select individual Nodes
A custom label	Select an arbitrary group of Nodes

Use the label keys and values present in your cluster. A budget selector selects Kubernetes Nodes. It doesn’t select API runner groups or override sandbox placement constraints, taints, tolerations, or RuntimeClasses.

Create a budget configuration

Save one of the following examples as the node-budget.json file. Start with "mode": "prepare". Preparation publishes budget resources. Enforcement is enabled in a later step.

95% CPU and memory on every Node

An empty selector matches all Nodes.

{
  "version": 1,
  "mode": "prepare",
  "managedNodes": {
    "selector": {},
    "defaultBudget": {
      "cpuPercent": 95,
      "memoryPercent": 95
    }
  }
}

Both percentages are required and must be integers from 0 through 100. A value of 0 means zero sandbox capacity.

50% on selected Node Pools only

This selects Nodes in either of two Node Pools. Replace [BATCH-POOL] and [INTERACTIVE-POOL] with their Node Pool label values:

{
  "version": 1,
  "mode": "prepare",
  "managedNodes": {
    "selector": {
      "matchExpressions": [
        {
          "key": "compute.coreweave.com/node-pool",
          "operator": "In",
          "values": [
            "[BATCH-POOL]",
            "[INTERACTIVE-POOL]"
          ]
        }
      ]
    },
    "defaultBudget": {
      "cpuPercent": 50,
      "memoryPercent": 50
    }
  }
}

Nodes outside managedNodes.selector receive zero sandbox budget. Once enforcement is enabled, sandboxes that require budget tokens can’t use those Nodes. Preview the selection:

kubectl --context "$KUBE_CONTEXT" get nodes \
  -l 'compute.coreweave.com/node-pool in ([BATCH-POOL],[INTERACTIVE-POOL])'

To preserve sandbox capacity on other Nodes, use Node Pool and group overrides.

Different budgets for Node Pools and custom groups

This configuration provides the following budgets:

95% CPU and memory by default.
50% CPU and memory for Nodes in the [CPU-POOL] Node Pool.
25% CPU and 40% memory for Nodes labeled [BUDGET-GROUP-LABEL]=[TRAINING-GROUP].

Replace [CPU-POOL] with the Node Pool label value. Replace [BUDGET-GROUP-LABEL] with a custom label key and [TRAINING-GROUP] with its value.

{
  "version": 1,
  "mode": "prepare",
  "managedNodes": {
    "selector": {},
    "defaultBudget": {
      "cpuPercent": 95,
      "memoryPercent": 95
    },
    "overrides": [
      {
        "name": "shared-cpu",
        "selector": {
          "matchLabels": {
            "compute.coreweave.com/node-pool": "[CPU-POOL]"
          }
        },
        "budget": {
          "cpuPercent": 50,
          "memoryPercent": 50
        }
      },
      {
        "name": "training",
        "selector": {
          "matchLabels": {
            "[BUDGET-GROUP-LABEL]": "[TRAINING-GROUP]"
          }
        },
        "budget": {
          "cpuPercent": 25,
          "memoryPercent": 40
        }
      }
    ]
  }
}

Selectors have the following behavior:

A matching override replaces the default percentages.
If multiple overrides match, the minimum percentage in each dimension wins.
Rule order doesn’t establish priority.
Overlap is reported as selector_conflict. Prefer disjoint groups when overlap is unintended.
Entries in matchLabels and separate expressions must all match.
Values within one In expression are alternatives.

In this example, a Node matching both overrides receives 25% CPU and 40% memory. For custom groups, configure labels through your Node Pool configuration where possible so replacement Nodes inherit them.

50% on specific Nodes

Replace [HOSTNAME-A] and [HOSTNAME-B] with the observed kubernetes.io/hostname label values for the Nodes you want to select:

{
  "version": 1,
  "mode": "prepare",
  "managedNodes": {
    "selector": {
      "matchExpressions": [
        {
          "key": "kubernetes.io/hostname",
          "operator": "In",
          "values": [
            "[HOSTNAME-A]",
            "[HOSTNAME-B]"
          ]
        }
      ]
    },
    "defaultBudget": {
      "cpuPercent": 50,
      "memoryPercent": 50
    }
  }
}

Hostname label values may differ from Node object names. Other Nodes receive zero sandbox budget. Node Pool or custom-group selectors are preferable when replacement Nodes should inherit the configuration automatically.

Apply the configuration

Store the configuration as a JavaScript Object Notation (JSON) string in the following field:

spec.overrides.env.SANDBOX_NODE_BUDGET_CONFIG

Select the configuration file to apply:

export BUDGET_FILE=node-budget.json

The following commands read the current runner and preserve its existing deployment override fields and environment variables. You don’t need the original runner configuration or policy file. Reuse this block after changing BUDGET_FILE to apply a different configuration:

umask 077

# cwic returns an array, even when fetching one runner.
cwic sandbox runner get "$RUNNER_ID" -o json > current-runner.json

jq --slurpfile budget "$BUDGET_FILE" '
  if length != 1 then
    error("expected one runner")
  else
    .[0]
  end |
  {
    spec: {
      overrides: (
        (.spec.overrides // {}) |
        .env = (
          (.env // {}) + {
            SANDBOX_NODE_BUDGET_CONFIG: ($budget[0] | tojson)
          }
        )
      )
    }
  }
' current-runner.json > node-budget-patch.json

cwic sandbox runner edit "$RUNNER_ID" -f node-budget-patch.json
cwic sandbox runner describe "$RUNNER_ID"

Repeat this process for every logical runner in the same physical cluster. Read each runner before you construct its patch. spec.overrides is replaced as an object, so preserving its sibling fields is necessary. For CoreWeave Intelligent CLI v1.44.0, use this environment-only patch. That release’s older policy schema can’t fully export the renamed HTTPS hostname fields from newer servers. If you reapply a full runner document, you could therefore replace its policy with incomplete data. The preceding patch doesn’t include or update the policy. Keep runner exports private and remove temporary files afterward:

rm current-runner.json node-budget-patch.json

Verify preparation

Preparation publishes these extended resources on Node status:

sandbox.coreweave.com/cpu-millicores
sandbox.coreweave.com/memory-bytes

Inspect their capacity and allocatable values:

kubectl --context "$KUBE_CONTEXT" get nodes -o json | jq '
  .items[] | {
    node: .metadata.name,
    cordoned: (.spec.unschedulable // false),
    physical: {
      cpu: .status.allocatable.cpu,
      memory: .status.allocatable.memory
    },
    budgetCapacity: {
      cpuMillicores:
        .status.capacity["sandbox.coreweave.com/cpu-millicores"],
      memoryBytes:
        .status.capacity["sandbox.coreweave.com/memory-bytes"]
    },
    budgetAllocatable: {
      cpuMillicores:
        .status.allocatable["sandbox.coreweave.com/cpu-millicores"],
      memoryBytes:
        .status.allocatable["sandbox.coreweave.com/memory-bytes"]
    }
  }'

Wait for capacity and allocatable to agree for both budget resources on every Node. Budget accounting includes the following details:

Existing bound sandboxes without budget tokens are subtracted from the published budget until they finish.
Other workloads consume physical resources, so available sandbox capacity can be lower than the configured budget.
Sandbox charges include helper containers, init-container peaks, and RuntimeClass overhead.
CPU and memory percentages don’t divide GPU capacity.

Where metrics are available, check the following metrics:

sandbox_runner_node_budget_ready
sandbox_runner_node_budget_writer
sandbox_runner_node_budget_nodes{state="converged"}
sandbox_runner_node_budget_policy

The sandbox_runner_node_budget_policy metric exposes mode and policy_hash labels. Check that every runner in the cluster reports mode="prepare" and the same policy_hash value. If you don’t have access to these metrics, ask CoreWeave to verify readiness before enabling enforcement.

Enable enforcement

After preparation is healthy, change only the mode:

jq '.mode = "enforce"' node-budget.json > node-budget-enforce.json
export BUDGET_FILE=node-budget-enforce.json

Repeat the read/build-patch/apply commands from Apply the configuration for every runner in the cluster. Use cwic sandbox runner describe "$RUNNER_ID" to check the runner rollout, and ask CoreWeave to confirm the cluster admission transition is complete. This transition coordinates when the cluster starts enforcing budgets. It includes retiring old shards and allowing pending workloads from before enforcement to bind or terminate. To verify enforcement, complete the following steps:

Create a small sandbox on each runner.
Confirm its Pod requests both budget resources.
If you perform a controlled saturation test, verify that new allocations stop when either the CPU or memory budget is exhausted.
Delete the test sandboxes.

The following capacity-exhaustion reason is expected:

CWSANDBOX_RUNNER_CAPACITY_EXHAUSTED

Configure Terraform-managed runners

If Terraform manages the runner, merge the following spec.overrides.env setting into its existing coreweave_sandbox_managed_runner resource:

spec = {
  overrides = {
    env = {
      # Preserve other configured environment variables.
      SANDBOX_NODE_BUDGET_CONFIG = file("${path.module}/node-budget.json")
    }

    # Preserve other configured override fields.
  }

  # Preserve other configured spec fields.
}

After preparation is healthy, update the JSON file from prepare to enforce and apply the corresponding Terraform change. If Terraform manages the runner, keep changes in Terraform so subsequent applies retain the intended configuration.

Reduce budgets or change Node membership

Nodes aren’t automatically cordoned or drained. Before you reduce percentages or change selectors or labels in a way that reduces capacity, complete the following steps:

Cordon the affected Nodes.
Apply the updated configuration to every runner in that cluster.
Wait for budget convergence.
Uncordon only the Nodes you cordoned for this change.

Cordon prevents new scheduling and leaves existing Pods running. Existing allocations can remain above the new ceiling until they finish. A budget reduction doesn’t evict them. An uncordoned reduction is logged as the following message:

decrease_without_cordon

To disable budgets entirely, coordinate the retirement procedure with CoreWeave. Removing the environment variable or disabling the feature gate directly doesn’t perform the required rollback sequence.

​Before you begin

​Identify the runner and Node labels

​Create a budget configuration

​95% CPU and memory on every Node

​50% on selected Node Pools only

​Different budgets for Node Pools and custom groups

​50% on specific Nodes

​Apply the configuration

​Verify preparation

​Enable enforcement

​Configure Terraform-managed runners

​Reduce budgets or change Node membership

​See also

Before you begin

Identify the runner and Node labels

Create a budget configuration

95% CPU and memory on every Node

50% on selected Node Pools only

Different budgets for Node Pools and custom groups

50% on specific Nodes

Apply the configuration

Verify preparation

Enable enforcement

Configure Terraform-managed runners

Reduce budgets or change Node membership

See also