Run a GPU Pod with Kata Containers

Container runtime selection is in private preview. Contact Support to request access before using a non-default runtime. Backwards compatibility between ncore releases isn’t guaranteed. An installed runtime doesn’t establish production support.

Use the cw-kata-nvidia-gpu RuntimeClass to run a GPU workload inside a QEMU virtual machine (VM) on CoreWeave Kubernetes Service (CKS). This guide deploys a CUDA vector-addition sample with the guest metrics agent required by CKS.

Prerequisites

Before you begin, confirm that you have the following:

A CKS cluster with cluster access configured and kubectl installed.
Kubernetes permissions to create Node Pools, create and delete workload Pods, inspect Nodes and RuntimeClasses, list Pods in cw-kata-system, and read workload Pod status, events, and logs.
A Node Pool approved by CoreWeave for GPU Kata testing that meets the platform requirements in the runtime catalog.
CoreWeave enablement of the Kata controllers, the cw-kata-nvidia-gpu RuntimeClass, and workload metrics scraping in your cluster.

Confirm that the class exists:

kubectl get runtimeclass cw-kata-nvidia-gpu -o yaml

If the class is absent, ask CoreWeave to confirm cluster enablement before continuing.

Configure a Node Pool

Configure a Node Pool for virtualization workloads:

Prepare the Node Pool manifest

Prepare a new Node Pool manifest following Create a Node Pool using Kubernetes. Use the instance type approved by CoreWeave for GPU Kata.

Enable virtualization and apply the manifest

Add virt: "true" under spec.nodeParameters in that manifest. The following is an excerpt, not a complete Node Pool manifest:

spec:
  nodeParameters:
    virt: "true"

This setting initializes the Nodes for virtualization workloads. Save the complete manifest in a file named gpu-kata-nodepool.yaml and apply it:

kubectl apply -f gpu-kata-nodepool.yaml

Check for GPU metrics placeholder Pods

After the Nodes join the cluster, check for GPU metrics placeholder Pods:

kubectl get pods -n cw-kata-system -o wide

The Kata metrics operator schedules one kata-metrics-placeholder-* Pod per unused GPU. These Pods provide GPU metrics while no workload uses the GPU. Resolve missing or unhealthy placeholder Pods with CoreWeave before deploying the workload.

Deploy the GPU Pod

Create and apply a Pod manifest that includes the GPU workload and guest metrics agent:

Save the Pod manifest

Save the following manifest in a file named gpu-kata-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd-kata
  namespace: default
  labels:
    kata.coreweave.com/telemetry: "true"
spec:
  runtimeClassName: cw-kata-nvidia-gpu
  restartPolicy: Never
  initContainers:
    - name: cks-metrics-agent
      restartPolicy: Always
      image: images.coreweave.com/cluster-images/cw-cks-metrics-agent-oci/cks-metrics-agent:v1.1.1
      command: ["/bin/sh", "-c"]
      args:
        - |-
          mknod -m 0600 /dev/kmsg c 1 11 2>/dev/null || true
          exec cks-metrics-agent --web.listen-address=:9400
      env:
        - name: CKS_COLLECT_INTERVAL
          value: 15s
        - name: VISIBLE_CDI_DEVICES
          value: nvidia.com/gpu=all
        - name: NVIDIA_DRIVER_CAPABILITIES
          value: all
      ports:
        - name: metrics
          containerPort: 9400
      startupProbe:
        httpGet:
          path: /readyz
          port: metrics
        periodSeconds: 5
        failureThreshold: 12
      livenessProbe:
        httpGet:
          path: /healthz
          port: metrics
        initialDelaySeconds: 15
        periodSeconds: 20
  containers:
    - name: cuda-vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      resources:
        requests:
          cpu: "1000m"
          memory: 10Gi
          hugepages-1Gi: 20Gi
          nvidia.com/gpu: "1"
        limits:
          cpu: "1000m"
          memory: 10Gi
          hugepages-1Gi: 20Gi
          nvidia.com/gpu: "1"

The RuntimeClass supplies virtualization Node selection, a matching hypervisor taint toleration, and scheduler memory overhead. If your Node Pool has additional taints, add only the tolerations required for that pool. See Workload scheduling.The Pod label kata.coreweave.com/telemetry: "true" identifies the workload for GPU metrics scraping. CoreWeave enables the scrape configuration in your preview cluster.The metrics agent runs as a native sidecar without additional privileges or explicit resource requests. Its startup probe waits for the agent’s HTTP server before starting the CUDA container. VISIBLE_CDI_DEVICES gives the agent access to the GPU assigned to the workload. Keep this sidecar in GPU Kata workloads so CKS can monitor GPU health. HTTP probe success doesn’t verify GPU collection or delivery to CKS monitoring.The sample requests one GPU, 10Gi of container memory, and 20Gi of 1 GiB hugepages. For details about how these values relate, see GPU Kata resource requirements.

Apply the Pod manifest

kubectl apply -f gpu-kata-pod.yaml

Verify the workload

Check the sample’s status and output, then remove the sample Pod:

Check the Pod's status and events

kubectl get pod cuda-vectoradd-kata -n default
kubectl describe pod cuda-vectoradd-kata -n default

Read the CUDA container's logs

kubectl logs cuda-vectoradd-kata -n default -c cuda-vectoradd

A successful run includes:

Test PASSED
Done

With restartPolicy: Never, the sample can reach a terminal phase. The native sidecar stops after the regular container exits.

Inspect the metrics sidecar's logs

Inspect the metrics sidecar’s logs and work with CoreWeave to confirm that GPU metrics reach CKS monitoring:

kubectl logs cuda-vectoradd-kata -n default -c cks-metrics-agent

Expected output resembles the following. Timestamps and metric counts can vary:

{"time":"2026-10-26T16:45:55.561357032Z","level":"INFO","msg":"DCGM collector initialized","collector":"dcgm","gpus":1,"metrics":63,"labels":3}
{"time":"2026-10-26T16:45:55.565333559Z","level":"INFO","msg":"gpu-thermal collector initialized","collector":"gpu-thermal","gpus":1,"gpu_class":"unknown"}
{"time":"2026-10-26T16:45:55.61560785Z","level":"INFO","msg":"NVLinks discovered","collector":"nvlink","count":0,"present":false}
{"time":"2026-10-26T16:45:55.666989657Z","level":"INFO","msg":"GPUs discovered","collector":"nvlink","count":1}
{"time":"2026-10-26T16:45:55.717091688Z","level":"INFO","msg":"starting cks-metrics-agent","collectors":["dcgm","gpu-thermal","nvlink","xid"],"interval":15000000000,"metrics":":9400/metrics"}
{"time":"2026-10-26T16:45:55.717257716Z","level":"INFO","msg":"listening","address":":9400"}
{"time":"2026-10-26T16:45:59.840393561Z","level":"INFO","msg":"shutting down"}

In this sample, the agent discovers one GPU and starts its metrics endpoint. The sidecar shuts down after the CUDA container exits. These logs show agent initialization; work with CoreWeave to confirm delivery to CKS monitoring.

Delete the sample Pod

After verification, delete the sample Pod:

kubectl delete pod cuda-vectoradd-kata -n default

Troubleshoot startup

Symptom	What to check
The Pod stays `Pending`.	Confirm that the Node Pool is ready and compatible with the runtime, and that the Nodes have enough GPU, memory, and hugepage capacity. Inspect the Pod events for scheduling failures.
`FailedCreatePodSandBox` with `DeadlineExceeded`	Check the hugepage requests and limits. Missing or insufficient hugepages can prevent QEMU from starting and appear as a timeout.
The CUDA container doesn’t start.	Inspect the `cks-metrics-agent` sidecar and its startup probe in the Pod events.

For unresolved failures, provide CoreWeave with the Pod events and container logs.

Create and manage clusters

Configure cluster access

Deploy workloads

Secure your cluster

Tutorials

Reference

Run a GPU Pod with Kata Containers

Prerequisites

Configure a Node Pool

Deploy the GPU Pod

Verify the workload

Troubleshoot startup

​Prerequisites

​Configure a Node Pool

​Deploy the GPU Pod

​Verify the workload

​Troubleshoot startup

Prerequisites

Configure a Node Pool

Deploy the GPU Pod

Verify the workload

Troubleshoot startup