Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt

Use this file to discover all available pages before exploring further.

Kubernetes is a powerful container orchestration platform that manages the underlying infrastructure for SUNK, including compute resources, networking, and storage. SUNK leverages Kubernetes to provide a scalable, flexible, and high-performance computing environment for running Slurm workloads. When using SUNK, you can interact with Kubernetes using the kubectl command-line tool. This guide explains the key concepts, benefits, and best practices for using kubectl to interact with Kubernetes in the context of SUNK.

​
Key concepts

First, some terminology and concepts to understand the relationship between Kubernetes and Slurm in the context of SUNK:
TermDescription
Kubernetes cluster and NodesA Kubernetes cluster is a collection of Kubernetes Nodes, which are (in CKS) physical machines that run Kubernetes components and containerized applications.
Kubernetes Nodes are capitalized as proper nouns.
Kubernetes PodA Kubernetes Pod is the smallest deployable unit in Kubernetes, representing a single instance of a running process, such as a Slurm node. Multiple Pods can run on a single Kubernetes Node.
Slurm cluster and nodesA Slurm cluster is a collection of Slurm nodes, where each node is a Kubernetes Pod running a slurmd container.
Slurm nodes are lowercase.
kubectlkubectl is the command-line tool for interacting with Kubernetes clusters. It allows you to inspect cluster resources, create, delete, and update objects, and view logs and events.
When you use a Slurm cluster deployed by SUNK, you’re operating within a Kubernetes environment. This means the underlying infrastructure is managed by Kubernetes, and many aspects of SUNK’s operation can be observed and controlled using kubectl, the command-line tool for Kubernetes.

​
Benefits

Here are some reasons why you might want to interact with Kubernetes via kubectl:
  • Visibility: You can use kubectl to see the status of the Kubernetes Pods where your Slurm jobs are running, providing insight into the underlying execution environment. Additionally, SUNK is deployed using Helm charts, which are managed by Kubernetes. kubectl can be used to inspect these deployments.
  • Debugging: If you encounter issues, kubectl can help you inspect logs, events, and the state of the Pods, aiding in troubleshooting.
  • Familiarity: If you already have experience with Kubernetes, using kubectl to observe SUNK offers a familiar way to interact with the system.
  • Configuration: Many aspects of SUNK’s configuration are managed as Kubernetes resources (such as ConfigMaps and Secrets), which you can interact with using kubectl.

​
Access Logs

Because Slurm daemons run within pods, their logs can be viewed using the kubectl logs command, optionally with the -f flag to follow the logs in real time. To access the logs of the Slurm Controller, which manages job submissions and scheduling, use this command: 
kubectl logs -f -l app.kubernetes.io/name=slurmctld -c slurmctld
To see what’s happening on a specific Slurm compute node, list the slurmd logs for that Pod. The name of a Slurm node matches the name of its corresponding Kubernetes Pod, so you can get the logs by substituting the node’s name for [POD-NAME] in the following: 
kubectl logs -f -c slurmd [POD-NAME]

​
Restart the Slurm Controller

If your jobs are stuck in a pending state, you may need to restart the Slurm Controller. To restart the Slurm Controller, first find the name of the Controller deployment: 
kubectl get deployments -l app.kubernetes.io/component=controller
Now that you have the deployment name, use the following command to restart it: 
kubectl rollout restart deployment [CONTROLLER-DEPLOYMENT-NAME]
Replace [CONTROLLER-DEPLOYMENT-NAME] with the name you found earlier. You can confirm that the restart is happening by checking the status of the rollout: 
kubectl rollout status deployment [CONTROLLER-DEPLOYMENT-NAME]
Restarting the Controller won’t cancel active jobs, but could fix problems involving jobs that are stuck pending. It’s a safe operation to perform when troubleshooting.

​
Restart the Slurm daemon

To restart the slurmd daemon, delete the Kubernetes Pod running the Slurm node. Restarting the Kubernetes Pod will remove the node from the Slurm pool of nodes, and re-register it when the Pod is recreated. To find the name of the Kubernetes Pod running the Slurm node, use the get pod command as follows: 
kubectl -n tenant-slurm get pod -o wide [SLURM-NODE-NAME]
To delete the Pod, use the delete pod command, replacing [POD-NAME] with the name provided by the get pod command: 
kubectl -n tenant-slurm delete pod [POD-NAME]
Last modified on April 20, 2026