Can I run workloads directly on the login node?
No. The login node is a shared resource for submitting jobs, managing files, and running lightweight commands. Compute-h …
Can I submit Slurm jobs from Kubernetes Pods using slurmrestd?
Yes. SUNK can deploy slurmrestd as an in-cluster Service so authenticated Pods can submit jobs through its REST API inst …
How do I fix torch.distributed rendezvous and TCPStore failures?
Your torch.distributed job fails during startup or rendezvous with a c10d or TCPStore error such as sendBytes failed … …
How do I run containers with Pyxis and Enroot in Slurm?
Pyxis and Enroot are NVIDIA tools that let you run OCI-compatible container images inside Slurm jobs. SUNK includes Pyxi …
How do I set up the SUNK Pod Scheduler to share Nodes between Pods and Slurm jobs?
The SUNK Pod Scheduler routes Kubernetes Pod placement through Slurm, so Pods and Slurm jobs can share the same nodes. C …
How do I submit my first training job?
After SSH-ing into the login node, use srun for interactive jobs and sbatch for batch scripts. For a quick smoke test, r …
What do NCCL errors mean and how do I diagnose them?
Most NCCL errors aren’t NCCL bugs. The message you see usually names the layer where NCCL gave up, not the layer that fa …
Why are CUDA toolkit libraries missing in a SUNK container?
SUNK compute nodes provide NVIDIA GPU drivers to Pyxis containers. They don’t install the full CUDA toolkit. Libraries s …
Why are Slurm commands not found inside a Pyxis container?
Custom container images don’t include Slurm CLI tools. sbatch, srun, squeue, sinfo, sacct, and scancel work on the login …
Why do sudo, chown, and chmod fail inside a Pyxis container?
By default, Enroot remaps container root to your real user ID. That isolation is expected on SUNK, and it’s why sudo, ta …
Why does Pyxis report File already exists after a job is preempted?
After a job is preempted or requeued, Pyxis can fail to start with ERROR File already exists if a leftover container dir …
Why does my SUNK-scheduled Kubernetes Pod fail to start?
A SUNK-scheduled Pod fails to start when Slurm and Kubernetes account for a node’s resources independently, or when a sc …
Why is my Slurm job stuck in PENDING?
A Slurm job stays in PENDING (PD) when the scheduler has accepted the job but can’t start it yet. Run squeue -u USER and …