Access Slurm compute nodes

Slurm jobs run on compute nodes, but the shell you submit from lives on the login node. When a job goes wrong, you often need to get onto the node where it’s running and inspect it: check GPU state, read a log the job never flushed, or run a command inside the container the job started. This guide covers the three ways to do that on a SUNK cluster, and when each one applies:

srun --overlap opens a shell on a node in an allocation you already hold.
srun --container-name=[NAME]:exec attaches to the container a job is already running, rather than starting a new one.
SSH through the login node with ProxyJump gives you a plain shell on the node, which is what tooling such as rsync and remote debuggers expect.

srun --overlap and the container-attach method are scoped to your own allocation. SSH isn’t. Unless an administrator has configured pam_slurm_adopt, a provisioned key opens any compute node, whether or not you hold a job on it. See SSH access on compute nodes.

Use compute node access to inspect and debug jobs you already hold an allocation for. Don’t use it to run work outside Slurm. Processes started outside a job escape Slurm’s resource accounting, interfere with running jobs, and can drain the node, taking it out of service for everyone. See Compute and login nodes.

Prerequisites

An active job allocation on the cluster. Every method on this page requires one.
Access to a Slurm login node. See Connect to the Slurm Login node.
For the container attach method, SUNK v7.x or later. Earlier versions carry a bug that can prevent Pyxis from finding a running container; SUNK v7.x includes a fix. See the SUNK v7.0.0 release notes. To find your cluster’s version, read spec.sunkVersion on the SunkCluster resource of a self-service cluster, or the chart version that helm list -n [NAMESPACE] reports for a Helm chart deployment.
For the SSH method, sshd running on the compute nodes. CoreWeave clusters run it by default, on both Helm chart and self-service deployments, so this usually needs no action. See SSH access on compute nodes.

Choose an access method

The following table matches each task to the method that fits it, along with what that method requires.

You want to	Use	Needs
Run a command or open a shell on a node in your job	`srun --overlap`	Nothing beyond an allocation
Run a command inside the container your job is already running	`srun --overlap --container-name=[NAME]:exec`	SUNK v7.x, a named container, the right node
Use SSH-based tooling such as `rsync`, `scp`, or a remote debugger	`ssh -J` through the login node	`sshd` on the compute nodes, which is the default

srun --overlap is the default choice. Choose SSH only when a tool you’re running requires an SSH endpoint and can’t work through srun.

Open a shell on a node in your allocation

srun --overlap starts a new job step inside an allocation you already hold. The --overlap flag lets that step share resources with the steps already running, instead of waiting for them to release the node.

Find the job you want to inspect and the nodes it holds.

squeue --me

You should see output similar to the following:

JOBID  PARTITION  NAME     USER          ST  TIME   NODES  NODELIST(REASON)
1335   h100       raytest  example-user  R   15:46  2      slurm-h100-231-[147,217]

Expand the compact node list into individual hostnames. Replace [JOB-ID] with the job ID from the previous step.
scontrol show hostnames $(squeue -j [JOB-ID] -h -o %N)
You should see output similar to the following:
slurm-h100-231-147 slurm-h100-231-217
Open an interactive shell on the job. Replace [JOB-ID] with your job ID.
srun --overlap --jobid [JOB-ID] --pty bash -i
Without a node selection, Slurm places the step on the first node of the allocation. For a multi-node job, that’s usually the rank-zero node.
Optional: To target a specific node instead, pass -w with a hostname from step 2.
srun --overlap --jobid [JOB-ID] -w [NODE-NAME] --pty bash -i

You now have a shell on the compute node, inside the job’s cgroup. Anything you run counts against the job’s resources, and it stops when the job ends. To run a single command instead of opening a shell, omit --pty and pass the command directly:

srun --overlap --jobid [JOB-ID] -w [NODE-NAME] nvidia-smi

Attach to a container running in your job

To get inside the running container, name it when the job creates it, then attach with the :exec suffix. That extra naming step matters because a shell from srun --overlap lands on the compute node, not inside the container your job started. The two have different filesystems, different environment variables, and different installed packages, so a debugging session on the node shows little about the environment the job’s process runs in.

This requires SUNK v7.x or later. On earlier versions the attach fails with "exec" flag was passed to --container-name but the container is not running, even when the container is running. See the SUNK v7.0.0 release notes.

Name the container when the job starts it

Pyxis only tracks a container by name if you give it one. Add --container-name to the srun or sbatch step that starts the container:

srun --container-image=/mnt/home/[USERNAME]/[IMAGE].sqsh \
     --container-name=[NAME]-${SLURM_JOB_ID} \
     --container-mounts=/mnt/home:/mnt/home \
     sleep infinity

Include ${SLURM_JOB_ID} in the name. Pyxis reuses an existing container filesystem when a matching name already exists on a node, so a fixed name such as interactive reuses whatever an earlier job left on that node. On a multi-node job, that yields containers with different contents on different nodes, all from the same image file. A job-scoped name also keeps two users off one cached filesystem. The cache directory belongs to the user who created it, so a second user who reuses the same name gets Permission denied reading the container’s root filesystem.

Attach to the named container

Attach from the login node with --container-name=[NAME]:exec. With the :exec suffix, Pyxis enters the running container and fails if it isn’t running, rather than quietly creating a new one.

srun --overlap --jobid [JOB-ID] -w [NODE-NAME] \
     --container-name=[NAME]-[JOB-ID]:exec --pty bash -i

Replace the placeholders with the following:

[JOB-ID]: the job ID from squeue --me. Because the job expanded ${SLURM_JOB_ID} into the container name, this is also the suffix on the name you attach to.
[NODE-NAME]: a hostname from scontrol show hostnames. Pass this every time. Pyxis creates one container per node, so the attach only succeeds on the node where that container is running.
[NAME]: the base container name you set when the job started the container.

To confirm you’re inside the container rather than on the compute node, check the filesystem, not the process list. Enroot doesn’t create a separate PID namespace, so ps shows the same processes inside and outside the container. Print the OS release instead. Inside the container, it comes from your image:

cat /etc/os-release

If your image and the compute node run the same distribution, look for a path or package that only your image has, such as your project directory or a Python package the job depends on. You’re now inside the job’s container, sharing its filesystem and running alongside the job’s own processes.

`--container-name:exec` compared to `--container-image`

Both flags give you a shell with your image’s filesystem, but they aren’t interchangeable:

--container-name=[NAME]:exec enters the container the job is already running. You see the job’s processes, its environment, and any files it has written since it started.
--container-image=[IMAGE].sqsh starts a fresh container from the image on the same node. You get the same software, but a clean filesystem and none of the job’s runtime state.

Use :exec when you’re debugging what a job is doing right now. Use --container-image when you want a scratch environment that happens to match the job’s, such as testing a fix before you resubmit. SSH is worth the extra setup when a tool on your workstation requires a real SSH endpoint on the compute node, such as rsync, scp, or a remote debugger. For interactive shells, srun --overlap is simpler, and it puts the session in the job’s cgroup without depending on how the cluster’s PAM stack is configured. Compute nodes have no public IP address, so your workstation can’t reach one directly. Route the connection through the login node with ProxyJump. This requires SSH access to the login node itself. If your cluster’s directory service isn’t configured for SSH and you use kubectl exec instead, ProxyJump isn’t available. See Connect to the Slurm Login node to check which access method applies to your cluster.

SSH access on compute nodes

CoreWeave clusters run sshd on compute nodes by default, so the SSH method is available without a configuration change. The Slurm chart CoreWeave publishes ships compute.ssh.enabled: true, and where the chart leaves the field unset, the SUNK operator defaults it to enabled. Either way, only an administrator who sets it to false turns compute SSH off. Users authenticate with the SSH keys that SUNK User Provisioning distributes, the same keys that grant login node access. To change the setting on a Helm chart deployment, set the following in your Slurm values.yaml:

compute:
  ssh:
    enabled: true

The SunkCluster resource on a self-service cluster has no field for this, so there’s no supported way to change it. Contact CoreWeave support if you need compute SSH turned off.

Because compute SSH is on by default, a user with a provisioned key can log in to any compute node, whether or not they hold a job on it. Work started that way escapes Slurm’s accounting and has drained nodes on production clusters. Administrators should configure pam_slurm_adopt, which denies logins to nodes where the user holds no allocation and places the session under the job’s extern step cgroup, subject to the job’s resource limits and cleaned up when the job ends. SUNK doesn’t configure it for you.

Connect with ProxyJump

ssh -J opens a connection to the login node and tunnels a second SSH connection through it to the compute node. Your key never leaves your workstation. The login node forwards encrypted bytes it can’t read. Replace [USERNAME] with your Slurm username, [LOGIN-NODE] with the same login node address you already use to reach the cluster, and [NODE-NAME] with a hostname from scontrol show hostnames. To find the login node address, see Connect to the Slurm Login node.

ssh -J [USERNAME]@[LOGIN-NODE] [USERNAME]@[NODE-NAME]

The login node resolves [NODE-NAME], so use the Slurm node name exactly as squeue reports it. If the name doesn’t resolve, use the node’s address from scontrol show node [NODE-NAME] instead.

Don’t use agent forwarding (ssh -A) to reach compute nodes. Agent forwarding exposes your SSH agent socket on the login node, and anyone with root there can use it to authenticate as you, to any host your key opens. ProxyJump solves the same problem without that exposure.

Make ProxyJump automatic

Rather than typing -J each time, put the jump in ~/.ssh/config on your workstation. Match your cluster’s node naming pattern so every compute node routes through the login node automatically:

~/.ssh/config

Host slurm-login
    HostName [LOGIN-NODE]
    User [USERNAME]
    IdentityFile ~/.ssh/id_ed25519

Host slurm-*
    User [USERNAME]
    ProxyJump slurm-login
    IdentityFile ~/.ssh/id_ed25519

With this in place, ssh slurm-h100-231-147 connects through the login node without further flags. Adjust the Host slurm-* pattern to match the node names in your cluster, and make sure it doesn’t also match the login node entry.

Troubleshooting

The following sections cover the failures you’re most likely to encounter when attaching to a container or connecting over SSH.

Attach fails when the container is not running

The attach fails with "exec" flag was passed to --container-name but the container is not running. The step landed on a node where that named container isn’t running. Pyxis creates one container per node, so Slurm can schedule an attach without -w onto a different node in the allocation.

Confirm which node runs the container, then target it with -w:

srun --overlap --jobid [JOB-ID] -w [NODE-NAME] bash -c 'enroot list -f'

Look for a row named pyxis_[NAME] with a non-empty PID. If the PID column is blank, the container isn’t running on that node, or the name maps to a leftover container from an earlier job.

If the name is stale, remove it on that node and resubmit the job with a job-scoped name:

srun --overlap --jobid [JOB-ID] -w [NODE-NAME] enroot remove -f pyxis_[NAME]

Container contents differ between nodes in the same job

The job uses a fixed --container-name, and Pyxis reused a leftover container filesystem on some nodes instead of importing the image again. Give the container a job-scoped name, as described in Name the container when the job starts it, or omit --container-name entirely for steps you don’t need to attach to. Without a name, Pyxis creates a job-scoped container and cleans it up when the job ends.

`Permission denied` when connecting over SSH to a compute node

Check these in order:

You hold an allocation on that specific node. With pam_slurm_adopt configured, a valid key alone isn’t enough. Confirm with squeue --me that the node appears in your job’s node list.
Your key is on the cluster. Compute nodes use the same provisioned keys as the login node. If you can’t reach the login node either, add a key under Slurm Attributes in your Cloud Console settings.
sshd is running on the node. Compute SSH is on by default, but an administrator can disable it with compute.ssh.enabled: false. A refused connection, rather than Permission denied, points here. Ask your administrator.

​Prerequisites

​Choose an access method

​Open a shell on a node in your allocation

​Attach to a container running in your job

​Name the container when the job starts it

​Attach to the named container

​--container-name:exec compared to --container-image

​Connect over SSH through the login node

​SSH access on compute nodes

​Connect with ProxyJump

​Make ProxyJump automatic

​Troubleshooting

​Attach fails when the container is not running

​Container contents differ between nodes in the same job

​Permission denied when connecting over SSH to a compute node

​Related

Prerequisites

Choose an access method

Open a shell on a node in your allocation

Attach to a container running in your job

Name the container when the job starts it

Attach to the named container

`--container-name:exec` compared to `--container-image`

Connect over SSH through the login node

SSH access on compute nodes

Connect with ProxyJump

Make ProxyJump automatic

Troubleshooting

Attach fails when the container is not running

Container contents differ between nodes in the same job

`Permission denied` when connecting over SSH to a compute node

Related