Skip to main content
A node that drains with NHC: check_file_test: -e /dev/infiniband returned 1 has lost its InfiniBand device mount inside the slurmd Pod. The node undrains after the next passing verification test, then drains again on the next job. Restart that Pod. Don’t restart the RDMA device plugin, and don’t reboot the node: the node’s InfiniBand is healthy, and only the Pod lost the device. Most Node Health Check (NHC) drains clear on their own. For the drain workflow, the full list of drain reasons, and how the sunk:verify-undrain marker works, see Drain and undrain Slurm nodes. This article covers one NHC reason that recurs instead of clearing.

​
What you see

The drain reason is NHC: check_file_test: -e /dev/infiniband returned 1 (sunk:verify-undrain). To read it, see Identify the drain reason. The node then cycles rather than staying drained:
  1. The slurmd Pod starts without /dev/infiniband.
  2. A job’s Epilog runs the file check, finds no InfiniBand devices, and drains the node.
  3. HPC Verification requests rdma/ib for itself, receives a working device, and passes.
  4. Because the reason carries sunk:verify-undrain, SUNK undrains the node on that result.
  5. The next job runs, and its Epilog fails the same check.
Steps 2 through 5 then repeat. HPC Verification passes because it requests its own device rather than reusing the one the slurmd Pod is missing. The test never encounters the problem that drained the node.

​
Why the Pod loses the device

The usual cause is a boot race during a node reboot. The RDMA device plugin restarts after the slurmd Pod has started, which leaves the Pod’s device mount stale. The fabric is healthy and the node still advertises rdma/ib as allocatable, so the only thing wrong is the mount inside one Pod. That’s why this appears in batches after a maintenance window, on the nodes that rebooted.

​
Confirm the diagnosis

Look for the devices inside the slurmd Pod. Replace [POD-NAME] with the name of the Pod that runs the Slurm node. To find it, use the get pod command in Restart the Slurm daemon:
If the directory is missing or empty inside the Pod while the node still reports rdma/ib as allocatable, the stale mount is confirmed.

​
Restart the slurmd Pod

Deleting the Pod re-registers the devices and breaks the loop. For the commands, see Restart the Slurm daemon. The node rejoins the Slurm pool when the Pod is recreated, and the next passing verification test undrains it through the normal sunk:verify-undrain cycle. Don’t restart the RDMA device plugin. By the time this drain reason appears, the plugin has recovered, and restarting it doesn’t refresh the mount in a Pod that’s already running.

​
When to file a support ticket

Open a support ticket in any of these situations:
  • The node returns to the same loop after you restart the slurmd Pod.
  • The /dev/infiniband directory is present inside the Pod, but the check still fails.
  • The node reports no allocatable rdma/ib at all, which is a different problem from a stale mount.
  • A drain reason carries sunk:verify-undrain but doesn’t match this one, and the node has cycled more than twice.
Include the node names, the full drain reason, and the output of the ls /dev/infiniband command you ran inside the Pod. Nodes Server Errors
Last modified on October 13, 2026