> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Why does my node keep draining for a missing InfiniBand device?

A node that drains with `NHC: check_file_test: -e /dev/infiniband returned 1` has lost its InfiniBand device mount inside the `slurmd` Pod. The node undrains after the next passing verification test, then drains again on the next job. Restart that Pod. Don't restart the RDMA device plugin, and don't reboot the node: the node's InfiniBand is healthy, and only the Pod lost the device.

Most Node Health Check (NHC) drains clear on their own. For the drain workflow, the full list of drain reasons, and how the `sunk:verify-undrain` marker works, see [Drain and undrain Slurm nodes](https://docs.coreweave.com/products/sunk/manage_sunk/drain-and-undrain-nodes.md). This article covers one NHC reason that recurs instead of clearing.

## What you see

The drain reason is `NHC: check_file_test: -e /dev/infiniband returned 1 (sunk:verify-undrain)`. To read it, see [Identify the drain reason](https://docs.coreweave.com/products/sunk/manage_sunk/drain-and-undrain-nodes.md#identify-the-drain-reason).

The node then cycles rather than staying drained:

1. The `slurmd` Pod starts without `/dev/infiniband`.
2. A job's Epilog runs the file check, finds no InfiniBand devices, and drains the node.
3. [HPC Verification](https://docs.coreweave.com/platform/fleet-management/hpc-verification.md) requests `rdma/ib` for itself, receives a working device, and passes.
4. Because the reason carries `sunk:verify-undrain`, SUNK undrains the node on that result.
5. The next job runs, and its Epilog fails the same check.

Steps 2 through 5 then repeat. HPC Verification passes because it requests its own device rather than reusing the one the `slurmd` Pod is missing. The test never encounters the problem that drained the node.

## Why the Pod loses the device

The usual cause is a boot race during a node reboot. The RDMA device plugin restarts after the `slurmd` Pod has started, which leaves the Pod's device mount stale. The fabric is healthy and the node still advertises `rdma/ib` as allocatable, so the only thing wrong is the mount inside one Pod. That's why this appears in batches after a maintenance window, on the nodes that rebooted.

## Confirm the diagnosis

Look for the devices inside the `slurmd` Pod. Replace `[POD-NAME]` with the name of the Pod that runs the Slurm node. To find it, use the `get pod` command in [Restart the Slurm daemon](https://docs.coreweave.com/products/sunk/manage_sunk/interact-with-kubernetes.md#restart-the-slurm-daemon):

```bash theme={"system"}
# List the InfiniBand devices visible inside the slurmd Pod. Read-only.
kubectl exec -n tenant-slurm [POD-NAME] -- ls /dev/infiniband
```

If the directory is missing or empty inside the Pod while the node still reports `rdma/ib` as allocatable, the stale mount is confirmed.

## Restart the slurmd Pod

Deleting the Pod re-registers the devices and breaks the loop. For the commands, see [Restart the Slurm daemon](https://docs.coreweave.com/products/sunk/manage_sunk/interact-with-kubernetes.md#restart-the-slurm-daemon). The node rejoins the Slurm pool when the Pod is recreated, and the next passing verification test undrains it through the normal `sunk:verify-undrain` cycle.

Don't restart the RDMA device plugin. By the time this drain reason appears, the plugin has recovered, and restarting it doesn't refresh the mount in a Pod that's already running.

## When to file a support ticket

[Open a support ticket](https://docs.coreweave.com/support/contact.md) in any of these situations:

* The node returns to the same loop after you restart the `slurmd` Pod.
* The `/dev/infiniband` directory is present inside the Pod, but the check still fails.
* The node reports no allocatable `rdma/ib` at all, which is a different problem from a stale mount.
* A drain reason carries `sunk:verify-undrain` but doesn't match this one, and the node has cycled more than twice.

Include the node names, the full drain reason, and the output of the `ls /dev/infiniband` command you ran inside the Pod.

## Related pages

* [Drain and undrain Slurm nodes](https://docs.coreweave.com/products/sunk/manage_sunk/drain-and-undrain-nodes.md): the drain workflow, every drain reason, and the `sunk:verify-undrain` automation.
* [Why is my node drained or cordoned?](https://docs.coreweave.com/support/sunk/articles/why-is-my-node-drained-or-cordoned): where to start when you don't know the reason yet.
* [Interact with SUNK using kubectl](https://docs.coreweave.com/products/sunk/manage_sunk/interact-with-kubernetes.md): how to reach the Pods behind a Slurm node.
* [HPC Verification](https://docs.coreweave.com/platform/fleet-management/hpc-verification.md): the hourly test that undrains a node once it passes.
* [SUNK troubleshooting](https://docs.coreweave.com/support/sunk.md): all SUNK triage flows.

[Nodes](https://docs.coreweave.com/support/sunk/tags/nodes.md)[Server Errors](https://docs.coreweave.com/support/sunk/tags/server-errors.md)