> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.
# What do GPU XID error codes mean and what should I do?
An XID is an error report from the NVIDIA driver, written to the kernel log of the Node that hosts the GPU. Not every XID is a hardware fault. Common codes such as XID 13 and XID 31 are bugs in your workload, and several others are follow-on events rather than root causes.
XIDs appear in the Node's kernel log (`dmesg`), in your training logs as the resulting CUDA error, and as XID counts on the Node Details Grafana dashboard. The number after `Xid` is the code, and the PCI address identifies the GPU:
```text theme={"system"}
NVRM: Xid (PCI:0000:1a:00): 31, pid=204811, name=python3, channel 0x00000002, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC0 GPCCLIENT_T1_2 faulted @ 0x7b54_2f000000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
```
## Is it your workload or the hardware?
Two patterns settle most cases:
* **One GPU that fails many different jobs is suspect hardware.** The same PCI address reporting the same XID while the same code runs cleanly on every other GPU is the strongest hardware signal.
* **One job that fails on many GPUs is suspect code.** The same XID on many GPUs at the same moment, in one job, is the workload or a crash cascade started by one rank. Find the first rank to fail.
## Common XID codes
| XID | Meaning | Classification | What to do |
| --- | ------------------------------------------------------------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 13 | Graphics engine exception: illegal instruction or out-of-bounds access. | Application | Debug your kernel or framework code. If it fires alongside XID 48 on the same GPU, the memory error is the root cause. Open a ticket. |
| 31 | GPU memory page fault. The GPU equivalent of a segfault. | Application | Run the workload under `compute-sanitizer --tool memcheck`. Hardware only when one GPU fails repeatedly while others run the same code cleanly. |
| 43 | GPU stopped processing. | Application | Look for a kernel that never completes. |
| 45 | Preemptive cleanup after a previous error. | Follow-on | Find the XID that preceded it. |
| 48 | Double-bit ECC error. An uncorrectable error in GPU memory. | Hardware | CoreWeave reboots the Node automatically. XID 171 (DRAM) or 172 (SRAM) and XID 63 (row remapped) accompany it. This is one memory error and its recovery, not several faults. XID 64 means the remap failed. CoreWeave takes that Node out of service. |
| 79 | GPU has fallen off the bus. | Hardware | CoreWeave takes the Node out of service automatically. Open a ticket if it stays in service or your job doesn't recover. |
| 94 | Contained memory error. | Informational | No action for a single occurrence. Sustained XID 94 on one GPU over days warrants a ticket. |
| 95 | Uncontained memory error. Not necessarily an ECC fault. | Uncertain | Usually software-triggered by an unclean job teardown where surviving ranks touch a peer GPU whose process exited. CoreWeave reboots the Node automatically. If XID 48 fired on the same GPU, that is the root cause, otherwise investigate the crash. |
| 109 | Context switch timeout, usually too many processes per GPU or unclean teardown. | Application | CoreWeave collects diagnostics and reboots the Node automatically. Check process count and cleanup in your job. |
| 119 | GSP RPC timeout. The GPU's firmware processor stopped responding. | Uncertain | CoreWeave resets the Node automatically. If several Nodes report it on the same deployment, investigate the workload. |
| 137 | NVLink privilege error. | Follow-on | Usually surviving ranks accessing memory of a rank that already exited. Find the first rank that crashed. |
| 145 | NVLink Data Layer error. | Uncertain | Often transient. Open a ticket if it persists across several Nodes. |
| 149 | NVLink link events. | Hardware or firmware | CoreWeave cordons the Node and reboots it once the workload finishes. Open a ticket if your job keeps failing on the same Node, or if throughput drops on a Node without the job failing. A non-fatal link event can leave a GPU degraded rather than crash it. |
| 154 | GPU recovery action changed. The driver is requesting a GPU or Node reset. | Platform action | Triggers CoreWeave's automatic remediation. The Node is cordoned and queued for reboot. XID 154 indicates the platform has already acted. Read the preceding XID for the cause. |
## When to open a ticket
Open a ticket when the same GPU fails repeatedly across distinct workloads, when a hardware XID fires and the Node isn't drained, or when an error is concentrated on one rack or NVLink domain while the same job runs cleanly elsewhere. Include the Node name, the GPU PCI address, the XID lines with timestamps, and the job or Pod that was running. CoreWeave collects the NVIDIA bug report from the Node itself.
For everything else, investigate your workload first. See [Contact support](/support/contact) to open a ticket.
## Related pages
* [How do I interpret GPU health events?](/support/cks/articles/how-do-i-interpret-gpu-health-events): XID counts and drain events in Grafana.
* [NVIDIA XID catalog](https://docs.nvidia.com/deploy/xid-errors/latest/analyzing-xid-catalog.html): the full list of codes.
[Nodes](/support/cks/tags/nodes)[Server Errors](/support/cks/tags/server-errors)