What do GPU XID error codes mean and what should I do?

An XID is an error report from the NVIDIA driver, written to the kernel log of the Node that hosts the GPU. Not every XID is a hardware fault. Common codes such as XID 13 and XID 31 are bugs in your workload, and several others are follow-on events rather than root causes. XIDs appear in the Node’s kernel log (dmesg), in your training logs as the resulting CUDA error, and as XID counts on the Node Details Grafana dashboard. The number after Xid is the code, and the PCI address identifies the GPU:

NVRM: Xid (PCI:0000:1a:00): 31, pid=204811, name=python3, channel 0x00000002, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC0 GPCCLIENT_T1_2 faulted @ 0x7b54_2f000000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

Is it your workload or the hardware?

Two patterns settle most cases:

One GPU that fails many different jobs is suspect hardware. The same PCI address reporting the same XID while the same code runs cleanly on every other GPU is the strongest hardware signal.
One job that fails on many GPUs is suspect code. The same XID on many GPUs at the same moment, in one job, is the workload or a crash cascade started by one rank. Find the first rank to fail.

Common XID codes

XID	Meaning	Classification	What to do
13	Graphics engine exception: illegal instruction or out-of-bounds access.	Application	Debug your kernel or framework code. If it fires alongside XID 48 on the same GPU, the memory error is the root cause. Open a ticket.
31	GPU memory page fault. The GPU equivalent of a segfault.	Application	Run the workload under `compute-sanitizer --tool memcheck`. Hardware only when one GPU fails repeatedly while others run the same code cleanly.
43	GPU stopped processing.	Application	Look for a kernel that never completes.
45	Preemptive cleanup after a previous error.	Follow-on	Find the XID that preceded it.
48	Double-bit ECC error. An uncorrectable error in GPU memory.	Hardware	CoreWeave reboots the Node automatically. XID 171 (DRAM) or 172 (SRAM) and XID 63 (row remapped) accompany it. This is one memory error and its recovery, not several faults. XID 64 means the remap failed. CoreWeave takes that Node out of service.
79	GPU has fallen off the bus.	Hardware	CoreWeave takes the Node out of service automatically. Open a ticket if it stays in service or your job doesn’t recover.
94	Contained memory error.	Informational	No action for a single occurrence. Sustained XID 94 on one GPU over days warrants a ticket.
95	Uncontained memory error. Not necessarily an ECC fault.	Uncertain	Usually software-triggered by an unclean job teardown where surviving ranks touch a peer GPU whose process exited. CoreWeave reboots the Node automatically. If XID 48 fired on the same GPU, that is the root cause, otherwise investigate the crash.
109	Context switch timeout, usually too many processes per GPU or unclean teardown.	Application	CoreWeave collects diagnostics and reboots the Node automatically. Check process count and cleanup in your job.
119	GSP RPC timeout. The GPU’s firmware processor stopped responding.	Uncertain	CoreWeave resets the Node automatically. If several Nodes report it on the same deployment, investigate the workload.
137	NVLink privilege error.	Follow-on	Usually surviving ranks accessing memory of a rank that already exited. Find the first rank that crashed.
145	NVLink Data Layer error.	Uncertain	Often transient. Open a ticket if it persists across several Nodes.
149	NVLink link events.	Hardware or firmware	CoreWeave cordons the Node and reboots it once the workload finishes. Open a ticket if your job keeps failing on the same Node, or if throughput drops on a Node without the job failing. A non-fatal link event can leave a GPU degraded rather than crash it.
154	GPU recovery action changed. The driver is requesting a GPU or Node reset.	Platform action	Triggers CoreWeave’s automatic remediation. The Node is cordoned and queued for reboot. XID 154 indicates the platform has already acted. Read the preceding XID for the cause.

When to open a ticket

Open a ticket when the same GPU fails repeatedly across distinct workloads, when a hardware XID fires and the Node isn’t drained, or when an error is concentrated on one rack or NVLink domain while the same job runs cleanly elsewhere. Include the Node name, the GPU PCI address, the XID lines with timestamps, and the job or Pod that was running. CoreWeave collects the NVIDIA bug report from the Node itself. For everything else, investigate your workload first. See Contact support to open a ticket.

How do I interpret GPU health events?: XID counts and drain events in Grafana.
NVIDIA XID catalog: the full list of codes.

Nodes Server Errors

​Is it your workload or the hardware?

​Common XID codes

​When to open a ticket

​Related pages

Is it your workload or the hardware?

Common XID codes

When to open a ticket

Related pages