Error message lookup - CoreWeave Docs

This page maps literal error strings to the page that explains each one. Paste or search for the exact text you see in a log line, a kubectl event, an S3 client response, or a Slurm reason. Then follow the link to the page that explains the cause and the fix. If you can’t find your error string here, browse the Support articles by product, or contact support. Before you open a ticket, gather the evidence listed in the support ticket templates.

How to use this page

Error strings are grouped by the surface where they appear: Kubernetes scheduling and Pods, capacity and quota, GPU and training, networking, and storage. Within each group, entries are listed by the substring you are most likely to search for. Match on the distinctive part of the message, not the full line, because most messages include Pod names, timestamps, and other variable text.

Sandbox access errors

Serverless sandboxes are publicly available. If an access check unexpectedly rejects a request, use the error below when contacting Support.

Error string	What it means	Where to go
`CWSANDBOX_SERVERLESS_NOT_ALLOWED`	The service rejected serverless access for the organization identified by the credential.	Contact CoreWeave Support with the request ID.

Scheduling and Pod errors

These strings appear in kubectl describe pod, kubectl get events, or Pod status on CKS.

Error string	What it usually means	Where to go
`FailedScheduling`	The scheduler can’t place the Pod on any Node. The accompanying text names the reason, for example insufficient GPUs, taints, or affinity.	Workload scheduling on CKS.
`0/N nodes are available`	No Node satisfies the Pod’s requests, taints, or affinity. The suffix lists the failing predicate per Node.	Workload scheduling on CKS.
`Insufficient nvidia.com/gpu`	No Node has enough free GPUs for the Pod’s request.	Node Pool status.
`Insufficient rdma/ib`	No Node has a free RDMA interface for the request.	InfiniBand and RoCE labels and Use GPUDirect RDMA.
`node(s) had untolerated taint(s)`	The Pod lacks a toleration for a taint CoreWeave or you applied to the Node.	Workload scheduling on CKS.
`ImagePullBackOff`, `ErrImagePull`	The kubelet can’t pull the container image: wrong name or tag, missing pull secret, or registry rate limiting.	Workload scheduling on CKS for namespace and secret context.
`CreateContainerConfigError`	The container references a missing ConfigMap, Secret, or key.	Check the referenced ConfigMap and Secret names against your manifest.
`OOMKilled`	A memory kill, but not always one you caused. Read the kernel out-of-memory constraint: `CONSTRAINT_MEMCG` is your container limit, `CONSTRAINT_NONE` is Node-level pressure.	How do I diagnose out-of-memory errors? and Recommended resource requests and limits.
`Multi-Attach error for volume`	A `ReadWriteOnce` volume is still attached to another Node, often after a Node failure or a stuck Pod.	Troubleshoot a Pending PVC.
`FailedMount`, `Unable to attach or mount volumes`	The volume can’t mount: missing PVC, wrong access mode, or a storage-side problem.	Troubleshoot a Pending PVC.

Capacity and quota errors

These strings appear on Node Pool status, in scheduling events, and in the Cloud Console.

Error string	What it usually means	Where to go
`CWInsufficientCapacity`	CoreWeave doesn’t currently have free capacity to satisfy the Node Pool’s `targetNodes`.	Node Pool status.
`CWOverQuota`	The request exceeds your organization’s quota for the instance type.	Node Pool status and Quota self-service.
`QueuedAwaitingCapacity`	The Node Pool request is valid and queued until capacity frees up.	Node Pool status.
`Requested node configuration is not available` (Slurm)	The Slurm job requests a partition, feature, or constraint that no Node provides.	Monitor Slurm node states.

GPU and training errors

These strings appear in training logs, GPU health events, and dmesg.

Error string	What it usually means	Where to go
`NCCL error`, `NCCL WARN`	A collective communication operation failed. The numeric code and message identify the cause, and the message often names a different layer than the one that failed.	What do NCCL errors mean?.
`NCCL error 3`, `internal error`, `ncclInternalError`	Almost never an NCCL bug, despite the wording. Usually a job layout that breaks NCCL’s rank-to-device mapping, a broken transport path, a GPU fault, or application memory corruption.	NCCL Error 3.
`NCCL error 2`, `unhandled system error`, `ncclSystemError`	A system call underneath NCCL failed: the network, the OS, or the driver.	NCCL Error 2.
`unhandled cuda error`, `ncclUnhandledCudaError`	NCCL Error 1: a CUDA call failed underneath NCCL. A GPU-side error, so start with GPU XIDs and the driver.	NCCL Error 1.
`remote process exited or there was a network error`, `ncclRemoteError`	NCCL Error 6: a peer rank crashed, or the network path to it failed. Find the rank that died first.	NCCL Error 6.
`WorkNCCL ... timed out`, `Watchdog caught collective operation timeout`	A `torch.distributed` collective didn’t complete within the timeout. Most often a starved dataloader or slow storage, not the fabric.	WorkNCCL timeout diagnostic order.
`vendor err 249`, `vendor err 129`	An InfiniBand transport error. 249 is an actively flapping link; 129 is transport retries exhausted, often from an `NCCL_IB_RETRY_CNT` above 7.	vendor err 249 compared to vendor err 129.
`ibv_reg_mr_iova2 failed`, `Invalid argument` (RDMA registration)	The container’s libibverbs or OFED userspace doesn’t match the node’s kernel RDMA modules. Not an IB fabric fault.	ibv_reg_mr_iova2 failed with Invalid argument.
`destroy_process_group ... was not called`	A previous run exited without tearing down, which can leave a corrupted torch.compile or Triton cache that hangs the next run during init.	Stuck in NCCL communicator init.
Exit code `137`, `139`, `143`, `132`, `135` (or `-9`, `-11`, `-15`, `-4`, `-7`)	A signal killed the training process, where the exit code is `128 + signal number`. The same code has several possible causes.	What do training job exit codes mean?.
`Illegal instruction`, `DataLoader worker pid ... killed by signal: Illegal instruction`	SIGILL. On Grace (aarch64) instances, x86_64 native code reaching an Arm CPU, often through a shared JIT cache or an x86 numpy or BLAS build.	SIGILL on aarch64 and Run jobs on Grace CPU instances.
`c10d sendBytes failed ... Broken pipe`, `waitForInput ... timed out`, `DistStoreError`, `TCP client failed to connect/validate to host`	A `torch.distributed` rendezvous failure. These strings look alike and have different causes, from a PyTorch version bug to a hostname that didn’t resolve.	Fix torch.distributed rendezvous and TCPStore failures.
`XID` (for example `XID 31`, `XID 79`, `XID 119`)	An NVIDIA driver-reported GPU error. The XID number classifies the fault, from an application bug to a hardware failure.	What do GPU XID error codes mean?.
`CUDA error: out of memory`	The process requested more GPU memory than is free. Reduce batch size or model footprint.	Recommended resource requests and limits.
`uncorrectable ECC error`, `Xid ... row remap`	A GPU memory hardware fault. CoreWeave detects these and replaces affected hardware.	Interpret GPU health events and Node cordoning.

Networking errors

These strings appear in application logs, kubectl events, and DNS lookups.

Error string	What it usually means	Where to go
`dial tcp ... connection refused`	The target Service or Pod isn’t listening, or the Service has no ready endpoints.	Networking support.
`no route to host`, `network is unreachable`	The destination isn’t reachable from the Pod’s network, often a policy, CIDR, or routing problem.	Networking support and VPC CIDR blocks.
`Temporary failure in name resolution`, `SERVFAIL`, `NXDOMAIN`	DNS resolution failed inside the Pod.	Networking support.
`i/o timeout` on a Service or external host	The connection started but received no response in time, often a policy or upstream problem.	Networking support.

Storage errors

These strings appear in NFS and PVC mounts, and in S3 client responses.

Error string	What it usually means	Where to go
`ESTALE`, `Stale file handle`	The NFS file handle is no longer valid, often after a remount or a deleted-and-recreated file.	About Distributed File Storage.
`ENOSPC`, `No space left on device`	The filesystem, quota, or local disk is full. On `/tmp` this is often a tmpfs limit, not a disk limit.	Manage quotas.
`Permission denied` on a shared path	The path is owned by a different user or group, or the access mode doesn’t allow the operation.	About Distributed File Storage.
`BucketNotEmpty`	A delete-bucket call failed because the bucket still contains objects or incomplete multipart uploads.	Empty and delete a bucket and Abort multipart uploads.
`405 Method Not Allowed`, `write suspended for this availability zone`	A write was attempted against a bucket or path where writes aren’t allowed, often an over-quota write suspension.	Manage Object Storage quotas.
`PathStyleRequestNotAllowed`, `InvalidRequest` (path-style)	The S3 client used path-style addressing. CoreWeave AI Object Storage expects virtual-hosted addressing.	Configure endpoints.
`AccessDenied`, `403 Forbidden` (S3)	The request isn’t authorized: wrong keys, missing bucket policy permission, or an organization policy block.	Bucket policies and Manage access keys.
`SignatureDoesNotMatch`	The request signature is wrong, usually a wrong secret key, clock skew, or an addressing-style mismatch.	Configure endpoints.

When the error is not listed

If your exact string isn’t here, the message is still your best search key. Try these steps in order:

Browse the Support articles for your product by symptom, for example “pod pending”, “mount failed”, or “GPU error”.
Check what CoreWeave manages to decide whether the problem is on the platform side or the workload side.
Review the glossary if the message includes a term you don’t recognize, such as a Node label or a condition.
If you still need help, contact support with the evidence from the support ticket templates.

​How to use this page

​Sandbox access errors

​Scheduling and Pod errors

​Capacity and quota errors

​GPU and training errors

​Networking errors

​Storage errors

​When the error is not listed