Getting started with Dedicated Inference

This guide walks you through deploying your first model on CoreWeave Dedicated Inference, the Bring Your Own Weights (BYOW) option for serving models on dedicated GPU infrastructure. It targets developers and ML engineers who want to serve their own model weights on managed GPU instances without operating the underlying serving stack. By the end, you have a running inference endpoint that responds to prompts through the OpenAI-compatible API. You can then integrate the endpoint into applications that already target OpenAI-style chat completions. For an overview of CoreWeave’s inference offerings, see the Inference introduction.

Prerequisites

Before you begin, verify that you have the following:

A CoreWeave account with Inference access enabled.
A CoreWeave API access token with the Inference Admin role.
Model weights uploaded to a CoreWeave Object Storage bucket. Dedicated Inference uses a bring-your-own-weights (BYOW) model. Download model weights from your model provider and upload them to Object Storage before starting.
The CoreWeave Intelligent CLI (cwic), which this guide uses for its primary examples. The guide also shows the equivalent curl requests and a CoreWeave Terraform provider configuration, so you can use curl or Terraform instead.
curl, which the final inference request uses even if you create resources with the CLI or Terraform.
Terraform or OpenTofu, only if you use the Terraform path. Running terraform init installs the CoreWeave Terraform provider from the registry.

Want to interact with the API programmatically? See the Inference API reference for the REST, gRPC, and Connect interfaces, or install a generated client from the Inference SDKs.

Set your API token

Set your API token as an environment variable so that subsequent commands can authenticate with the CoreWeave API. Replace [API-TOKEN] with your token. For details about creating a token, see Manage API access tokens.

export CW_API_TOKEN="[API-TOKEN]"
export CW_BASE_URL="https://api.coreweave.com"

CoreWeave Intelligent CLI
curl

Authenticate cwic once. It stores the token in ~/.cwic/config.json and reuses it for every subsequent cwic inference command.

cwic auth login "${CW_API_TOKEN}"

To open console.coreweave.com/tokens in your browser and paste a different token interactively, run the command without an argument:

cwic auth login

Verify you’re authenticated:

cwic auth whoami

The exported CW_API_TOKEN and CW_BASE_URL variables authenticate the curl examples in this guide.

Grant inference access to your bucket

Dedicated Inference reads model weights from CoreWeave Object Storage using a dedicated service account. Attach the following bucket policy to your weights bucket so the service can list and read its contents. Replace [BUCKET-NAME] with the name of the bucket containing your model weights.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowInference",
      "Effect": "Allow",
      "Principal": {
        "CW": [
          "arn:aws:iam::cw4637:coreweave/uvAGGQSxxXeeQBJzcGsD9"
        ]
      },
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::[BUCKET-NAME]",
        "arn:aws:s3:::[BUCKET-NAME]/*"
      ]
    }
  ]
}

Save the policy to a file (for example, inference-bucket-policy.json) and apply it with the AWS CLI. If you have configured a named AWS CLI profile for CoreWeave (for example, cw), pass it with --profile [PROFILE-NAME]. Omit --profile if your CoreWeave credentials are in the default profile.

This bucket policy for inference has the potential to override your existing permissions and bucket policy. This put-bucket-policy command is a replace, not a merge.

aws s3api put-bucket-policy \
  --bucket [BUCKET-NAME] \
  --policy file://inference-bucket-policy.json \
  --endpoint-url https://cwobject.com \
  --profile [PROFILE-NAME]

The Principal value is the canonical CoreWeave Inference service account and is the same for all customers. A missing bucket policy is the most common cause of deployments that fail to load weights. For tooling alternatives (s3cmd, Boto3, Terraform), how to scope access to a specific path prefix, and notes on the Cloud Console, see Grant inference access to your bucket.

Create a gateway

With the bucket policy in place, provision the gateway. A gateway provides the external endpoint that routes traffic to your model deployments. It handles authentication and load balancing.

Prefer to manage both resources as code? Skip ahead to Deploy with Terraform. You still run the parameters queries in this section and Create a deployment to choose valid zone, instance type, and engine version values.

First, query the available zones so you can place the gateway in a region where Dedicated Inference capacity is offered:

CoreWeave Intelligent CLI
curl

cwic inference gateway parameters

curl "${CW_BASE_URL}/v1alpha1/inference/gateways/parameters" \
  -H "Authorization: Bearer ${CW_API_TOKEN}"

The response’s zones array lists the zones currently available for gateway creation.

If this first request fails, check the message field. Both of the following errors return gRPC code 7 (HTTP 403):

"organization is not allowed to perform this operation": your organization isn’t enabled for Dedicated Inference yet. Contact your CoreWeave representative or CoreWeave support with your organization ID to request access.
"insufficient permissions for this operation": your API access token is missing the Inference Admin role. Add the role to the token’s identity, then retry.

Then create a gateway in one of the available zones listed in the previous step. Replace [ZONE-NAME] with a zone from the response. The gateway name must be a valid hostname label containing no more than 38 letters, digits, or hyphens. It must begin and end with a letter or digit, and it cannot contain dots. The 38-character limit accommodates DNS and TLS certificate constraints. If the hostname-label validation fails, you receive validation error: name: must be a valid hostname label. This example creates a gateway with CoreWeave IAM authentication and body-based routing, which routes requests based on the model field in the request body. Body-based routing is the default and follows OpenAI API conventions.

CoreWeave Intelligent CLI
curl

cwic inference gateway create \
  --name my-first-gateway \
  --zones [ZONE-NAME] \
  --auth coreweave \
  --routing body \
  --api-type openai

The output is a table with the new gateway’s ID, NAME, ZONES, AUTH, ROUTING, STATUS, and ENDPOINTS columns. Save the value from the ID column for the next step. You need this ID to associate deployments with the gateway. Add -o json to get the same fields as machine-readable JSON.

curl -X POST "${CW_BASE_URL}/v1alpha1/inference/gateways" \
  -H "Authorization: Bearer ${CW_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-first-gateway",
    "zones": ["[ZONE-NAME]"],
    "coreWeaveAuth": {},
    "bodyBasedRouting": {
      "apiType": "API_TYPE_OPENAI"
    }
  }'

The response includes the gateway specification and creation timestamps. The status.status and endpoints fields appear on subsequent GET requests once the gateway is provisioned.

{
  "gateway": {
    "spec": {
      "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "name": "my-first-gateway",
      "organizationId": "[ORG-UID]",
      "zones": ["US-EAST-04A"],
      "coreWeaveAuth": {},
      "bodyBasedRouting": {
        "apiType": "API_TYPE_OPENAI"
      }
    },
    "status": {
      "createdAt": "2026-04-14T12:00:00Z",
      "updatedAt": "2026-04-14T12:00:00Z"
    }
  }
}

Save the gateway ID from gateway.spec.id for the next step. You need this ID to associate deployments with the gateway.

If the create request returns {"code":8, "message":"gateway quota exceeded: maximum is 0"} (HTTP 429), your organization has no inference gateway quota. Contact your CoreWeave representative or CoreWeave support to request inference quota for your organization.

Either way, export the gateway ID:

export CW_GATEWAY_ID="[GATEWAY-ID]"

You now have a gateway that can accept inference traffic but doesn’t yet route to any model. The next section attaches a deployment to it.

The gateway may take a few moments to become ready. You can check its status with a GET request to /v1alpha1/inference/gateways/{id}, or with cwic inference gateway get [GATEWAY-ID].

View all gateways

You can view all existing gateways with the following command. Use it to confirm your gateway was created correctly.

CoreWeave Intelligent CLI
curl

cwic inference gateway list

Parse the output with jq to make the response more readable.

curl "${CW_BASE_URL}/v1alpha1/inference/gateways" -H "Authorization: Bearer ${CW_API_TOKEN}"

curl "${CW_BASE_URL}/v1alpha1/inference/gateways" -H "Authorization: Bearer ${CW_API_TOKEN}" | jq .

Create a deployment

A deployment configures a model serving instance with your chosen runtime, GPU type, and model weights. Attaching the deployment to the gateway you just created makes the model reachable through the gateway endpoint. First, query the available instance types and runtime versions so you can pick a GPU type and runtime that match your model:

CoreWeave Intelligent CLI
curl

cwic inference deployment parameters

curl "${CW_BASE_URL}/v1alpha1/inference/deployments/parameters" \
  -H "Authorization: Bearer ${CW_API_TOKEN}"

The response lists the gateway IDs that can be referenced by a new deployment under gatewayIds. It also groups the available options into two objects: resourceParameters.instanceTypes lists the GPU instance types you can deploy on, and runtimeParameters.runtimeVersions lists the available versions for each engine (such as vllm). Use these to choose a valid [GATEWAY-ID], [INSTANCE-TYPE], and [ENGINE-VERSION] below. If resourceParameters or instanceTypes is empty, no instance type is currently available for the request. Confirm that your organization has an eligible gateway and inference quota. When you filter by gateway IDs, also confirm that the selected gateways share an available instance type. Then create a deployment that references your gateway. Replace the placeholder values:

[GATEWAY-ID]: The gateway ID from the previous step (creating the gateway, or the inference/gateways list command).
[INSTANCE-TYPE]: An instance type from the inference/deployments/parameters response in the previous step. For details on each type, see GPU instances.
[MODEL-NAME]: A name for your model (from 4 to 63 characters). The gateway uses this name to route inference requests to this deployment.
[BUCKET-NAME]: The CoreWeave Object Storage bucket containing the model weights.
[MODEL-PATH]: The path within the bucket to the model directory.
[ENGINE-VERSION]: The vllm runtime version to serve your model with. The inference/deployments/parameters response (above) returns the available versions.

The deployment and model names serve different purposes and follow different validation rules:

Deployment name: Identifies the deployment in the management API. It must be a valid hostname label containing no more than 63 letters, digits, or hyphens. It must begin and end with a letter or digit, and it cannot contain dots.
Model name: Identifies the model in inference requests. It must be 4 to 63 characters long and can include letters, digits, periods, underscores, colons, slashes, and hyphens.

If the deployment name fails validation, you receive validation error: name: must be a valid hostname label.

An S3 path breaks down into [BUCKET-NAME] and [MODEL-PATH]. For example, s3://test-bucket/raw/Qwen/Qwen3.5-0.8B/2fc06364715b967f1860aea9cf38778875588b17 breaks down into:

[BUCKET-NAME]  test-bucket
[MODEL-PATH]   raw/Qwen/Qwen3.5-0.8B/2fc06364715b967f1860aea9cf38778875588b17

If you receive {"code":3, "message":"model path has no objects: path ..."}, the model path is invalid or inaccessible. If you receive {"code":3, "message":"model bucket is not accessible..."}, the bucket is invalid or inaccessible.

CoreWeave Intelligent CLI
curl

cwic inference deployment create \
  --name my-first-deployment \
  --gateway-ids ${CW_GATEWAY_ID} \
  --engine vllm \
  --engine-version [ENGINE-VERSION] \
  --instance-type [INSTANCE-TYPE] \
  --gpu-count 1 \
  --model-name [MODEL-NAME] \
  --model-bucket [BUCKET-NAME] \
  --model-path [MODEL-PATH] \
  --min 1 \
  --max 1 \
  --traffic-weight 100

The --traffic-weight flag has no effect for a single deployment: it controls traffic splitting only when two or more deployments share the same model name on a gateway. Leave the value at 100 for this guide. For details, see Traffic weights.The output is a table that includes the new deployment’s ID and STATUS columns. Save the value from the ID column so subsequent commands can reference the deployment. Add -o json to get the same fields as machine-readable JSON.

curl -X POST "${CW_BASE_URL}/v1alpha1/inference/deployments" \
  -H "Authorization: Bearer ${CW_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-first-deployment",
    "gatewayIds": ["[GATEWAY-ID]"],
    "runtime": {
      "engine": "vllm",
      "version": "[ENGINE-VERSION]"
    },
    "resources": {
      "instanceType": "[INSTANCE-TYPE]",
      "gpuCount": 1
    },
    "model": {
      "name": "[MODEL-NAME]",
      "bucket": "[BUCKET-NAME]",
      "path": "[MODEL-PATH]"
    },
    "autoscaling": {
      "min": 1,
      "max": 1
    },
    "traffic": {
      "weight": 100
    }
  }'

The traffic.weight field has no effect for a single deployment: it controls traffic splitting only when two or more deployments share the same model name on a gateway. Leave the value at 100 for this guide. For details, see Traffic weights.The response includes the deployment ID and creation timestamps. The status.status field appears when you poll the deployment with a GET request.

{
  "deployment": {
    "spec": {
      "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
      "name": "my-first-deployment",
      "organizationId": "[ORG-UID]",
      "gatewayIds": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"],
      "runtime": { "engine": "vllm", "version": "[ENGINE-VERSION]" },
      "resources": { "instanceType": "gd-8xh100ib-i128", "gpuCount": 1 },
      "model": { "name": "my-model", "bucket": "my-bucket", "path": "models/my-model/" },
      "autoscaling": { "min": 1, "max": 1 },
      "traffic": { "weight": 100 }
    },
    "status": {
      "createdAt": "2026-04-14T12:10:00Z",
      "updatedAt": "2026-04-14T12:10:00Z"
    }
  }
}

Save the deployment ID from deployment.spec.id so subsequent commands can reference the deployment.

Either way, export the deployment ID:

export CW_DEPLOYMENT_ID="[DEPLOYMENT-ID]"

The deployment is registered, but the inference engine isn’t yet serving traffic. The next section covers monitoring it until it becomes ready.

View all deployments

You can view all existing deployments with the following command. Use it to confirm your deployment was created correctly.

CoreWeave Intelligent CLI
curl

cwic inference deployment list

Parse the output with jq to make the response more readable.

curl "${CW_BASE_URL}/v1alpha1/inference/deployments" -H "Authorization: Bearer ${CW_API_TOKEN}"

curl "${CW_BASE_URL}/v1alpha1/inference/deployments" -H "Authorization: Bearer ${CW_API_TOKEN}" | jq .

Deploy with Terraform

The previous two sections created a gateway and a deployment by calling the API directly. If you manage infrastructure as code, you can define both resources in a single Terraform configuration and apply them together instead.

If you already created your gateway and deployment with the CLI or curl, skip this section and continue at Wait for the deployment to start. If you are starting here, first run the gateway and deployment parameters queries from Create a gateway and Create a deployment to choose valid zone, instance type, and engine version values.

When prompted, enter values corresponding to [ZONE-NAME], [ENGINE-VERSION], [INSTANCE-TYPE], [MODEL-NAME], [BUCKET-NAME], and [MODEL-PATH], using the same guidance as the API examples above. The configuration declares each value as a required variable. When you run terraform apply, Terraform prompts you to enter the corresponding value without the brackets.

main.tf

terraform {
  required_providers {
    coreweave = {
      source = "coreweave/coreweave"
    }
  }
}

variable "zone_name" {
  type        = string
  description = "Zone returned by the inference gateway parameters endpoint."
}

variable "engine_version" {
  type        = string
  description = "vLLM version returned by the inference deployment parameters endpoint."
}

variable "instance_type" {
  type        = string
  description = "GPU instance type returned by the inference deployment parameters endpoint."
}

variable "model_name" {
  type        = string
  description = "Model name used to route inference requests."
}

variable "bucket_name" {
  type        = string
  description = "Object Storage bucket containing the model weights."
}

variable "model_path" {
  type        = string
  description = "Path to the model directory within the bucket."
}

resource "coreweave_inference_gateway" "example" {
  name  = "my-first-gateway"
  zones = [var.zone_name]

  auth = {
    coreweave = {}
  }

  routing = {
    body_based = {
      api_type = "API_TYPE_OPENAI"
    }
  }
}

resource "coreweave_inference_deployment" "example" {
  name        = "my-first-deployment"
  gateway_ids = [coreweave_inference_gateway.example.id]

  runtime = {
    engine  = "vllm"
    version = var.engine_version
  }

  resources = {
    instance_type = var.instance_type
    gpu_count     = 1
  }

  model = {
    name   = var.model_name
    bucket = var.bucket_name
    path   = var.model_path
  }

  autoscaling = {
    min = 1
    max = 1
  }

  traffic = {
    weight = 100
  }
}

output "gateway_id" {
  value = coreweave_inference_gateway.example.id
}

output "gateway_endpoint" {
  value = one(coreweave_inference_gateway.example.endpoints)
}

output "deployment_id" {
  value = coreweave_inference_deployment.example.id
}

Because the deployment’s gateway_ids field references the gateway resource’s id attribute, Terraform creates the gateway first and the deployment second, even though both are defined in the same configuration.

export COREWEAVE_API_TOKEN="${CW_API_TOKEN}"
terraform init
terraform apply

Export the resulting IDs so you can continue with the rest of this guide:

export CW_GATEWAY_ID=$(terraform output -raw gateway_id)
export CW_DEPLOYMENT_ID=$(terraform output -raw deployment_id)

For the full resource schema, including optional fields like autoscaling capacity classes and gateway routing modes, see the coreweave_inference_gateway and coreweave_inference_deployment resource references.

Wait for the deployment to start

After creation, the deployment loads model weights and starts the inference engine. Poll the deployment status until it reaches STATUS_READY. This typically takes several minutes.

CoreWeave Intelligent CLI
curl

cwic inference deployment get ${CW_DEPLOYMENT_ID}

Repeat the command and check the STATUS column until it shows READY.

curl "${CW_BASE_URL}/v1alpha1/inference/deployments/${CW_DEPLOYMENT_ID}" \
  -H "Authorization: Bearer ${CW_API_TOKEN}"

Check deployment.status.status in the response. Continue polling until you see STATUS_READY:

{
  "deployment": {
    "status": {
      "status": "STATUS_READY"
    }
  }
}

To poll automatically, use this loop. It exits when the deployment reaches STATUS_READY, fails with STATUS_ERROR or STATUS_FAILED, or reaches the 15-minute timeout:

deadline=$(( $(date +%s) + 900 ))
while [ "$(date +%s)" -lt "$deadline" ]; do
  status_deployment=$(curl -sS "${CW_BASE_URL}/v1alpha1/inference/deployments/${CW_DEPLOYMENT_ID}" \
    -H "Authorization: Bearer ${CW_API_TOKEN}" \
    | jq -r '.deployment.status.status // "UNKNOWN"')
  echo "Status: $status_deployment"
  case "$status_deployment" in
    STATUS_READY) echo "Deployment is ready."; break ;;
    STATUS_ERROR|STATUS_FAILED) echo "Deployment failed."; exit 1 ;;
  esac
  sleep 30
done

if [ "$status_deployment" != "STATUS_READY" ]; then
  echo "Timed out waiting for the deployment to become ready."
  exit 1
fi

This loop uses jq to parse the response. Install jq with your package manager (for example, brew install jq on macOS or apt install jq on Debian or Ubuntu).

Once the deployment is running, retrieve the gateway endpoint URL:

CoreWeave Intelligent CLI
curl

cwic inference gateway get ${CW_GATEWAY_ID}

The endpoint URL appears in the ENDPOINTS column.

curl "${CW_BASE_URL}/v1alpha1/inference/gateways/${CW_GATEWAY_ID}" \
  -H "Authorization: Bearer ${CW_API_TOKEN}"

The gateway.status.endpoints field contains an array of endpoint URLs for inference requests. The first entry is the primary endpoint:

{
  "gateway": {
    "status": {
      "status": "STATUS_READY",
      "endpoints": ["https://my-first-gateway.[ORG-UID].gw.cwinference.com"]
    }
  }
}

Export the endpoint for the next step:

export CW_GATEWAY_ENDPOINT="[GATEWAY-ENDPOINT]"

The gateway’s public DNS record and TLS certificate provision asynchronously while the gateway is created. Wait until the gateway reaches STATUS_READY and exposes an endpoint. If your first inference request fails with an SSL handshake error or DNS resolution failure, wait a few minutes and retry.

Send an inference request

With the deployment ready and the gateway endpoint exported, you can now send your first inference request to verify the end-to-end path. The gateway exposes an OpenAI-compatible API. With body-based routing, the gateway routes requests based on the model field in the request body. Send a chat completion request using the model name from your deployment:

curl -X POST "${CW_GATEWAY_ENDPOINT}/v1/chat/completions" \
  -H "Authorization: Bearer ${CW_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "[MODEL-NAME]",
    "messages": [
      {
        "role": "user",
        "content": "What is CoreWeave?"
      }
    ],
    "max_tokens": 256
  }'

The same CoreWeave API access token used for the management API authenticates inference requests when the gateway uses coreWeaveAuth. The response is an OpenAI-compatible chat completion. Beyond the standard fields, vLLM also returns several engine-specific fields (token_ids, prompt_logprobs, kv_transfer_params, and others). These are typically null for normal requests, so you can ignore them.

{
  "id": "chatcmpl-bcbcbae71847bf87",
  "object": "chat.completion",
  "created": 1777477750,
  "model": "[MODEL-NAME]",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "CoreWeave is a cloud infrastructure provider...",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": null
      },
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 24,
    "total_tokens": 104,
    "completion_tokens": 80,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "kv_transfer_params": null
}

A successful response confirms that your gateway and deployment are working. Alternatively, use the Python openai library:

from openai import OpenAI

client = OpenAI(
    base_url="[GATEWAY-ENDPOINT]/v1/",
    api_key="[API-TOKEN]",
)

response = client.chat.completions.create(
    model="[MODEL-NAME]",
    messages=[
        {"role": "user", "content": "What is CoreWeave?"},
    ],
    max_tokens=256,
)
print(response.choices[0].message.content)

Update a deployment

Over time, you may need to adjust capacity, swap the GPU type, or point the deployment at new model weights. To change a deployment’s configuration, send a PATCH request with the complete deployment specification. All fields are required, not just the ones you change. Omitting a field either fails validation (for required fields) or reverts it to its default, so start from the command you used to create the deployment and change only the fields you want to update. This example increases the autoscaling maximum from 1 to 4:

Changing model.name changes the route key that clients send to the gateway. Update clients to use the new model name after the deployment update completes.

CoreWeave Intelligent CLI
curl

Like the REST PATCH request, cwic inference deployment update sends the complete deployment specification. Omitted flags revert to their default, so re-specify every flag from the create command and change only the ones you want to update:

cwic inference deployment update ${CW_DEPLOYMENT_ID} \
  --name my-first-deployment \
  --gateway-ids ${CW_GATEWAY_ID} \
  --engine vllm \
  --engine-version [ENGINE-VERSION] \
  --instance-type [INSTANCE-TYPE] \
  --gpu-count 1 \
  --model-name [MODEL-NAME] \
  --model-bucket [BUCKET-NAME] \
  --model-path [MODEL-PATH] \
  --min 1 \
  --max 4 \
  --traffic-weight 100

curl -X PATCH "${CW_BASE_URL}/v1alpha1/inference/deployments/${CW_DEPLOYMENT_ID}" \
  -H "Authorization: Bearer ${CW_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "[DEPLOYMENT-ID]",
    "name": "my-first-deployment",
    "gatewayIds": ["[GATEWAY-ID]"],
    "runtime": {
      "engine": "vllm",
      "version": "[ENGINE-VERSION]"
    },
    "resources": {
      "instanceType": "[INSTANCE-TYPE]",
      "gpuCount": 1
    },
    "model": {
      "name": "[MODEL-NAME]",
      "bucket": "[BUCKET-NAME]",
      "path": "[MODEL-PATH]"
    },
    "autoscaling": {
      "min": 1,
      "max": 4
    },
    "traffic": {
      "weight": 100
    }
  }'

If you created the deployment with Terraform, change the coreweave_inference_deployment resource block and run terraform apply instead of calling the API directly.

Observability

To view logs from your deployment:

Go to console.coreweave.com and open Grafana under the Observability section.
In Grafana, click Explore in the left nav, then select CoreWeave Logs from the data source dropdown.
In the query builder, click Code on the right, then enter the following query:
{cluster="cwinference", container="vllm-server"}
Click Run query in the top-right corner to see the logs.

Clean up

When you no longer need the inference endpoint, delete the resources to stop incurring charges. You must delete deployments before their associated gateway.

CoreWeave Intelligent CLI
curl

cwic inference deployment delete ${CW_DEPLOYMENT_ID}

cwic inference gateway delete ${CW_GATEWAY_ID}

curl -X DELETE "${CW_BASE_URL}/v1alpha1/inference/deployments/${CW_DEPLOYMENT_ID}" \
  -H "Authorization: Bearer ${CW_API_TOKEN}"

curl -X DELETE "${CW_BASE_URL}/v1alpha1/inference/gateways/${CW_GATEWAY_ID}" \
  -H "Authorization: Bearer ${CW_API_TOKEN}"

If you created these resources with Terraform, run terraform destroy instead.

After the deployment and gateway are deleted, the inference resources from this guide no longer incur usage charges.

Next steps

You now have a working baseline: a gateway, a deployment serving a model, and a successful inference response. Explore these resources to learn more about CoreWeave Inference and to take the deployment beyond a single-replica baseline.

Gateways: Configure authentication, routing strategies, and traffic splitting.
Models and deployments: Learn about runtime configuration, GPU selection, and deployment options.
Scaling: Configure autoscaling and reserve GPU capacity.
Billing: Understand pricing and optimize inference costs.
Inference API reference: Explore the full API surface.
CoreWeave Provider: Explore the full Terraform resource and data source schema for Inference.

​Prerequisites

​Set your API token

​Grant inference access to your bucket

​Create a gateway

​View all gateways

​Create a deployment

​View all deployments

​Deploy with Terraform

​Wait for the deployment to start

​Send an inference request

​Update a deployment

​Observability

​Clean up

​Next steps

Prerequisites

Set your API token

Grant inference access to your bucket

Create a gateway

View all gateways

Create a deployment

View all deployments

Deploy with Terraform

Wait for the deployment to start

Send an inference request

Update a deployment

Observability

Clean up

Next steps