Skip to main content
A deployment in Dedicated Inference configures a model serving instance that runs your model on dedicated GPU infrastructure. Each deployment specifies the model weights, inference runtime, GPU type, and scaling parameters. This page is for users who create or manage Dedicated Inference deployments. It covers how to configure model weights, select an inference runtime, choose GPU resources, and manage traffic across deployments so you can serve your models reliably on dedicated infrastructure.

Bring your own weights

Dedicated Inference uses a Bring Your Own Weights (BYOW) model. You provide the model artifacts, and CoreWeave handles the infrastructure required to serve them. Model weights must be stored in a CoreWeave AI Object Storage bucket. When you create a deployment, you must specify the bucket name and the path to the model directory within the bucket. When a deployment starts, CoreWeave loads your model weights onto the selected GPU infrastructure and serves requests through the associated gateway.
Dedicated Inference doesn’t pull model weights directly from external sources such as Hugging Face. Download your model weights and upload them to an AI Object Storage bucket before creating a deployment.

Grant inference access to your bucket

Dedicated Inference reads your model weights from AI Object Storage using the Dedicated Inference service account. Attach the following bucket policy to your weights bucket so the service can list and read its contents.
Attaching a bucket policy removes the implicit access your organization’s users had to the bucket. This policy grants access only to the service account, so on its own it locks your users out with AccessDenied errors. To keep their access, add the AllowOrgUsers statement from Grant inference access to your bucket. For trainers that publish RL Rollouts checkpoints, also grant the checkpoint publishing permissions.
Replace [BUCKET-NAME] with the name of the bucket containing your model weights.
If you store models only under a specific prefix, you can narrow object access to that prefix:
The bucket-level ARN (arn:aws:s3:::[BUCKET-NAME]) must remain as-is for s3:ListBucket. Save the policy to a file (for example, inference-bucket-policy.json) and apply it with the AWS CLI. If you’ve configured a named AWS CLI profile for CoreWeave, pass it with --profile [PROFILE-NAME]. Omit the flag if your CoreWeave credentials are in the default profile.
For other tools (s3cmd, Boto3, Terraform) and the full reference, see Manage bucket policies.
The Cloud Console doesn’t support setting bucket policies. Apply the policy with the AWS CLI, an S3 SDK, or Terraform. Organization access policies in the Console are a different mechanism and won’t grant the inference service account access to your bucket.
The Principal value in the preceding policy is the canonical Dedicated Inference service account and is the same for all customers. If a deployment fails to load weights with a 520 or timeout despite reaching STATUS_READY, a missing bucket policy is the most common cause.
After the policy is in place, deployments can read model weights from the bucket when they start.

Model configuration

The model field in a deployment specifies where to find the model weights and what name to use for routing: The model name is how clients identify which deployment should handle their request. For body-based routing, the model field in the request body must match this name. For path-based routing, the model name is the first segment of the URL path.

Inference runtimes

The inference runtime is the engine that loads your model weights and serves requests. CoreWeave manages the inference runtime. The runtime field configures which engine and version to use. You must specify an engine when you create a deployment.

Supported engines

Runtime configuration

Query the available runtime versions and configuration options from the deployment parameters endpoint:

Engine configuration options

The engineConfig field accepts engine-specific key-value pairs that control model serving behavior. Examples of supported keys for dynamo-vllm and vllm: The deployment parameters endpoint returns the full list of allowed configuration keys for each engine under runtimeParameters.runtimeConfigOptions.

SGLang configuration options

The dynamo-sglang engine runs SGLang behind NVIDIA Dynamo and is in preview. It uses the same deployment API, gateways, traffic weights, and autoscaling as the other engines, but its serving options use SGLang keys instead of vLLM keys. Query the deployment parameters endpoint before choosing a version or configuration. Available versions are under runtimeParameters.runtimeVersions.dynamo-sglang.versions, allowed keys under runtimeParameters.runtimeConfigOptions.dynamo-sglang.allowedKeys, and allowed environment variables under runtimeParameters.engineEnvOptions.dynamo-sglang.allowedNames. Use only the options allowed for your selected engine version. Not every option in the upstream SGLang documentation is available on Dedicated Inference. All engineConfig values are strings. To enable a boolean flag, set its value to an empty string, for example "enable-multimodal": "". CoreWeave checks key names but not values. A key outside the allowed list is rejected when you create the deployment. An invalid value, malformed JSON in preferred-sampling-params, or an unsupported combination of flags reaches the engine instead, and the deployment never reaches STATUS_READY. The following table lists commonly used SGLang keys: SGLang chooses the execution backends for your model and GPU when you leave them unset. Some models also override the page size you request. For example, models with DeepSeek sparse attention (DSA), such as GLM 5.3 Flash, use a page size of 64 on NVIDIA GPUs instead of the default 16. Set these keys only when you need a specific value. Allowed environment variables include DYN_TCP_MAX_MESSAGE_SIZE, which applies to all containers, and DYN_ROUTER_MODE and DYN_ROUTER_SESSION_AFFINITY_TTL_SECS, which apply to the frontend. Leave DYN_ROUTER_MODE at its default: any mode other than kv turns off KV cache-aware routing. Worker variables cover health checks, NVIDIA Collective Communications Library (NCCL), and SGLang settings. Check engineEnvOptions before setting them. For a deployment example, see Configure SGLang. For flag definitions, see the SGLang server arguments.

GPU selection

Deployments run on dedicated GPU infrastructure. The resources field configures the hardware allocation that each replica uses to serve your model. Query the available instance types from the deployment parameters endpoint:
The response includes the available instance types under resourceParameters.instanceTypes.

Choose an instance type

Match the instance type to your model’s requirements:
  • Model size: Choose a GPU with enough memory to fit your model weights and the inference runtime’s working memory (KV cache).
  • Throughput: Higher-end GPUs (H200, B200) provide more compute and memory bandwidth for faster inference.
  • Multi-GPU: For models that exceed a single GPU’s memory, increase gpuCount to allocate additional GPUs per replica.

Traffic weights

The traffic.weight field controls how a gateway distributes requests across deployments that serve the same model name. It takes effect only when two or more deployments share a model name on the same gateway. A single deployment receives all of its model’s traffic regardless of its weight, so you can leave the example value of 100 in place until you add a second deployment for that model. When multiple deployments share a model name, the gateway splits requests between them in proportion to their weights, which supports canary releases and A/B testing:
  • Two deployments with equal weights (for example, 100 each) each receive approximately half of the requests over time. The gateway doesn’t guarantee strict request-by-request alternation.
  • If deployment A has weight: 90 and deployment B has weight: 10, deployment A receives 90% of traffic and deployment B receives 10%.
Setting weight to 0 stops traffic to a deployment without deleting it. Traffic weights are separate from the load balancing a gateway performs within a single deployment. Across the replicas of one deployment, the gateway balances requests to optimize throughput and latency. Across multiple deployments that share a model name, traffic.weight controls the share of traffic each deployment receives.

Disable a deployment

Set the disabled field to true to stop a deployment from serving traffic without deleting it. A disabled deployment retains its configuration. To re-enable it, set disabled to false.

Deployment lifecycle

A deployment reports its current state through the status field so you can track provisioning, updates, and failures. Deployments go through the following states:

Manage deployments

Manage deployments through the Dedicated Inference API. For per-operation request and response schemas, see the DeploymentService pages in the API reference.
Update requests (PATCH) require the complete deployment specification, not just the fields you’re changing. Omitted fields revert to their defaults, except the hotLoad section: omitting or changing it is rejected, so resend it unchanged.
For a step-by-step walkthrough, see the Getting started guide.
Last modified on November 10, 2026