Bring your own weights
Dedicated Inference uses a Bring Your Own Weights (BYOW) model. You provide the model artifacts, and CoreWeave handles the infrastructure required to serve them. Model weights must be stored in a CoreWeave AI Object Storage bucket. When you create a deployment, you must specify the bucket name and the path to the model directory within the bucket. When a deployment starts, CoreWeave loads your model weights onto the selected GPU infrastructure and serves requests through the associated gateway.Dedicated Inference doesn’t pull model weights directly from external sources such as Hugging Face. Download your model weights and upload them to an AI Object Storage bucket before creating a deployment.
Grant inference access to your bucket
Dedicated Inference reads your model weights from AI Object Storage using the Dedicated Inference service account. Attach the following bucket policy to your weights bucket so the service can list and read its contents. Replace[BUCKET-NAME] with the name of the bucket containing your model weights.
arn:aws:s3:::[BUCKET-NAME]) must remain as-is for s3:ListBucket.
Save the policy to a file (for example, inference-bucket-policy.json) and apply it with the AWS CLI. If you’ve configured a named AWS CLI profile for CoreWeave, pass it with --profile [PROFILE-NAME]. Omit the flag if your CoreWeave credentials are in the default profile.
s3cmd, Boto3, Terraform) and the full reference, see Manage bucket policies.
The
Principal value in the preceding policy is the canonical Dedicated Inference service account and is the same for all customers. If a deployment fails to load weights with a 520 or timeout despite reaching STATUS_READY, a missing bucket policy is the most common cause.Model configuration
Themodel field in a deployment specifies where to find the model weights and what name to use for routing:
The model
name is how clients identify which deployment should handle their request. For body-based routing, the model field in the request body must match this name. For path-based routing, the model name is the first segment of the URL path.
Inference runtimes
The inference runtime is the engine that loads your model weights and serves requests. CoreWeave manages the inference runtime. Theruntime field configures which engine and version to use. You must specify an engine when you create a deployment.
Supported engines
Runtime configuration
Query the available runtime versions and configuration options from the deployment parameters endpoint:
Engine configuration options
TheengineConfig field accepts engine-specific key-value pairs that control model serving behavior. Examples of supported keys for dynamo-vllm and vllm:
The deployment parameters endpoint returns the full list of allowed configuration keys for each engine under
runtimeParameters.runtimeConfigOptions.
SGLang configuration options
Thedynamo-sglang engine runs SGLang behind NVIDIA Dynamo and is in preview. It uses the same deployment API, gateways, traffic weights, and autoscaling as the other engines, but its serving options use SGLang keys instead of vLLM keys.
Query the deployment parameters endpoint before choosing a version or configuration. Available versions are under runtimeParameters.runtimeVersions.dynamo-sglang.versions, allowed keys under runtimeParameters.runtimeConfigOptions.dynamo-sglang.allowedKeys, and allowed environment variables under runtimeParameters.engineEnvOptions.dynamo-sglang.allowedNames. Use only the options allowed for your selected engine version. Not every option in the upstream SGLang documentation is available on Dedicated Inference.
All engineConfig values are strings. To enable a boolean flag, set its value to an empty string, for example "enable-multimodal": "".
CoreWeave checks key names but not values. A key outside the allowed list is rejected when you create the deployment. An invalid value, malformed JSON in preferred-sampling-params, or an unsupported combination of flags reaches the engine instead, and the deployment never reaches STATUS_READY.
The following table lists commonly used SGLang keys:
SGLang chooses the execution backends for your model and GPU when you leave them unset. Some models also override the page size you request. For example, models with DeepSeek sparse attention (DSA), such as GLM 5.3 Flash, use a page size of 64 on NVIDIA GPUs instead of the default
16. Set these keys only when you need a specific value.
Allowed environment variables include DYN_TCP_MAX_MESSAGE_SIZE, which applies to all containers, and DYN_ROUTER_MODE and DYN_ROUTER_SESSION_AFFINITY_TTL_SECS, which apply to the frontend. Leave DYN_ROUTER_MODE at its default: any mode other than kv turns off KV cache-aware routing. Worker variables cover health checks, NVIDIA Collective Communications Library (NCCL), and SGLang settings. Check engineEnvOptions before setting them.
For a deployment example, see Configure SGLang. For flag definitions, see the SGLang server arguments.
GPU selection
Deployments run on dedicated GPU infrastructure. Theresources field configures the hardware allocation that each replica uses to serve your model.
Query the available instance types from the deployment parameters endpoint:
resourceParameters.instanceTypes.
Choose an instance type
Match the instance type to your model’s requirements:- Model size: Choose a GPU with enough memory to fit your model weights and the inference runtime’s working memory (KV cache).
- Throughput: Higher-end GPUs (H200, B200) provide more compute and memory bandwidth for faster inference.
- Multi-GPU: For models that exceed a single GPU’s memory, increase
gpuCountto allocate additional GPUs per replica.
Traffic weights
Thetraffic.weight field controls how a gateway distributes requests across deployments that serve the same model name. It takes effect only when two or more deployments share a model name on the same gateway. A single deployment receives all of its model’s traffic regardless of its weight, so you can leave the example value of 100 in place until you add a second deployment for that model.
When multiple deployments share a model name, the gateway splits requests between them in proportion to their weights, which supports canary releases and A/B testing:
- Two deployments with equal weights (for example,
100each) each receive approximately half of the requests over time. The gateway doesn’t guarantee strict request-by-request alternation. - If deployment A has
weight: 90and deployment B hasweight: 10, deployment A receives 90% of traffic and deployment B receives 10%.
weight to 0 stops traffic to a deployment without deleting it.
Traffic weights are separate from the load balancing a gateway performs within a single deployment. Across the replicas of one deployment, the gateway balances requests to optimize throughput and latency. Across multiple deployments that share a model name, traffic.weight controls the share of traffic each deployment receives.
Disable a deployment
Set thedisabled field to true to stop a deployment from serving traffic without deleting it. A disabled deployment retains its configuration. To re-enable it, set disabled to false.
Deployment lifecycle
A deployment reports its current state through thestatus field so you can track provisioning, updates, and failures. Deployments go through the following states: