coreweave_inference_deployment (Resource)

​

Example usage

# Look up available parameters first (optional but recommended).
data "coreweave_inference_deployment_parameters" "deploy_params" {}

resource "coreweave_inference_deployment" "example" {
  name        = "my-llm"
  gateway_ids = [tolist(data.coreweave_inference_deployment_parameters.deploy_params.gateway_ids)[0]]

  runtime = {
    engine  = "vllm"
    version = "0.8.5"
    engine_config = {
      "max-model-len" = "8192"
    }
    engine_env = {
      VLLM_USE_FLASHINFER_MOE_FP4 = "0"
    }
  }

  resources = {
    instance_type = "H100_80GB_SXM5"
    gpu_count     = 1
  }

  model = {
    name   = "meta-llama/Llama-3.1-8B"
    bucket = "my-model-bucket"
    path   = "models/llama-3.1-8b"
  }

  autoscaling = {
    min              = 1
    max              = 4
    priority         = 100
    capacity_classes = ["CAPACITY_CLASS_RESERVED", "CAPACITY_CLASS_ON_DEMAND"]
    concurrency      = 16
  }

  traffic = {
    weight = 100
  }

  # To enable checkpoint hot-loading on a supported Dynamo runtime, configure
  # this immutable object at creation. Changes require deployment replacement.
  # hot_load = {
  #   bucket          = "my-checkpoint-bucket"
  #   path_prefix     = "runs/my-model"
  #   transition_mode = "TRANSITION_MODE_ASYNC"
  # }
}

​

Schema

​

Required

• autoscaling (Attributes) Autoscaling configuration. (see below for nested schema)
• gateway_ids (Set of String) The gateway IDs to associate the deployment with. At least one is required.
• model (Attributes) Model configuration. (see below for nested schema)
• name (String) The name of the deployment. Must be a valid hostname label.
• resources (Attributes) GPU resource configuration for the deployment. (see below for nested schema)
• runtime (Attributes) Runtime selection and configuration. (see below for nested schema)

​

Optional

• disabled (Boolean) Whether the deployment is disabled.
• hot_load (Attributes) Checkpoint hot-loading configuration. Immutable after creation; adding, changing or removing it replaces the deployment. (see below for nested schema)
• pd (Attributes) Prefill/decode disaggregation. When set, resources and autoscaling configure the decode pool and this object configures the prefill pool. Requires the dynamo-vllm engine and an organization enabled for prefill/decode disaggregation. Adding, changing or removing it updates the deployment in place. (see below for nested schema)
• traffic (Attributes) Traffic configuration. Omit to accept the API default (weight 0, which normalizes to 100% when no other deployment shares the model name). After apply, weight is populated from the API. (see below for nested schema)

​

Read-Only

• conditions (Attributes List) Detailed status conditions for the deployment. (see below for nested schema)
• created_at (String) RFC3339 timestamp of when the deployment was created.
• id (String) The unique identifier of the deployment.
• organization_id (String) The organization ID that owns the deployment.
• status (String) The current status of the deployment. See the Inference API overview for status values. Creation and updates that disable the deployment return once resources are applied rather than once the deployment is serving, so the status may not be ready immediately. An update to a serving deployment instead blocks until the rollout reaches STATUS_READY. Poll the API for live readiness.
• updated_at (String) RFC3339 timestamp of when the deployment was last updated.

​

Nested Schema for autoscaling

• max (Number) Maximum number of instances. Must be ≥1.
• min (Number) Minimum number of instances. Must be ≥1.

• capacity_classes (List of String) Ordered preference list of capacity classes to use. Order is significant: the first satisfiable class wins. Allowed values: CAPACITY_CLASS_RESERVED, CAPACITY_CLASS_ON_DEMAND.
• concurrency (Number) Concurrency per instance target (≥1). Controls latency vs throughput tradeoffs.
• priority (Number) Priority for cross-deployment scaling (0-1000). Higher values win when there is contention.

​

Nested Schema for model

• bucket (String) The CAIOS bucket the model is stored in. The inference service account must have bucket access.
• name (String) The model name used in API requests (e.g. the /models endpoint). Length must be 4-63 characters.
• path (String) The CAIOS path to the model and its configuration files.

​

Nested Schema for resources

• gpu_count (Number) Number of GPUs per instance. Must be one of: 1, 2, 4, 8, 16, 32, 64. Values above 16 require an enabled organization.
• instance_type (String) The instance type to use.

​

Nested Schema for runtime

• engine (String) The inference engine to use.

• engine_config (Map of String) Engine-specific configuration key/value pairs.
• engine_env (Map of String) Engine-specific environment variables to inject into the model runtime container. Variable names must come from the selected engine version’s server-side allow list, exposed by data.coreweave_inference_deployment_parameters.<name>.runtime_engine_options[<engine>].versions[<version>].env_options; unsupported names are rejected by the API.
• version (String) The version of the engine. If not set, defaults to the latest available version. Must follow semver format (e.g. 1.2.3).

​

Nested Schema for hot_load

• bucket (String) CoreWeave Object Storage bucket holding checkpoint snapshots.
• transition_mode (String) Request transition mode during checkpoint swaps. Use TRANSITION_MODE_ASYNC.

• path_prefix (String) Path prefix holding snapshots. Defaults to an empty string (bucket root). The snapshot identity is appended to this prefix.

​

Nested Schema for pd

• prefill_autoscaling (Attributes) Autoscaling configuration for the prefill pool. The deployment-wide autoscaling.priority applies to both pools. The prefill pool uses the API default capacity classes and does not inherit autoscaling.capacity_classes. (see below for nested schema)
• prefill_resources (Attributes) GPU resource configuration for the prefill pool. (see below for nested schema)

​

Nested Schema for pd.prefill_autoscaling

• max (Number) Maximum number of prefill instances. Must be ≥1.
• min (Number) Minimum number of prefill instances. Must be ≥1.

• concurrency (Number) Concurrency per prefill instance target (≥1).

​

Nested Schema for pd.prefill_resources

• gpu_count (Number) Number of GPUs per prefill instance. Must be one of: 1, 2, 4, 8, 16, 32, 64. Values above 16 require an enabled organization.
• instance_type (String) The instance type to use for prefill workers.

​

Nested Schema for traffic

• weight (Number) Traffic weight (0-1000). Values are normalized into percentages across deployments with the same model name.

​

Nested Schema for conditions

• last_update_time (String) RFC3339 timestamp of the last condition transition.
• message (String) A human-readable message about the condition’s last transition.
• reason (String) A short, machine-readable reason for the condition’s last transition.
• status (String) The condition status (True, False, or Unknown).
• type (String) The condition type (e.g. Ready, Progressing).

​

Import

terraform import coreweave_inference_deployment.example {{deployment-id}}