> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Automation

> Configure a training sweep that collects data, evaluates every result, and promotes a qualifying winner.

Automation runs the complete improvement loop for one task. It creates ordinary datasets, fine-tunes, evaluations, and routing revisions, so every result remains visible and usable after the run.

## Configure Automation in Studio

Open the task, select **Automation**, and choose **Create Automation**. Configure the models, dataset, evaluation, and rollout in the four-step wizard, then select **Run Automation** and confirm the summary.

The sections below explain each choice. Agents and scripts can create the same complete workflow with one request.

<Accordion title="API: Start Automation (PUT /tasks/{alias}/automation)">
  One `PUT` request configures the complete workflow and starts its first run. The example below:

  * waits for 5,000 training examples from task traffic;
  * creates a dataset with 20% reserved for evaluation;
  * trains three candidate models;
  * compares them with the current model on 200 examples;
  * sends 25% of version 1 traffic to the winner when it matches or beats the reference.

  Before running it, replace the task alias, entity, model references, and date range with values from your account. The `teacher` must be a model currently receiving traffic for the selected task version.

  ```bash theme={"system"}
  export DISTILLATION_API="https://distillation.training.wandb.ai/v1"
  export WANDB_ENTITY="your-team"
  export TASK_ALIAS="ticket-classifier"
  export AUTOMATION_AFTER="2026-08-01T00:00:00.000Z"
  export AUTOMATION_BEFORE="2026-09-01T00:00:00.000Z"

  curl --request PUT \
    --url "$DISTILLATION_API/tasks/$TASK_ALIAS/automation" \
    --header "Authorization: Bearer $WANDB_API_KEY" \
    --header "Wandb-Entity: $WANDB_ENTITY" \
    --header "Content-Type: application/json" \
    --data "{
      \"teacher\": \"openai/gpt-5.6-sol\",
      \"dataset\": {
        \"source\": \"task\",
        \"task_version\": 1,
        \"after\": \"$AUTOMATION_AFTER\",
        \"before\": \"$AUTOMATION_BEFORE\",
        \"min_train_examples\": 5000,
        \"sampling\": {
          \"strategy\": \"random\",
          \"limit\": 100000
        },
        \"split\": {
          \"val\": 0.2,
          \"by\": \"trace\"
        }
      },
      \"models\": [
        {
          \"base_model\": \"Qwen/Qwen3.6-27B\"
        },
        {
          \"base_model\": \"Qwen/Qwen3-30B-A3B-Instruct-2507\"
        },
        {
          \"base_model\": \"meta-llama/Llama-3.1-8B-Instruct\"
        }
      ],
      \"eval\": {
        \"spec\": {
          \"type\": \"h2h_judge\",
          \"judge_model_ref\": \"openai/gpt-5.6-sol\",
          \"judge_prompt\": \"Choose the response that is more correct, helpful, and complete for the user. Return a tie when they are equally good.\"
        },
        \"sample_size\": 200
      },
      \"rollout\": {
        \"task_version\": 1,
        \"min_win_rate\": 0.5,
        \"winner_traffic\": 0.25
      }
    }"
  ```

  The request returns the resolved configuration and current run. Omitted model configurations are chosen automatically after the dataset is ready. Each object in `models` creates exactly one fine-tune.

  <Note>
    The `before` timestamp can be in the future. Automation starts as soon as it finds enough examples and fails when the window closes without reaching `min_train_examples`.
  </Note>

  **Start from an existing dataset**

  To skip traffic collection, use a ready dataset that belongs to the task. Replace the `dataset` object in the request above with:

  ```json theme={"system"}
  {
    "dataset": {
      "source": "existing",
      "dataset_id": "550e8400-e29b-41d4-a716-446655440000"
    }
  }
  ```

  Add `relabel_run_id` when the Automation should train and evaluate against a completed relabel output:

  ```json theme={"system"}
  {
    "dataset": {
      "source": "existing",
      "dataset_id": "550e8400-e29b-41d4-a716-446655440000",
      "relabel_run_id": "8ea49017-64a7-45ef-ad18-0ccbc8170f42"
    }
  }
  ```

  **Check progress**

  Poll the same resource to read the current stage, progress, generated resources, and final outcome:

  ```bash theme={"system"}
  curl --url "$DISTILLATION_API/tasks/$TASK_ALIAS/automation" \
    --header "Authorization: Bearer $WANDB_API_KEY" \
    --header "Wandb-Entity: $WANDB_ENTITY"
  ```

  See [Create or replace Automation](/model-distillation/reference/management) for the complete request and response schemas.
</Accordion>

## Step 1: Models

Add one entry for every model you want to train. Each entry creates exactly one fine-tune. Repeating a base model creates another run; it never expands into hidden extra models.

* Leave configuration automatic to let the pinned heuristic choose a point in a small batch-size and peak-learning-rate sweep.
* Open customization to set the fine-tune name or training parameters explicitly.
* Up to six automatic entries for one base model receive distinct configurations.

## Step 2: Dataset

Choose one source:

### Build from task traffic

Select the task version, active teacher target, fixed date range, minimum training examples, maximum sample, and validation percentage. Automation waits until the selected range contains enough usable teacher examples before freezing a dataset snapshot.

Defaults are 5,000 minimum training examples, at most 100,000 sampled traces, and 20% validation data.

### Use an existing dataset

Select a ready dataset and either original outputs or a completed relabel run. The run starts from that immutable data immediately instead of collecting traces.

## Step 3: Evaluation

Choose the judge model, prompt, and sample size. Automation runs one head-to-head evaluation that compares every trained model with the selected dataset output on the same held-out rows. The default sample size is 200.

Automation will not promote from incomplete results. Every required case must complete without remaining failures.

## Step 4: Rollout

Choose:

* the task version that should receive the winner;
* the minimum head-to-head win rate;
* the percentage of traffic assigned to the winner.

The default threshold is 50%, which means parity with the reference. The default winner traffic is 100%. Any remaining traffic is distributed across current targets in their existing proportions.

## Run outcomes

| Status           | Meaning                                                  |
| ---------------- | -------------------------------------------------------- |
| Collecting       | Waiting for enough qualifying task traffic               |
| Building dataset | Freezing or validating the selected data                 |
| Training         | One or more sweep models are running                     |
| Evaluating       | Complete models are being judged together                |
| Promoting        | The winner cleared the threshold and routing is updating |
| Promoted         | Routing contains the winner at the configured share      |
| Rejected         | Evaluation completed, but no winner met the threshold    |
| Failed           | A required stage could not complete                      |

Stopping Automation cancels its controller and leaves created datasets, fine-tunes, and evaluations intact. An already submitted training job continues as an ordinary fine-tune. Routing is never rolled back automatically.