Skip to main content
Automation runs the complete improvement loop for one task. It creates ordinary datasets, fine-tunes, evaluations, and routing revisions, so every result remains visible and usable after the run.

​
Configure Automation in Studio

Open the task, select Automation, and choose Create Automation. Configure the models, dataset, evaluation, and rollout in the four-step wizard, then select Run Automation and confirm the summary. The sections below explain each choice. Agents and scripts can create the same complete workflow with one request.
One PUT request configures the complete workflow and starts its first run. The example below:
  • waits for 5,000 training examples from task traffic;
  • creates a dataset with 20% reserved for evaluation;
  • trains three candidate models;
  • compares them with the current model on 200 examples;
  • sends 25% of version 1 traffic to the winner when it matches or beats the reference.
Before running it, replace the task alias, entity, model references, and date range with values from your account. The teacher must be a model currently receiving traffic for the selected task version.
The request returns the resolved configuration and current run. Omitted model configurations are chosen automatically after the dataset is ready. Each object in models creates exactly one fine-tune.
The before timestamp can be in the future. Automation starts as soon as it finds enough examples and fails when the window closes without reaching min_train_examples.
Start from an existing datasetTo skip traffic collection, use a ready dataset that belongs to the task. Replace the dataset object in the request above with:
Add relabel_run_id when the Automation should train and evaluate against a completed relabel output:
Check progressPoll the same resource to read the current stage, progress, generated resources, and final outcome:
See Create or replace Automation for the complete request and response schemas.

​
Step 1: Models

Add one entry for every model you want to train. Each entry creates exactly one fine-tune. Repeating a base model creates another run; it never expands into hidden extra models.
  • Leave configuration automatic to let the pinned heuristic choose a point in a small batch-size and peak-learning-rate sweep.
  • Open customization to set the fine-tune name or training parameters explicitly.
  • Up to six automatic entries for one base model receive distinct configurations.

​
Step 2: Dataset

Choose one source:

​
Build from task traffic

Select the task version, active teacher target, fixed date range, minimum training examples, maximum sample, and validation percentage. Automation waits until the selected range contains enough usable teacher examples before freezing a dataset snapshot. Defaults are 5,000 minimum training examples, at most 100,000 sampled traces, and 20% validation data.

​
Use an existing dataset

Select a ready dataset and either original outputs or a completed relabel run. The run starts from that immutable data immediately instead of collecting traces.

​
Step 3: Evaluation

Choose the judge model, prompt, and sample size. Automation runs one head-to-head evaluation that compares every trained model with the selected dataset output on the same held-out rows. The default sample size is 200. Automation will not promote from incomplete results. Every required case must complete without remaining failures.

​
Step 4: Rollout

Choose:
  • the task version that should receive the winner;
  • the minimum head-to-head win rate;
  • the percentage of traffic assigned to the winner.
The default threshold is 50%, which means parity with the reference. The default winner traffic is 100%. Any remaining traffic is distributed across current targets in their existing proportions.

​
Run outcomes

Stopping Automation cancels its controller and leaves created datasets, fine-tunes, and evaluations intact. An already submitted training job continues as an ordinary fine-tune. Routing is never rolled back automatically.
Last modified on August 25, 2026