Skip to main content
Evaluations answer whether a candidate preserves the task behavior you care about. They use stable validation rows and store one durable result per entry and participant.

Run your first evaluation

Compare a fine-tune with reference outputs using a head-to-head judge.

​
Evaluation types

A judge model compares each candidate response with a primary output. Use this for open-ended generation where exact text equality is too strict. Score = (wins + 0.5 × ties) / completed rows; 50% means parity with the primary.
Compares canonical assistant outputs deterministically. Use it for structured or discrete answers where equivalent output should be identical.
Reads a boolean field and reports precision, recall, F1, and the confusion matrix. Unreadable candidate output hurts both precision and recall; unreadable reference output is skipped.

​
Choose participants

Head-to-head evaluations accept original outputs, relabel runs, tuned models, and direct provider models. The first participant is primary. Fine-tunes from the dataset appear near the top and are grouped by base model. Exact-match and categorization evaluations require a separate reference and accept model or provider candidates.

​
Read live results

Results are provisional until all requested rows settle. Studio shows:
  • completed rows versus total rows;
  • wins, ties, losses, and score for head-to-head participants;
  • failed rows and an immediate error-message summary;
  • per-row verdicts in the dataset workbench.
Adding a participant or changing settings reconciles only affected work. Unchanged completed cases and reusable model outputs are preserved.

​
Retries and failure

Each case receives up to three automatic attempts. The evaluation stops retrying after that bound and can complete with failed rows. Once it has settled, Retry failed cases creates a new batch containing only failures; completed results remain intact.
Use identical routing parameters when comparing provider and fine-tuned models. A model that is already a task target automatically receives the highest matching task version’s parameter overrides.
Wait until the evaluation has settled, then submit a failed-only retry batch. Completed results are preserved.
Use Evaluations Quick Start for the create request. See Retry failed evaluation cases for this operation.
Last modified on August 25, 2026