> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Compare outputs on stable validation samples and understand live, provisional results.

Evaluations answer whether a candidate preserves the task behavior you care about. They use stable validation rows and store one durable result per entry and participant.

<Card title="Run your first evaluation" href="/model-distillation/studio/evaluations-quickstart" arrow="true">
  Compare a fine-tune with reference outputs using a head-to-head judge.
</Card>

## Evaluation types

<Accordion title="Head to Head">
  A judge model compares each candidate response with a primary output. Use this for open-ended generation where exact text equality is too strict. Score = `(wins + 0.5 × ties) / completed rows`; 50% means parity with the primary.
</Accordion>

<Accordion title="Exact Match">
  Compares canonical assistant outputs deterministically. Use it for structured or discrete answers where equivalent output should be identical.
</Accordion>

<Accordion title="Categorization">
  Reads a boolean field and reports precision, recall, F1, and the confusion matrix. Unreadable candidate output hurts both precision and recall; unreadable reference output is skipped.
</Accordion>

## Choose participants

Head-to-head evaluations accept original outputs, relabel runs, tuned models, and direct provider models. The first participant is primary. Fine-tunes from the dataset appear near the top and are grouped by base model.

Exact-match and categorization evaluations require a separate reference and accept model or provider candidates.

## Read live results

Results are provisional until all requested rows settle. Studio shows:

* completed rows versus total rows;
* wins, ties, losses, and score for head-to-head participants;
* failed rows and an immediate error-message summary;
* per-row verdicts in the dataset workbench.

Adding a participant or changing settings reconciles only affected work. Unchanged completed cases and reusable model outputs are preserved.

## Retries and failure

Each case receives up to three automatic attempts. The evaluation stops retrying after that bound and can complete with failed rows. Once it has settled, **Retry failed cases** creates a new batch containing only failures; completed results remain intact.

<Tip>
  Use identical routing parameters when comparing provider and fine-tuned models. A model that is already a task target automatically receives the highest matching task version's parameter overrides.
</Tip>

<Accordion title="API: Retry failed cases (POST /evals/{evalId}/retry)">
  Wait until the evaluation has settled, then submit a failed-only retry batch. Completed results are preserved.

  ```bash theme={"system"}
  curl --request POST \
    --url "https://distillation.training.wandb.ai/v1/evals/$EVAL_ID/retry" \
    --header "Authorization: Bearer $WANDB_API_KEY" \
    --header "Wandb-Entity: your-team"
  ```

  Use [Evaluations Quick Start](/model-distillation/studio/evaluations-quickstart) for the create request. See [Retry failed evaluation cases](/model-distillation/reference/management) for this operation.
</Accordion>