Evaluations Quick Start - CoreWeave Docs

Head-to-head evaluation is the best starting point for open-ended tasks where exact text equality would be too strict.

Before you begin, create a dataset with validation rows and add every fine-tune or provider model you want to compare.

Open the dataset's Evaluate tab

Select New evaluation, choose Head to Head, and give the evaluation a descriptive name.

Choose the reference

Select original or relabeled dataset output as the primary participant. Candidates will be scored against this desired behavior.

Add candidate models

Add fine-tunes from the dataset first, then any base or provider models you want to compare. Fine-tunes are grouped by base model in the selector.

Configure the judge

Select a capable judge model and write instructions that define what a better answer means for this task. Start with 200 validation rows when enough are available.

Run and inspect

Start the evaluation. Scores update while rows finish. Open individual rows to read the judge explanation and use the failure indicator to inspect provider or parsing errors.

Decide whether to deploy

A 50% score means parity with the reference: wins count as 1, ties as 0.5, and losses as 0. Promote only after the completed score and per-row behavior meet your threshold.

Failed cases retry automatically up to three attempts. Once the run settles, Retry failed cases requeues only failures and preserves completed results.

API: Create an evaluation (POST /evals)

The first participant is the primary. Replace the candidate model reference with the artifact or provider model you want to evaluate:

curl --request POST \
  --url "https://distillation.training.wandb.ai/v1/evals" \
  --header "Authorization: Bearer $WANDB_API_KEY" \
  --header "Wandb-Entity: your-team" \
  --header "Content-Type: application/json" \
  --data "{
    \"name\": \"ticket-classifier-comparison\",
    \"dataset_id\": \"$DATASET_ID\",
    \"spec\": {
      \"type\": \"h2h_judge\",
      \"judge_model_ref\": \"openai/gpt-5.6-sol\",
      \"judge_prompt\": \"Choose the more correct and helpful answer. Return a tie when they are equally good.\"
    },
    \"participants\": [
      {\"kind\": \"original\"},
      {\"kind\": \"model\", \"model_ref\": \"wandb-inference/your-fine-tuned-model\"}
    ],
    \"sample_size\": 200
  }"

The response contains the evaluation ID. Poll GET /evals/{evalId} until the evaluation settles; see Create an evaluation.

Last modified on August 25, 2026