> ## Documentation Index > Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt > Use this file to discover all available pages before exploring further. # Create an evaluation ## OpenAPI ````yaml /openapi/model-distillation/management.openapi.yaml post /evals openapi: 3.1.0 info: title: Model Distillation Management API version: 1.0.0 description: | Manage providers, tasks, routing versions, datasets, relabeling, fine-tunes, evaluations, analytics, and closed-loop training Automation. This is the same API used by Model Distillation Studio. servers: - url: https://distillation.training.wandb.ai/v1 description: Production security: - WandbApiKey: [] tags: - name: Providers description: OpenAI-compatible endpoints, credentials, model catalogs, and pricing. - name: Tasks description: Stable task identity and task-level information. - name: Routing description: Versioned model targets, weights, and request parameters. - name: Datasets description: >- Reproducible data snapshots, entries, relabeling, and reusable model outputs. - name: Fine-tunes description: Supervised fine-tuning jobs and hosted model artifacts. - name: Evaluations description: Head-to-head, exact-match, and categorization evaluations. - name: Automation description: Closed-loop dataset, sweep, evaluation, and promotion runs. - name: Analytics description: Task traffic, token, error, and estimated-cost read models. - name: Studio preferences description: Per-task display preferences used by Studio. paths: /evals: parameters: - $ref: '#/components/parameters/WandbEntity' post: tags: - Evaluations summary: Create an evaluation operationId: createEval requestBody: required: true content: application/json: schema: type: object properties: name: type: string minLength: 1 maxLength: 128 description: >- A short, scannable evaluation name. Prefer 2-5 words and avoid embedding the dataset name, every compared model, or configuration details. dataset_id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ spec: oneOf: - type: object properties: type: type: string const: h2h_judge judge_model_ref: type: string pattern: ^[a-z0-9][a-z0-9_-]{0,63}\/\S+$ judge_prompt: default: >- You are an intelligent and fair judge of chatbots. Evaluate the following two responses and choose which one is better. If the outputs are of similar quality, you can mark them as a tie. type: string minLength: 1 reference_conversation_count: default: 0 type: integer minimum: 0 maximum: 50 required: - type - judge_model_ref additionalProperties: false - type: object additionalProperties: false required: - type properties: type: type: string const: exact_match - type: object additionalProperties: false required: - type - field properties: type: type: string const: categorization field: type: string minLength: 1 maxLength: 256 participants: minItems: 1 type: array items: anyOf: - oneOf: - type: object additionalProperties: false required: - kind properties: kind: type: string const: original - type: object properties: kind: type: string const: relabel relabel_run_id: type: string format: uuid pattern: >- ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$ required: - kind - relabel_run_id additionalProperties: false - type: object properties: kind: type: string const: model model_ref: type: string pattern: ^[a-z0-9][a-z0-9_-]{0,63}\/\S+$ required: - kind - model_ref additionalProperties: false - type: object properties: kind: type: string const: provider model_ref: type: string pattern: ^[a-z0-9][a-z0-9_-]{0,63}\/\S+$ required: - kind - model_ref additionalProperties: false - oneOf: - type: object properties: kind: type: string const: model model_ref: type: string pattern: ^[a-z0-9][a-z0-9_-]{0,63}\/\S+$ required: - kind - model_ref additionalProperties: false - type: object properties: kind: type: string const: provider model_ref: type: string pattern: ^[a-z0-9][a-z0-9_-]{0,63}\/\S+$ required: - kind - model_ref additionalProperties: false reference: oneOf: - type: object additionalProperties: false required: - kind properties: kind: type: string const: original - type: object properties: kind: type: string const: model model_ref: type: string pattern: ^[a-z0-9][a-z0-9_-]{0,63}\/\S+$ required: - kind - model_ref additionalProperties: false - type: object properties: kind: type: string const: provider model_ref: type: string pattern: ^[a-z0-9][a-z0-9_-]{0,63}\/\S+$ required: - kind - model_ref additionalProperties: false sample_size: type: integer exclusiveMinimum: 0 maximum: 9007199254740991 required: - name - dataset_id - spec - participants - sample_size additionalProperties: false responses: '202': description: Evaluation cases were queued. content: application/json: schema: $ref: '#/components/schemas/Eval' default: $ref: '#/components/responses/Error' components: parameters: WandbEntity: name: Wandb-Entity in: header required: false description: >- Accessible personal or team entity. Omit to use the API key's default entity. schema: type: string schemas: Eval: type: object additionalProperties: true required: - id - dataset_id - name - spec - participants - sample_size - status - progress - created_at - updated_at properties: id: type: string format: uuid dataset_id: type: string format: uuid name: type: string spec: $ref: '#/components/schemas/EvalSpec' participants: type: array items: $ref: '#/components/schemas/EvalParticipant' reference: oneOf: - $ref: '#/components/schemas/EvalParticipant' - type: 'null' sample_size: type: integer status: type: string enum: - queued - running - completed - failed - stale progress: type: object required: - total - queued - running - completed - failed properties: total: type: integer queued: type: integer running: type: integer completed: type: integer failed: type: integer results: oneOf: - $ref: '#/components/schemas/JsonObject' - type: 'null' failure_summary: type: object additionalProperties: true created_at: type: string format: date-time updated_at: type: string format: date-time EvalSpec: oneOf: - type: object additionalProperties: false required: - type - judge_model_ref properties: type: type: string const: h2h_judge judge_model_ref: $ref: '#/components/schemas/ModelRef' judge_prompt: type: string minLength: 1 - type: object additionalProperties: false required: - type properties: type: type: string const: exact_match - type: object additionalProperties: false required: - type - field properties: type: type: string const: categorization field: type: string minLength: 1 maxLength: 256 EvalParticipant: oneOf: - type: object additionalProperties: false required: - kind properties: kind: type: string const: original - type: object additionalProperties: false required: - kind - relabel_run_id properties: kind: type: string const: relabel relabel_run_id: type: string format: uuid - type: object additionalProperties: false required: - kind - model_ref properties: kind: type: string enum: - model - provider model_ref: $ref: '#/components/schemas/ModelRef' JsonObject: type: object additionalProperties: true ErrorResponse: type: object required: - error properties: error: type: object required: - message - type properties: message: type: string type: type: string example: error: message: Task 'missing' not found in entity 'your-team' type: not_found ModelRef: type: string pattern: ^[a-z0-9][a-z0-9_-]{0,63}/\S+$ example: openai/gpt-5.6-sol responses: Error: description: Request failed. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' securitySchemes: WandbApiKey: type: http scheme: bearer bearerFormat: W&B API key description: A personal or service-account W&B API key. ````