> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Upload a Dataset

> Create a dataset from a JSONL file you already have by uploading it through the Management API.

If your training examples already exist outside the project, upload them as a JSONL file instead of recording traffic through the [inference proxy](/model-distillation/proxy/overview). Model Distillation validates the file, assigns training and validation splits, and creates a dataset that works with relabeling, fine-tuning, and evaluations like any other dataset.

Uploads are available only through the Management API. The UI shows the resulting dataset and an **Uploaded source** panel in the dataset settings, but it doesn't offer a file picker.

This page describes how an upload progresses, how to prepare the file, how to upload it in parts, and how to read the validation results.

## Prerequisites

Before you begin, make sure you have the following:

* A W\&B API key for a team that has access to Model Distillation. The examples read it from the `WANDB_API_KEY` environment variable, pass it in the `Authorization` header, and name the entity in the `Wandb-Entity` header.
* A project in that team. The project alias appears in every request path. To create one, see the [Quick Start](/model-distillation/quickstart).
* The tools for the tab you plan to use in [Upload the file](#upload-the-file). You need only one of the following sets:
  * `curl` together with `jq` and `split`.
  * Python 3.9 or later with the `requests` package.
  * Node.js 18 or later, with no extra packages.

## How an upload works

An upload is an *import session* scoped to a project. One session holds exactly one JSONL file, which you send directly to object storage in parts. The session moves through the following states:

| State                                            | Meaning                                                                                                        |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- |
| `uploading`                                      | The session accepts part uploads.                                                                              |
| `queued`, `validating`, `auditing`, `committing` | The upload is complete. Model Distillation parses rows, checks splits and duplicates, and writes the dataset.  |
| `ready`                                          | The dataset exists. Its ID is in `dataset_id`.                                                                 |
| `failed`                                         | Validation or dataset creation failed. `error` explains why, and `validation.errors` lists row-level problems. |
| `cancelled`                                      | You deleted the session before it reached `ready`.                                                             |
| `expired`                                        | The session stayed in `uploading` for 7 days without being completed.                                          |

A session that is still `uploading` 7 days after creation expires. Part and completion requests to it return `410 Gone`, and an hourly job moves it to `expired` and discards its parts. Uploaded objects and incomplete multipart uploads are deleted 8 days after they were written, whether or not the session completed.

An entity is the team or personal account named in the `Wandb-Entity` header. Each entity can have two sessions in progress at a time, with up to 2 GiB of declared file size between them. A session counts toward both limits from creation until it reaches `ready`, `failed`, `cancelled`, or `expired`. A request that would exceed either limit returns `429 Too Many Requests`. To free capacity, cancel a session you no longer need, or wait for a running session to finish.

## Prepare the file

Before you create an import session, make sure the file matches the row format and stays within the limits described in this section. Validation runs only after you complete the upload, and a failed session can't be reopened. To fix a formatting problem, create a new session and upload the file again.

The file must be UTF-8 JSONL: one JSON object per line, with no duplicate keys within an object.

### Row format

Each row is an OpenAI Chat Completions request whose last message is the assistant response to train toward. Model Distillation stores the preceding messages, plus any `tools`, `tool_choice`, and `response_format`, as the input, and the final assistant message as the output.

```json theme={"system"}
{"messages":[{"role":"system","content":"Classify the ticket."},{"role":"user","content":"My invoice is wrong."},{"role":"assistant","content":"billing"}],"row_id":"ticket-1042"}
```

A row must contain at least two messages and can carry up to 1,000. `tools` accepts up to 128 function tools and 1 MiB of JSON. A row whose last message isn't from the assistant fails with `missing_assistant_target`.

### Optional fields

Rows can carry the following optional fields:

| Field        | Purpose                                                                                                                   |
| ------------ | ------------------------------------------------------------------------------------------------------------------------- |
| `row_id`     | Stable identifier, up to 512 characters. Must be unique within the file. Without it, the line number identifies the row.  |
| `group_id`   | Keeps related rows in the same split, up to 512 characters. Defaults to the row identity.                                 |
| `split`      | `train` or `val`. Required on every row when `split_policy.mode` is `preserve`, and ignored when the mode is `automatic`. |
| `metadata`   | Arbitrary JSON, up to 64 KiB and 10 levels deep.                                                                          |
| `provenance` | Arbitrary JSON that records where the row came from, with the same limits as `metadata`.                                  |

Rows must not contain other top-level fields.

### Limits

Files and import sessions must stay within the following limits:

| Limit                                                  | Value                       |
| ------------------------------------------------------ | --------------------------- |
| File size                                              | 1 GiB                       |
| Rows per file                                          | 100,000                     |
| Row size                                               | 10 MiB                      |
| JSON nesting                                           | 64 levels                   |
| Part size                                              | 32 MiB, fixed by the server |
| Sessions in progress per entity                        | 2                           |
| Total `size_bytes` of sessions in progress, per entity | 2 GiB                       |

## Upload the file

An upload proceeds in stages: create the session, upload the file in parts, complete the upload, and poll until the dataset is ready. The following tabs show the whole flow as one script in `curl`, Python, and JavaScript. Numbered comments mark the stages, and [How the script works](#how-the-script-works) explains each one.

The scripts use a project alias of `ticket-classifier` and a file named `tickets.jsonl`. Before you run a script, replace the entity, alias, and filename with your own, and set `WANDB_API_KEY` in your environment. Use a new `idempotency_key` for each distinct upload.

<Tabs>
  <Tab title="curl">
    ```bash theme={"system"}
    #!/usr/bin/env bash
    # Requires curl, jq, and split.
    set -euo pipefail

    API="https://distillation.training.wandb.ai/v1/tasks/ticket-classifier"
    ENTITY="your-team"
    FILE="tickets.jsonl"
    auth=(--header "Authorization: Bearer $WANDB_API_KEY" --header "Wandb-Entity: $ENTITY")

    # 1. Create the import session.
    size=$(wc -c < "$FILE" | tr -d ' ')
    curl --silent --fail --request POST --url "$API/dataset-imports" "${auth[@]}" \
      --header "Content-Type: application/json" \
      --data "$(jq -n --arg name "$FILE" --argjson size "$size" '{
        idempotency_key: "support-tickets-v1",
        name: "Support tickets (uploaded)",
        file: { name: $name, size_bytes: $size },
        split_policy: { mode: "automatic", val_fraction: 0.1 },
        duplicate_policy: { split_overlap: "drop_train" }
      }')" --output session.json
    import_id=$(jq -r '.id' session.json)
    part_size=$(jq -r '.file.part_size_bytes' session.json)
    part_count=$(jq -r '.file.part_count' session.json)

    # 2. Request a signed upload URL for every part.
    curl --silent --fail --request POST --url "$API/dataset-imports/$import_id/parts" "${auth[@]}" \
      --header "Content-Type: application/json" \
      --data "$(jq -n --argjson count "$part_count" '{ part_numbers: [range(1; $count + 1)] }')" \
      --output urls.json

    # 3. Upload the parts. Part numbers start at 1. Signed URLs carry their own authorization.
    split -b "$part_size" -d -a 4 "$FILE" part-
    n=1
    for chunk in part-*; do
      url=$(jq -r --argjson n "$n" '.parts[] | select(.part_number == $n) | .url' urls.json)
      curl --silent --fail --request PUT --upload-file "$chunk" "$url"
      n=$((n + 1))
    done
    rm part-*

    # 4. Optional: read the session to see which parts arrived.
    curl --silent --fail --url "$API/dataset-imports/$import_id" "${auth[@]}" | jq '.file | {uploaded_bytes, parts}'

    # 5. Complete the upload. This queues validation and dataset creation.
    curl --silent --fail --request POST --url "$API/dataset-imports/$import_id/complete" "${auth[@]}" --output /dev/null

    # 6. Poll until the session is ready or failed.
    while true; do
      curl --silent --fail --url "$API/dataset-imports/$import_id" "${auth[@]}" --output status.json
      case "$(jq -r '.state' status.json)" in
        ready|failed) break ;;
      esac
      sleep 10
    done
    jq '{state, dataset_id, error}' status.json
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={"system"}
    # Requires Python 3.9 or later and the requests package.
    import os
    import time

    import requests

    API = "https://distillation.training.wandb.ai/v1/tasks/ticket-classifier"
    HEADERS = {
        "Authorization": f"Bearer {os.environ['WANDB_API_KEY']}",
        "Wandb-Entity": "your-team",
    }
    FILE = "tickets.jsonl"

    # 1. Create the import session.
    response = requests.post(
        f"{API}/dataset-imports",
        headers=HEADERS,
        json={
            "idempotency_key": "support-tickets-v1",
            "name": "Support tickets (uploaded)",
            "file": {"name": FILE, "size_bytes": os.path.getsize(FILE)},
            "split_policy": {"mode": "automatic", "val_fraction": 0.1},
            "duplicate_policy": {"split_overlap": "drop_train"},
        },
    )
    response.raise_for_status()
    session = response.json()
    import_id = session["id"]
    part_size = session["file"]["part_size_bytes"]
    part_count = session["file"]["part_count"]

    # 2. Request a signed upload URL for every part.
    response = requests.post(
        f"{API}/dataset-imports/{import_id}/parts",
        headers=HEADERS,
        json={"part_numbers": list(range(1, part_count + 1))},
    )
    response.raise_for_status()
    url_by_part = {part["part_number"]: part["url"] for part in response.json()["parts"]}

    # 3. Upload the parts. Part numbers start at 1. Signed URLs carry their own authorization.
    with open(FILE, "rb") as handle:
        for part_number in range(1, part_count + 1):
            chunk = handle.read(part_size)
            requests.put(url_by_part[part_number], data=chunk).raise_for_status()

    # 4. Optional: read the session to see which parts arrived.
    progress = requests.get(f"{API}/dataset-imports/{import_id}", headers=HEADERS).json()
    uploaded = {part["part_number"] for part in progress["file"]["parts"]}
    missing = [n for n in range(1, part_count + 1) if n not in uploaded]

    # 5. Complete the upload. This queues validation and dataset creation.
    requests.post(f"{API}/dataset-imports/{import_id}/complete", headers=HEADERS).raise_for_status()

    # 6. Poll until the session is ready or failed.
    while True:
        status = requests.get(f"{API}/dataset-imports/{import_id}", headers=HEADERS).json()
        if status["state"] in ("ready", "failed"):
            break
        time.sleep(10)

    print(status["state"], status["dataset_id"], status["error"])
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={"system"}
    // Requires Node.js 18 or later. No packages to install.
    import { open, stat } from "node:fs/promises";

    const API = "https://distillation.training.wandb.ai/v1/tasks/ticket-classifier";
    const HEADERS = {
      Authorization: `Bearer ${process.env.WANDB_API_KEY}`,
      "Wandb-Entity": "your-team",
    };
    const FILE = "tickets.jsonl";

    async function call(path, init = {}) {
      const response = await fetch(`${API}${path}`, { ...init, headers: { ...HEADERS, ...init.headers } });
      if (!response.ok) throw new Error(`${init.method ?? "GET"} ${path} failed: ${await response.text()}`);
      return response.status === 204 ? null : response.json();
    }

    // 1. Create the import session.
    const { size } = await stat(FILE);
    const session = await call("/dataset-imports", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        idempotency_key: "support-tickets-v1",
        name: "Support tickets (uploaded)",
        file: { name: FILE, size_bytes: size },
        split_policy: { mode: "automatic", val_fraction: 0.1 },
        duplicate_policy: { split_overlap: "drop_train" },
      }),
    });
    const importId = session.id;
    const partSize = session.file.part_size_bytes;
    const partCount = session.file.part_count;

    // 2. Request a signed upload URL for every part.
    const urls = await call(`/dataset-imports/${importId}/parts`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ part_numbers: Array.from({ length: partCount }, (_, i) => i + 1) }),
    });
    const urlByPart = new Map(urls.parts.map((part) => [part.part_number, part.url]));

    // 3. Upload the parts. Part numbers start at 1. Signed URLs carry their own authorization.
    const handle = await open(FILE, "r");
    try {
      for (let partNumber = 1; partNumber <= partCount; partNumber++) {
        const offset = (partNumber - 1) * partSize;
        const chunk = Buffer.alloc(Math.min(partSize, size - offset));
        await handle.read(chunk, 0, chunk.length, offset);
        const put = await fetch(urlByPart.get(partNumber), { method: "PUT", body: chunk });
        if (!put.ok) throw new Error(`Part ${partNumber} failed with ${put.status}`);
      }
    } finally {
      await handle.close();
    }

    // 4. Optional: read the session to see which parts arrived.
    const progress = await call(`/dataset-imports/${importId}`);
    const uploaded = new Set(progress.file.parts.map((part) => part.part_number));
    const missing = Array.from({ length: partCount }, (_, i) => i + 1).filter((n) => !uploaded.has(n));

    // 5. Complete the upload. This queues validation and dataset creation.
    await call(`/dataset-imports/${importId}/complete`, { method: "POST" });

    // 6. Poll until the session is ready or failed.
    let status;
    while (true) {
      status = await call(`/dataset-imports/${importId}`);
      if (status.state === "ready" || status.state === "failed") break;
      await new Promise((resolve) => setTimeout(resolve, 10_000));
    }
    console.log(status.state, status.dataset_id, status.error);
    ```
  </Tab>
</Tabs>

### How the script works

The numbered comments in each script correspond to the following stages.

<Steps>
  <Step title="Create the import session">
    The script sends the filename and exact byte size, the split policy, and an optional duplicate policy. To learn what each policy does, see [Splits and duplicates](#splits-and-duplicates). The request also requires a unique `idempotency_key`, so a retried request returns the existing session instead of creating a second one.

    The response is `201 Created` with the session. The script keeps `id`, `file.part_size_bytes`, and `file.part_count` from it. The following excerpt shows those fields:

    ```json theme={"system"}
    {
      "id": "3f9c2d1e-7b4a-4c58-9e21-0d6f8a1b2c3d",
      "state": "uploading",
      "dataset_id": null,
      "total_bytes": 73400320,
      "expires_at": "2026-09-24T18:02:11.000Z",
      "file": {
        "name": "tickets.jsonl",
        "size_bytes": 73400320,
        "part_size_bytes": 33554432,
        "part_count": 3,
        "state": "pending",
        "uploaded_bytes": 0,
        "parts": []
      }
    }
    ```
  </Step>

  <Step title="Request upload URLs">
    The script requests a signed URL for every part number from 1 through `file.part_count`. The response lists each `part_number` with its `url`, plus `expires_in_seconds`. Each URL is valid for 15 minutes. If the URLs expire before you use them, request them again.
  </Step>

  <Step title="Upload the parts">
    The script reads the file in `file.part_size_bytes` chunks and sends a `PUT` request with each chunk to the URL that matches its part number. Part numbers start at 1, so the first chunk is part 1. Every part except the last must be exactly `file.part_size_bytes` long, and the last part holds the remainder. You can send the parts in any order and in parallel.

    The signed URLs carry their own authorization, so the script doesn't add the `Authorization` or `Wandb-Entity` headers to these requests.
  </Step>

  <Step title="Optional: Check which parts arrived">
    To resume after an interrupted transfer, read the session. `file.parts` lists each uploaded part with its size, and `file.uploaded_bytes` totals them. Request new URLs for any missing parts and send them again. The Python and JavaScript scripts compute the missing part numbers, and the `curl` script prints the list of uploaded parts. None of the scripts resends missing parts, because a single uninterrupted run doesn't produce any.
  </Step>

  <Step title="Complete the upload">
    When every part is present, the script completes the upload. Model Distillation verifies that the parts match the declared size, computes a digest, and queues validation and dataset creation.

    The response is `202 Accepted`. If a part is missing, the response is `409 Conflict` with type `upload_incomplete`. If a part has the wrong size, the type is `upload_manifest_mismatch`. In both cases, the session stays in `uploading`, so you can fix the parts and complete the upload again. If you complete an upload that's already complete, the request returns the current session without error.
  </Step>

  <Step title="Poll until the dataset is ready">
    The script reads the session every 10 seconds until `state` is `ready` or `failed`. While validation runs, `validation.validated_rows` counts parsed rows, and `counters` reports `staged_rows`, `rejected_rows`, and `rows_by_split`.

    When `state` is `ready`, `dataset_id` identifies the new dataset. Use it with relabeling, fine-tuning, and evaluations, or open it in the UI. If `state` is `failed`, see [Read validation results](#read-validation-results).
  </Step>
</Steps>

## Splits and duplicates

`split_policy` determines how rows are divided between training and validation:

* **`preserve`** keeps the `split` value on each row. A row without `split` fails validation with `missing_split`.
* **`automatic`** ignores any `split` value on the rows. It hashes each row's `group_id`, or its row identity when `group_id` is absent, and assigns the fraction in `val_fraction` to the validation split. The default fraction is 0.2. Rows that share a `group_id` are always assigned to the same split.

`duplicate_policy.split_overlap` determines what happens when an identical input appears in both splits after assignment:

* **`reject`**, the default, records a validation error for each overlapping row, and the import fails.
* **`drop_train`** drops the training copies and keeps the validation copies.

A `row_id` that appears more than once in the file fails validation with `duplicate_row_id` under either policy.

## Read validation results

A `failed` session reports `error` as a short summary, and `validation` carries the details:

* `error_count` is the total number of problems found.
* `errors` lists up to 100 problems, ordered by line, each with `physical_line`, `code`, and `message`.
* `errors_truncated` is `true` when more problems exist than the list shows.

Common codes are `invalid_json`, `invalid_utf8`, `duplicate_json_key`, `row_too_large`, `missing_assistant_target`, `tools_too_large`, `missing_split`, and `duplicate_row_id`. Fix the file and create a new session. A failed session can't be reopened.

## Cancel an upload

To discard a session, delete it. Cancellation works in every state except `ready`, so you can also abandon a session that is still validating or that failed. The response is `204 No Content`, the state becomes `cancelled`, uploaded parts are discarded, and any partially created dataset is removed. Repeating the request on a cancelled session returns `204 No Content` again.

```bash theme={"system"}
curl --request DELETE \
  --url "https://distillation.training.wandb.ai/v1/tasks/ticket-classifier/dataset-imports/$IMPORT_ID" \
  --header "Authorization: Bearer $WANDB_API_KEY" \
  --header "Wandb-Entity: your-team"
```

A session that already reached `ready` returns `409 Conflict` with type `dataset_import_ready`. To remove the dataset it created, delete the dataset instead. See [Delete a dataset](/model-distillation/studio/datasets#delete-a-dataset).

## Errors

Import session requests can return the following errors.

| Status                     | Type                                          | Cause                                                                                       |
| -------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `403 Forbidden`            | `dataset_uploads_disabled`                    | Uploads aren't enabled for the entity.                                                      |
| `409 Conflict`             | `idempotency_conflict`                        | The `idempotency_key` was already used with a different request body.                       |
| `409 Conflict`             | `dataset_import_not_uploading`                | The session has left `uploading`, so it no longer accepts new parts.                        |
| `409 Conflict`             | `upload_incomplete`                           | You tried to complete the upload before every part arrived.                                 |
| `409 Conflict`             | `upload_manifest_mismatch`                    | An uploaded part, or the assembled file, doesn't match the declared size.                   |
| `409 Conflict`             | `dataset_import_ready`                        | You tried to cancel a session that already created its dataset.                             |
| `409 Conflict`             | `dataset_import_state_changed`                | The session changed state during your request. Read it again and retry.                     |
| `410 Gone`                 | `dataset_import_expired`                      | The session is older than 7 days. Create a new one.                                         |
| `422 Unprocessable Entity` | `validation_error`                            | A part number is greater than `file.part_count`.                                            |
| `429 Too Many Requests`    | `dataset_import_concurrency_limit`            | The entity already has 2 sessions in progress.                                              |
| `429 Too Many Requests`    | `dataset_import_byte_quota`                   | The new file would push the entity past 2 GiB of declared size across sessions in progress. |
| `502 Bad Gateway`          | `object_storage_error`                        | Upload storage rejected the request. Retry later.                                           |
| `503 Service Unavailable`  | `uploads_unavailable`, `temporal_unavailable` | Upload storage or the validation queue isn't reachable. Retry later.                        |

For request and response schemas, see the following pages in the Management API reference:

* [Create a dataset import session](/model-distillation/reference/management/datasets/create-a-dataset-import-session)
* [Create upload URLs for file parts](/model-distillation/reference/management/datasets/create-upload-urls-for-file-parts)
* [Get a dataset import session](/model-distillation/reference/management/datasets/get-a-dataset-import-session)
* [Complete and validate a dataset import](/model-distillation/reference/management/datasets/complete-and-validate-a-dataset-import)
* [Cancel a dataset import](/model-distillation/reference/management/datasets/cancel-a-dataset-import)

## Next steps

<CardGroup cols={2}>
  <Card title="Relabeling" icon="pen-to-square" href="/model-distillation/studio/relabeling">
    Ask a stronger model to rewrite the assistant responses in the uploaded dataset without changing the original rows.
  </Card>

  <Card title="Fine-tuning" icon="graduation-cap" href="/model-distillation/studio/fine-tuning">
    Train a supported base model on the uploaded dataset, using the original outputs or a relabeled output set.
  </Card>
</CardGroup>


## Related topics

- [Datasets](/model-distillation/studio/datasets.md)