Skip to main content
A multipart upload (MPU) uploads one object as a set of independent parts, which CoreWeave AI Object Storage assembles into a single object when the upload completes. Use multipart uploads for large objects:
  • Parts upload in parallel, which increases throughput.
  • You can retry a failed part without restarting the whole upload.
  • AI Object Storage preserves the parts, and LOTA caches each part independently, so it can spread a multipart object across the cache of several Nodes and serve the parts in parallel. LOTA caches an object uploaded with a single PutObject call as one unit on one Node, which can create a read bottleneck for large objects. For more information, see Use multipart uploads for large objects.
Multipart uploads are subject to the part size and part count limits. Each part can be at most 5 GiB. All parts except the last must be at least 5 MiB. An upload can contain up to 10,000 parts. We recommend a part size of at least 50 MiB. The following sections describe two ways to run a multipart upload: with a high-level client that manages the parts for you, or with the low-level API for direct control over each part.

Prerequisites

Before you start, make sure you have:
  • An Access Key for the organization that owns the bucket.
  • An S3-compatible client configured for AI Object Storage with virtual-hosted addressing.
  • Permission to call s3:PutObject on the bucket. The Rclone example also requires s3:GetObject to verify the uploaded object with HeadObject. Aborting an upload also requires s3:AbortMultipartUpload, and listing uploads or parts requires s3:ListBucketMultipartUploads or s3:ListMultipartUploadParts, respectively.

Upload with a high-level client

High-level clients switch to a multipart upload automatically when a file exceeds a size threshold, then split, upload, and complete the parts for you. For better performance, raise the default part size to 50 MiB.
The AWS CLI uses a multipart upload for files larger than multipart_threshold and splits them into parts of multipart_chunksize. Both default to 8 MiB.
  1. In your cw profile, set both to 50 MiB:
    Set the multipart threshold and part size with AWS CLI
  2. Upload the file. Replace LOCAL_FILE_PATH, BUCKET_NAME, and OBJECT_KEY with your values.
    Upload a large file with AWS CLI
When the upload finishes, the file appears in the bucket as a single object. To tune concurrency for bulk transfers, see Optimize s5cmd performance and Optimize Rclone performance.

Upload parts with the low-level API

When you need direct control over the upload, for example to upload parts from different processes or machines, use the low-level API. A low-level multipart upload has three steps:
  1. CreateMultipartUpload starts the upload and returns an UploadId.
  2. UploadPart uploads each part with a part number from 1 to 10,000 and returns an ETag for that part.
  3. CompleteMultipartUpload assembles the object from a list of the part numbers and ETag values, sorted by part number in ascending order.
The following Boto3 script uploads a local file in 50 MiB parts, one part at a time. To upload parts in parallel, call upload_part() from multiple threads or processes with the same UploadId. If any step fails, the script aborts the upload so the parts don’t remain in the bucket. With 50 MiB parts, an upload can hold up to about 488 GiB. For larger files, increase PART_SIZE.
  1. Set environment variables for your CoreWeave credentials:
  2. Replace LOCAL_FILE_PATH, BUCKET_NAME, and OBJECT_KEY with your values, then run the script:
    Multipart upload with the low-level Boto3 API
The object appears in the bucket only after CompleteMultipartUpload succeeds.

Clean up incomplete uploads

A client can stop before it calls CompleteMultipartUpload or AbortMultipartUpload, for example because the process crashes or loses its connection. The uploaded parts then remain in the bucket as an incomplete multipart upload. Incomplete uploads consume storage, don’t appear in normal object listings, and prevent the bucket from being deleted: DeleteBucket returns BucketNotEmpty. You can remove incomplete uploads in the following ways:
Last modified on November 20, 2026