- Parts upload in parallel, which increases throughput.
- You can retry a failed part without restarting the whole upload.
- AI Object Storage preserves the parts, and LOTA caches each part independently, so it can spread a multipart object across the cache of several Nodes and serve the parts in parallel. LOTA caches an object uploaded with a single
PutObjectcall as one unit on one Node, which can create a read bottleneck for large objects. For more information, see Use multipart uploads for large objects.
Prerequisites
Before you start, make sure you have:- An Access Key for the organization that owns the bucket.
- An S3-compatible client configured for AI Object Storage with virtual-hosted addressing.
- Permission to call
s3:PutObjecton the bucket. The Rclone example also requiress3:GetObjectto verify the uploaded object withHeadObject. Aborting an upload also requiress3:AbortMultipartUpload, and listing uploads or parts requiress3:ListBucketMultipartUploadsors3:ListMultipartUploadParts, respectively.
Upload with a high-level client
High-level clients switch to a multipart upload automatically when a file exceeds a size threshold, then split, upload, and complete the parts for you. For better performance, raise the default part size to 50 MiB.- AWS CLI
- Boto3
- s5cmd
- Rclone
The AWS CLI uses a multipart upload for files larger than
multipart_threshold and splits them into parts of multipart_chunksize. Both default to 8 MiB.-
In your
cwprofile, set both to 50 MiB:Set the multipart threshold and part size with AWS CLI -
Upload the file. Replace
LOCAL_FILE_PATH,BUCKET_NAME, andOBJECT_KEYwith your values.Upload a large file with AWS CLI
Upload parts with the low-level API
When you need direct control over the upload, for example to upload parts from different processes or machines, use the low-level API. A low-level multipart upload has three steps:CreateMultipartUploadstarts the upload and returns anUploadId.UploadPartuploads each part with a part number from 1 to 10,000 and returns anETagfor that part.CompleteMultipartUploadassembles the object from a list of the part numbers andETagvalues, sorted by part number in ascending order.
upload_part() from multiple threads or processes with the same UploadId. If any step fails, the script aborts the upload so the parts don’t remain in the bucket. With 50 MiB parts, an upload can hold up to about 488 GiB. For larger files, increase PART_SIZE.
-
Set environment variables for your CoreWeave credentials:
-
Replace
LOCAL_FILE_PATH,BUCKET_NAME, andOBJECT_KEYwith your values, then run the script:Multipart upload with the low-level Boto3 API
CompleteMultipartUpload succeeds.
Clean up incomplete uploads
A client can stop before it callsCompleteMultipartUpload or AbortMultipartUpload, for example because the process crashes or loses its connection. The uploaded parts then remain in the bucket as an incomplete multipart upload. Incomplete uploads consume storage, don’t appear in normal object listings, and prevent the bucket from being deleted: DeleteBucket returns BucketNotEmpty.
You can remove incomplete uploads in the following ways:
- To find and remove existing incomplete uploads, see Abort incomplete multipart uploads.
- To remove them automatically, add an
AbortIncompleteMultipartUploadlifecycle rule to every bucket that receives large uploads.