Migrate data to AI Object Storage

This guide explains how to copy data between AI Object Storage buckets using s5cmd and Rclone. Use it to move datasets into CoreWeave AI Object Storage so that your workloads can read and write to a high-performance, S3-compatible store. You can also use s3cmd or Cyberduck to copy data to CoreWeave AI Object Storage. You can also use the same approaches to transfer data from any S3-compatible storage service, such as AWS S3, Azure Blob Storage, or Google Cloud Storage.

Choose a migration tool

Tool	Best for	Notes
s5cmd	High-performance parallel transfers, and transfers involving Local Storage	Recommended for most use cases. High default concurrency (256 workers).
Rclone	Complex sync operations, and transfers between diverse cloud providers	Do not use with Local Storage because of known kernel panic issues. Safe for AI Object Storage, Distributed File Storage, and external systems.

Prerequisites

Before you start, ensure you have the following:

The primary endpoint for AI Object Storage, https://cwobject.com, requires TLS v1.3. Ensure your S3-compatible tools and OpenSSL support TLS v1.3.
An organization access policy configured.
An access key and secret key for each bucket. For CoreWeave AI Object Storage, use either of these methods:
- Create keys with Workload Identity Federation.
- Create keys with a Cloud Console token.
If a bucket is on a different S3-compatible platform, such as AWS S3, Azure Blob Storage, or Google Cloud Storage, refer to the respective documentation to generate access keys.

Migrate data with s5cmd

s5cmd is a high-performance, parallel S3 and local filesystem execution tool. CoreWeave maintains a fork of s5cmd that defaults to virtual-style addressing for cwobject.com and cwlota.com, which is required for AI Object Storage.

Use the CoreWeave fork of s5cmdThe upstream s5cmd defaults to path-style addressing, which AI Object Storage doesn’t support. You must use the CoreWeave fork for compatibility with AI Object Storage.

Install s5cmd

Download the latest release binary for your platform from the CoreWeave s5cmd releases page. The CoreWeave fork is based on the latest upstream s5cmd and the only difference is that it defaults to virtual-style addressing for AI Object Storage. Other S3-compatible backends are unaffected, so this fork can safely replace any existing s5cmd installation.
After downloading, make the binary executable and move it to a directory in your PATH:
```
chmod +x s5cmd && sudo mv s5cmd /usr/local/bin/
```
Verify the installation:
```
s5cmd version
```

Configure credentials

s5cmd reads credentials from environment variables or the AWS shared credentials file. Set the following environment variables with your AI Object Storage access keys: Replace [ACCESS-KEY-ID] and [SECRET-ACCESS-KEY] with your AI Object Storage access key credentials.

export AWS_ACCESS_KEY_ID=[ACCESS-KEY-ID]
export AWS_SECRET_ACCESS_KEY=[SECRET-ACCESS-KEY]

If you’ve already configured credentials for AI Object Storage following the Get started guide, you can use your existing credentials file instead:

export AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials

Copy objects between buckets

Copy objects from a source bucket to a target bucket: Replace [SOURCE-BUCKET] and [TARGET-BUCKET] with the names of your source and target buckets.

s5cmd --endpoint-url https://cwobject.com \
      cp 's3://[SOURCE-BUCKET]/*' s3://[TARGET-BUCKET]/

This copies all objects from the source bucket to the target bucket. s5cmd preserves the source directory structure by default. To flatten the source directory structure, use the --flatten flag.

s5cmd --endpoint-url https://cwobject.com \
      cp --flatten 's3://[SOURCE-BUCKET]/*' s3://[TARGET-BUCKET]/

Escape wildcards in zshShells like zsh treat the * character as a file globbing wildcard. Wrap wildcard expressions in single quotes to prevent unexpected behavior:

# Correct - wildcards are passed to s5cmd
s5cmd cp 's3://bucket/*.gz' ./local-dir/

# Incorrect - shell expands the wildcard first
s5cmd cp s3://bucket/*.gz ./local-dir/

To copy a specific file:

s5cmd --endpoint-url https://cwobject.com \
      cp s3://[SOURCE-BUCKET]/path/to/file.txt s3://[TARGET-BUCKET]/path/to/file.txt

To copy from a local directory to a bucket:

s5cmd --endpoint-url https://cwobject.com \
      cp '/local/path/*' s3://[TARGET-BUCKET]/

Optimize s5cmd performance

s5cmd provides two main options for tuning parallelism:

--numworkers: Sets the size of the global worker pool (default: 256). This flag controls how many files transfer concurrently.
--concurrency: A cp command option that sets the number of parts uploaded or downloaded in parallel for a single file (default: 5). This option is useful for large files that use multipart transfers.

For many small files, increase --numworkers to maximize parallel file transfers:

s5cmd --endpoint-url https://cwobject.com \
      --numworkers 512 \
      cp 's3://[SOURCE-BUCKET]/*' s3://[TARGET-BUCKET]/

For a few large files, keep --numworkers low and increase --concurrency to maximize multipart upload parallelism:

s5cmd --endpoint-url https://cwobject.com \
      --numworkers 4 \
      cp --concurrency 16 's3://[SOURCE-BUCKET]/*' s3://[TARGET-BUCKET]/

For mixed workloads, balance both options:

s5cmd --endpoint-url https://cwobject.com \
      --numworkers 64 \
      cp --concurrency 8 's3://[SOURCE-BUCKET]/*' s3://[TARGET-BUCKET]/

Additional s5cmd operations

s5cmd supports many S3 operations beyond copy. Useful commands include:

# List objects in a bucket
s5cmd --endpoint-url https://cwobject.com ls s3://[BUCKET-NAME]/

# Remove objects
s5cmd --endpoint-url https://cwobject.com rm 's3://[BUCKET-NAME]/[PREFIX]/*'

# Sync directories (copy only changed files)
s5cmd --endpoint-url https://cwobject.com sync 's3://[SOURCE-BUCKET]/*' s3://[TARGET-BUCKET]/

For the full list of commands, run s5cmd --help.

Migrate data with Rclone

If s5cmd doesn’t fit your workflow, or you need to sync across multiple cloud providers, use Rclone instead. Rclone manages files across cloud storage providers. It supports complex sync operations and works with many storage backends.

Do not use Rclone with Local StorageRclone has known kernel panic issues when reading from and writing to Local Storage volumes. To move data between Local Storage and AI Object Storage, use s5cmd, s3cmd, or aws s3 cp instead.Rclone is safe to use with AI Object Storage and other systems, including Distributed File Storage, other Object Storage buckets, and external systems.

Install Rclone

Follow the Rclone installation guide to install Rclone. This guide uses Rclone version v1.69.

Configure Rclone

Create source and destination profiles in your Rclone config. To locate the config file, run:
```
rclone config file
```
If the config file doesn’t exist, Rclone reports the default location where you should create the config file:
```
Configuration file doesn't exist, but rclone will use this path:
/home/username/.config/rclone/rclone.conf
```
If an active config file exists, Rclone reports its actual location with a similar message.

Edit the config file with your preferred text editor and add the following profiles. Replace [ACCESS-KEY-ID] and [SECRET-ACCESS-KEY] with your AI Object Storage access key credentials:

~/.config/rclone/rclone.conf

[source]
type = s3
provider = Other
access_key_id = [ACCESS-KEY-ID]
secret_access_key = [SECRET-ACCESS-KEY]
endpoint = https://cwobject.com
force_path_style = false
no_check_bucket = true

[target]
type = s3
provider = Other
access_key_id = [ACCESS-KEY-ID]
secret_access_key = [SECRET-ACCESS-KEY]
endpoint = https://cwobject.com
force_path_style = false
no_check_bucket = true

Save and close the file. You now have two named Rclone profiles, source and target, that you can reference in subsequent commands to copy data between buckets.

Copy objects between buckets

Copy objects from the source to the target: Replace [SOURCE-BUCKET] and [TARGET-BUCKET] with the names of your source and target buckets.

rclone copy source:[SOURCE-BUCKET] target:[TARGET-BUCKET] \
      --progress --stats 15s

This copies all objects:

From the source bucket on the source profile.
To the target bucket on the target profile.

The Rclone options --progress --stats 15s print a progress bar with estimated time to completion and detailed transfer statistics every 15 seconds. For versioned buckets, Rclone copies the latest version of each object by default. See Use Rclone with versioned buckets and objects for more details, including specific guidance on how to work with specific versions and delete markers.

Optimize Rclone performance

To optimize Rclone throughput with AI Object Storage, use flags to fine-tune parallelism and chunking for large files. The main flags are:

--transfers: Sets the number of concurrent file transfers. Adjust this flag to use the available network bandwidth between your environment and CoreWeave, or between CoreWeave regions.
--checkers: Controls the number of concurrent file checks for equality. Adjust this flag when transferring many small files.
--s3-chunk-size: Defines the chunk size used to upload files larger than the upload_cutoff or files with unknown sizes. Larger chunks reduce HTTP requests but increase memory usage.
--s3-upload-concurrency: Sets the level of concurrency for multipart uploads.

To check the default values for each flag, use rclone help flags. Rclone has many flags, so use grep to filter the output.

rclone help flags | grep -E -- '--checkers|--transfers|--s3-chunk-size|--s3-upload-concurrency'

Example output

  --checkers int                Number of checkers to run in parallel (default 8)
  --transfers int               Number of file transfers to run in parallel (default 4)
  --s3-chunk-size SizeSuffix    Chunk size to use for uploading (default 5Mi)
  --s3-upload-concurrency int   Concurrency for multipart uploads and copies (default 4)

Optimization guidelines

Use these guidelines to optimize Rclone throughput when copying data to and from CoreWeave AI Object Storage:

To maximize migration throughput, consider the combined effects of --transfers and --s3-upload-concurrency as multiplicative: Total streams ≈ --transfers × --s3-upload-concurrency
- For transfers dominated by many small or medium-sized files (KBs to MBs), increase --transfers to a value between 8 and 32 to move multiple files in parallel, but keep --s3-upload-concurrency at a lower value, between 1 and 4, because small files don’t benefit from multipart uploads.
- For transfers with a few large files (hundreds of GB), do the opposite. Set --transfers to 1 or 2 to avoid initiating too many multipart uploads, and increase --s3-upload-concurrency to a large value, between 8 and 16, to upload multiple parts of each large file in parallel and saturate available bandwidth.
Increase --s3-chunk-size to 50 MB for best performance. The default is 5 MB.
Monitor memory use. Estimate Rclone’s RAM needs with this calculation: RAM ≈ --transfers × (--s3-upload-concurrency × (--s3-chunk-size + --buffer-size))
- Each active stream uses --buffer-size (the default 16 MB).
- Each multipart chunk consumes --s3-chunk-size.
Increase one flag at a time while monitoring with rclone --progress --stats 15s.
Stop tuning when throughput plateaus or retries increase. This method ensures that you maximize the available bandwidth without overloading local I/O, system memory, or remote service limits.

Usage example

The following command copies data from the source bucket to the target bucket, using the recommended flags. Use --s3-chunk-size 50M for best performance, and adjust the other flags based on your data size, number of files, and available bandwidth.

rclone copy source:[SOURCE-BUCKET] target:[TARGET-BUCKET] \
      --progress \
      --stats 15s \
      --transfers 64 \
      --checkers 128 \
      --s3-chunk-size 50M \
      --s3-upload-concurrency 10

Check bucket usage

If you have s3cmd installed, check the usage of the target bucket with:

s3cmd du --human-readable-sizes s3://[BUCKET-NAME]

The output shows the total size of the bucket and the number of objects it contains.

Example output

238M    2303 objects s3://my-bucket/

​Choose a migration tool

​Prerequisites

​Migrate data with s5cmd

​Install s5cmd

​Configure credentials

​Copy objects between buckets

​Optimize s5cmd performance

​Additional s5cmd operations

​Migrate data with Rclone

​Install Rclone

​Configure Rclone

​Copy objects between buckets

​Optimize Rclone performance

​Optimization guidelines

​Usage example

​Check bucket usage

Choose a migration tool

Prerequisites

Migrate data with s5cmd

Install s5cmd

Configure credentials

Copy objects between buckets

Optimize s5cmd performance

Additional s5cmd operations

Migrate data with Rclone

Install Rclone

Configure Rclone

Copy objects between buckets

Optimize Rclone performance

Optimization guidelines

Usage example

Check bucket usage