Manage objects - CoreWeave Docs

This guide explains how to manage objects stored in CoreWeave AI Object Storage buckets using S3-compatible tools, including the AWS CLI, s3cmd, Boto3, and s5cmd. For high-performance or bulk transfers, see Migrate data to AI Object Storage for s5cmd (use the CoreWeave fork for AI Object Storage). Alternatively, you can use Cyberduck to manage your buckets and objects in a graphical interface. The Cloud Console doesn’t support managing objects. To manage versioned buckets with rclone, see Versioned buckets.

Prerequisites

This guide presumes you have the following:

An active CoreWeave account.
A valid API Access Key and Secret Token key pair.
Adequate permissions to manage objects in CoreWeave AI Object Storage (for example, s3:PutObject and s3:DeleteObject). For more information, see Object Storage S3 Permissions.
An organization access policy set.
A properly configured SDK of your choosing.
The primary endpoint, https://cwobject.com, requires TLS v1.3. Ensure your S3-compatible tools and OpenSSL use TLS v1.3.
Make sure your S3 configuration uses the correct endpoint URL and has virtual-hosted addressing enabled.

Configure your endpoint

Before running the commands in this guide, make sure your S3 client is configured with:

Virtual-hosted addressing enabled (s3.addressing_style = virtual). Path-style addressing is not supported. See Set virtual addressing style.
The correct endpoint URL:
- http://cwlota.com when running inside a CoreWeave cluster (LOTA, best performance).
- https://cwobject.com when running outside a CoreWeave cluster.

Add objects

Upload objects to a bucket so your data is available for downstream applications, sharing, or processing. Choose the tab that matches the tool you use.

AWS CLI
s3cmd
Boto3

Ensure you have the AWS CLI installed and configured.Use the s3 cp command to copy a file into a bucket addressed using the s3:// scheme.

Example command

aws s3 cp [LOCAL-FILE-PATH] s3://[BUCKET-NAME]

Example output

upload: ./my-important-file.txt to s3://my-bucket-name/my-important-file.txt

Ensure you have s3cmd installed, configured, and correctly credentialed.Use the put command to copy an object into the bucket.

Example command

s3cmd put [LOCAL-FILE-PATH] s3://[BUCKET-NAME]

This returns the following output, confirming your object is stored in your bucket:

Example output

upload: 'my-important-file.txt' -> 's3://my-bucket-name/my-important-file.txt'

Set environment variables for your CoreWeave credentials:
```
export ACCESS_KEY_ID="[ACCESS-KEY-ID]"
export SECRET_ACCESS_KEY="[SECRET-ACCESS-KEY]"
```
Alternatively, configure your CoreWeave credentials to work with the AWS CLI. We recommend using a separate profile for CoreWeave AI Object Storage to avoid conflicts with your other AWS profiles and S3-compatible services. If you don’t set up this configuration, you might encounter errors when using AI Object Storage.
Configure CoreWeave credentials
1. Create a new credentials file and profile in your CoreWeave configuration directory.
  Create a new credentials file and profile
  AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
2. When prompted, provide the following values:
  - AWS Access Key ID: The Access Key ID of your CoreWeave AI Object Storage Access Key.
  - AWS Secret Access Key: The Secret Key of your CoreWeave AI Object Storage Access Key.
  - Default region name (Optional): To set a default region, see CoreWeave Availability Zones.
  - Default output format: Use json for JSON output.
3. Set the default endpoint URL to the appropriate endpoint for your use case:
  - The primary endpoint, https://cwobject.com, for use outside a CoreWeave cluster.
  - The LOTA endpoint, http://cwlota.com, for use inside a CoreWeave cluster. The LOTA endpoint routes to the LOTA path for best performance.
  Set the primary endpoint for local development
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
4. Set the S3 addressing_style to virtual:
  Set virtual addressing style
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set s3.addressing_style virtual --profile cw
If you set endpoint_url and s3.addressing_style directly in your code (for example, in a Boto3 Config object), you can skip steps 3 and 4. The profile only needs the access key, secret key, and region.

Use the following Python script to upload a file to your S3 bucket. Replace the following placeholders with the appropriate values for your request:

[BUCKET-NAME]: The name of the bucket to upload the file to.
[LOCAL-FILE-PATH]: The path to the local file to upload.
[OBJECT-NAME]: The name of the object to upload. This is optional. If you don’t set a value, the upload uses the same string as your [LOCAL-FILE-PATH].

Upload a file to a bucket

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

# Create the S3 client
s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

# Define your bucket name and the file path
bucket_name = '[BUCKET-NAME]'
file_name = '[LOCAL-FILE-PATH]'
object_name = '[OBJECT-NAME]'  # This is optional. If you don't set a value, the upload uses the same string as your file_name.

# Upload file
s3.upload_file(file_name, bucket_name, object_name)

print(f'File {file_name} uploaded to {bucket_name}/{object_name}')

If this succeeds, a confirmation message is printed.

Example output

File /path/to/my-file uploaded to my-bucket-name/my-important-file

Verify object integrity

CoreWeave AI Object Storage supports checksum algorithms for verifying object integrity on upload and download. For the full list of supported algorithms, see Checksum algorithms.

Quota limit errors

If you try to upload an object to a bucket in an Availability Zone where capacity quota limits have been reached, you receive an error message: <Message>The account is write suspended.</Message> To resolve this, you can request a quota increase.

List buckets and their contents

Listing buckets and their contents helps you confirm which buckets exist in your account and verify which objects are stored in a given bucket. You can list buckets and their contents using S3-compatible tools such as the AWS CLI, s3cmd, or Boto3. If you’re working with versioned buckets, you can use rclone to list buckets and their contents, including delete markers.

AWS CLI
s3cmd
Boto3

If you want to see all of your available buckets, use the ls command:

aws s3 ls

To list all the objects in a bucket, use the ls command to target a bucket path.

aws s3 ls s3://[BUCKET-NAME]

The terminal returns a listing for your selected bucket showing each object, its size, and its last modified date.

Example file listing output

2024-10-14 15:35:10 123456 my-first-file.txt
2024-10-14 16:45:22 234567 another-file-of-mine.txt

To list all your available buckets, use the ls command:

s3cmd ls

To see the contents of a specific bucket, target the bucket’s path using the s3:// scheme.

Example command

s3cmd ls s3://[BUCKET-NAME]/

This command outputs a list of file paths including the bucket name and file name. For example:

Example output

2024-10-14 15:35 123456 s3://my-bucket-name/my-first-file.txt
2024-10-14 16:45 234567 s3://my-bucket-name/another-file-of-mine.txt

Set environment variables for your CoreWeave credentials:
```
export ACCESS_KEY_ID="[ACCESS-KEY-ID]"
export SECRET_ACCESS_KEY="[SECRET-ACCESS-KEY]"
```
Alternatively, configure your CoreWeave credentials to work with the AWS CLI. We recommend using a separate profile for CoreWeave AI Object Storage to avoid conflicts with your other AWS profiles and S3-compatible services. If you don’t set up this configuration, you might encounter errors when using AI Object Storage.
Configure CoreWeave credentials
1. Create a new credentials file and profile in your CoreWeave configuration directory.
  Create a new credentials file and profile
  AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
2. When prompted, provide the following values:
  - AWS Access Key ID: The Access Key ID of your CoreWeave AI Object Storage Access Key.
  - AWS Secret Access Key: The Secret Key of your CoreWeave AI Object Storage Access Key.
  - Default region name (Optional): To set a default region, see CoreWeave Availability Zones.
  - Default output format: Use json for JSON output.
3. Set the default endpoint URL to the appropriate endpoint for your use case:
  - The primary endpoint, https://cwobject.com, for use outside a CoreWeave cluster.
  - The LOTA endpoint, http://cwlota.com, for use inside a CoreWeave cluster. The LOTA endpoint routes to the LOTA path for best performance.
  Set the primary endpoint for local development
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
4. Set the S3 addressing_style to virtual:
  Set virtual addressing style
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set s3.addressing_style virtual --profile cw
If you set endpoint_url and s3.addressing_style directly in your code (for example, in a Boto3 Config object), you can skip steps 3 and 4. The profile only needs the access key, secret key, and region.

List all your buckets programmatically using the following script:

List all buckets

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

response = s3.list_buckets()

print("Existing buckets:")
for bucket in response['Buckets']:
    print(f'  {bucket["Name"]}')

To view the contents of a specific bucket, replace [BUCKET-NAME] with the name of your bucket:

List objects in a bucket

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

bucket_name = '[BUCKET-NAME]'

response = s3.list_objects_v2(Bucket=bucket_name)

if 'Contents' in response:
    print(f"Objects in {bucket_name}:")
    for obj in response['Contents']:
        print(f'  {obj["Key"]} (Size: {obj["Size"]} bytes)')
else:
    print(f"No objects found in {bucket_name}.")

The resulting output looks similar to the following:

Example output

Objects in my-bucket-name:
my-first-file.txt (Size: 4500 bytes)
another-file-of-mine.txt (Size: 5400 bytes)

Delete an object from a bucket

Delete objects you no longer need to free up capacity and keep your buckets organized. Deletions are permanent on buckets that don’t have versioning enabled, so confirm the target object before you run these commands.

AWS CLI
s3cmd
Boto3

To delete specific objects from a bucket, use the rm command with the AWS CLI.

aws s3 rm s3://[BUCKET-NAME]/[OBJECT-NAME]

When this succeeds, a confirmation message like this one is printed:

Example output

delete: s3://my-bucket-name/my-important-file.txt

To delete an object from a bucket, use the del command.

Example command

s3cmd del s3://[BUCKET-NAME]/[OBJECT-NAME]

If this succeeds, a confirmation message is printed:

Example output

Object s3://my-bucket-name/my-important-file.txt deleted

Set environment variables for your CoreWeave credentials:
```
export ACCESS_KEY_ID="[ACCESS-KEY-ID]"
export SECRET_ACCESS_KEY="[SECRET-ACCESS-KEY]"
```
Alternatively, configure your CoreWeave credentials to work with the AWS CLI. We recommend using a separate profile for CoreWeave AI Object Storage to avoid conflicts with your other AWS profiles and S3-compatible services. If you don’t set up this configuration, you might encounter errors when using AI Object Storage.
Configure CoreWeave credentials
1. Create a new credentials file and profile in your CoreWeave configuration directory.
  Create a new credentials file and profile
  AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
2. When prompted, provide the following values:
  - AWS Access Key ID: The Access Key ID of your CoreWeave AI Object Storage Access Key.
  - AWS Secret Access Key: The Secret Key of your CoreWeave AI Object Storage Access Key.
  - Default region name (Optional): To set a default region, see CoreWeave Availability Zones.
  - Default output format: Use json for JSON output.
3. Set the default endpoint URL to the appropriate endpoint for your use case:
  - The primary endpoint, https://cwobject.com, for use outside a CoreWeave cluster.
  - The LOTA endpoint, http://cwlota.com, for use inside a CoreWeave cluster. The LOTA endpoint routes to the LOTA path for best performance.
  Set the primary endpoint for local development
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
4. Set the S3 addressing_style to virtual:
  Set virtual addressing style
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set s3.addressing_style virtual --profile cw
If you set endpoint_url and s3.addressing_style directly in your code (for example, in a Boto3 Config object), you can skip steps 3 and 4. The profile only needs the access key, secret key, and region.

Use the following script to delete an object from a bucket. Replace [BUCKET-NAME] with the name of the bucket and [OBJECT-NAME] with the key of the object to delete.

Delete an object from a bucket

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

bucket_name = '[BUCKET-NAME]'
object_name = '[OBJECT-NAME]'

s3.delete_object(Bucket=bucket_name, Key=object_name)

print(f'Object {object_name} deleted from {bucket_name}')

If successful, a success message is printed:

Example output

Object my-important-file deleted from my-bucket-name

For more information about accessing and interacting with the contents of your buckets, see the official Amazon documentation for S3 buckets.

Rename an object

Rename an object to change its key without re-uploading the underlying data. This is useful when you need to correct a typo, restructure a key prefix, or align object names with a new naming convention. Make sure you have the following to rename an object:

s3:PutObject and s3:DeleteObject permissions.
A bucket that does not have versioning enabled, now or in the past.
A request scoped to renaming a single object within the same bucket. The source and destination keys must be in the same bucket.

For more information, see Rename Objects.

AWS CLI
s3cmd
Boto3

To rename an object, use the aws s3api rename-object command with the AWS CLI.Replace the following placeholders with the appropriate values for your request:

[BUCKET-NAME]: The name of the bucket containing the object.
[SOURCE-OBJECT-NAME]: The current name of the object.
[DESTINATION-OBJECT-NAME]: The new name for the object.

Rename object

aws s3api rename-object \
  --bucket [BUCKET-NAME] \
  --key [DESTINATION-OBJECT-NAME] \
  --rename-source /[BUCKET-NAME]/[SOURCE-OBJECT-NAME]

When this succeeds, a confirmation message is printed:

Example output

rename: s3://my-bucket-name/my-important-file.txt -> s3://my-bucket-name/my-new-file.txt

To rename an object, use the mv command with s3cmd.

Copy and delete operationThe s3cmd mv command performs a copy and delete operation instead of an atomic rename operation. For atomic rename operations, use the AWS CLI or Boto3 with the RenameObject API.

To prevent accidental overwrites when uploading objects, use the If-None-Match: * header with PutObject. For details, see Conditional writes.

Replace the following placeholders with the appropriate values for your request:

[BUCKET-NAME]: The name of the bucket containing the source object.
[SOURCE-OBJECT-NAME]: The name of the source object to rename.
[DESTINATION-OBJECT-NAME]: The new name for the object.

Rename object

s3cmd mv s3://[BUCKET-NAME]/[SOURCE-OBJECT-NAME] s3://[BUCKET-NAME]/[DESTINATION-OBJECT-NAME]

Set environment variables for your CoreWeave credentials:
```
export ACCESS_KEY_ID="[ACCESS-KEY-ID]"
export SECRET_ACCESS_KEY="[SECRET-ACCESS-KEY]"
```
Alternatively, configure your CoreWeave credentials to work with the AWS CLI. We recommend using a separate profile for CoreWeave AI Object Storage to avoid conflicts with your other AWS profiles and S3-compatible services. If you don’t set up this configuration, you might encounter errors when using AI Object Storage.
Configure CoreWeave credentials
1. Create a new credentials file and profile in your CoreWeave configuration directory.
  Create a new credentials file and profile
  AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
2. When prompted, provide the following values:
  - AWS Access Key ID: The Access Key ID of your CoreWeave AI Object Storage Access Key.
  - AWS Secret Access Key: The Secret Key of your CoreWeave AI Object Storage Access Key.
  - Default region name (Optional): To set a default region, see CoreWeave Availability Zones.
  - Default output format: Use json for JSON output.
3. Set the default endpoint URL to the appropriate endpoint for your use case:
  - The primary endpoint, https://cwobject.com, for use outside a CoreWeave cluster.
  - The LOTA endpoint, http://cwlota.com, for use inside a CoreWeave cluster. The LOTA endpoint routes to the LOTA path for best performance.
  Set the primary endpoint for local development
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
4. Set the S3 addressing_style to virtual:
  Set virtual addressing style
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set s3.addressing_style virtual --profile cw
If you set endpoint_url and s3.addressing_style directly in your code (for example, in a Boto3 Config object), you can skip steps 3 and 4. The profile only needs the access key, secret key, and region.

Use the rename_object method to rename an object. Replace [BUCKET-NAME], [SOURCE-OBJECT-NAME], and [DESTINATION-OBJECT-NAME] with the appropriate values for your request.

Rename an object

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

response = s3.rename_object(
    Bucket='[BUCKET-NAME]',
    Key='[DESTINATION-OBJECT-NAME]',
    RenameSource='/[BUCKET-NAME]/[SOURCE-OBJECT-NAME]'
)

print('Object renamed successfully.')

​Prerequisites

​Add objects

​Verify object integrity

​Quota limit errors

​List buckets and their contents

​Delete an object from a bucket

​Rename an object

Prerequisites

Add objects

Verify object integrity

Quota limit errors

List buckets and their contents

Delete an object from a bucket

Rename an object