Manage objects - CoreWeave Docs

This guide explains how to manage objects stored in CoreWeave AI Object Storage buckets using S3-compatible tools, including the AWS CLI, s3cmd, Boto3, and s5cmd. For high-performance or bulk transfers, see Migrate data to AI Object Storage for s5cmd (use the CoreWeave fork for AI Object Storage). Alternatively, you can use Cyberduck to manage your buckets and objects in a graphical interface. Currently, the Cloud Console does not support managing objects. To manage versioned buckets with rclone, see Versioned buckets.

Prerequisites

This guide presumes you have the following:

An active CoreWeave account.
A valid API Access Key and Secret Token key pair.
Adequate permissions to manage objects in CoreWeave AI Object Storage (for example, s3:PutObject and s3:DeleteObject). See Object Storage S3 Permissions for more information.
An organization access policy set.
A properly configured SDK of your choosing.
Note that the primary endpoint, https://cwobject.com, requires TLS v1.3. Ensure your S3-compatible tools and OpenSSL use TLS v1.3.
Make sure your S3 configuration uses the correct endpoint URL and has virtual-hosted addressing enabled.

Configure your endpoint

Before running the commands in this guide, make sure your S3 client is configured with:

Virtual-hosted addressing enabled (s3.addressing_style = virtual). Path-style addressing is not supported. See Set Endpoints.
The correct endpoint URL:
- http://cwlota.com when running inside a CoreWeave cluster (LOTA, best performance)
- https://cwobject.com when running outside a CoreWeave cluster

Add objects

AWS CLI
s3cmd
Boto3

Ensure you have the AWS CLI installed and configured.Use the s3 cp command to copy a file into a bucket addressed using the s3:// scheme.

Example command

aws s3 cp [LOCAL-FILE-PATH] s3://[BUCKET-NAME]

Example output

upload: ./my-important-file.txt to s3://my-bucket-name/my-important-file.txt

Ensure you have s3cmd installed, configured and correctly credentialed.Use the put command to copy an object into the bucket.

Example command

s3cmd put [LOCAL-FILE-PATH] s3://[BUCKET-NAME]

This returns the following output, confirming your object has been stored in your bucket:

Example output

upload: 'my-important-file.txt' -> 's3://my-bucket-name/my-important-file.txt'

Set environment variables for your CoreWeave credentials:
```
export ACCESS_KEY_ID="[ACCESS-KEY-ID]"
export SECRET_ACCESS_KEY="[SECRET-ACCESS-KEY]"
```
Alternatively, configure your CoreWeave credentials to work with the AWS CLI. Using a separate profile for CoreWeave AI Object Storage is recommended to avoid conflicts with your other AWS profiles and S3-compatible services; if you do not set up this configuration, you may encounter errors when using AI Object Storage.
Configure CoreWeave credentials
1. Create a new credentials file and profile in your CoreWeave configuration directory.
  Create a new credentials file and profile
  AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
2. When prompted for information, provide the following values:
  - AWS Access Key ID: The Access Key ID of your CoreWeave AI Object Storage Access Key.
  - AWS Secret Access Key: The Secret Key of your CoreWeave AI Object Storage Access Key.
  - Default region name: Optional. To set a default region, refer to the CoreWeave Availability Zones.
  - Default output format: Use json for JSON output.
3. Set the default endpoint URL to the appropriate endpoint for your use case:
  - The primary endpoint, https://cwobject.com, for use when running outside of a CoreWeave cluster.
  - The LOTA endpoint, http://cwlota.com, for use when running inside a CoreWeave cluster. The LOTA endpoint routes to the LOTA path for best performance.
  Set the primary endpoint for local development
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
4. Set the S3 addressing_style to virtual:
  Set virtual addressing style
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set s3.addressing_style virtual --profile cw

Use the following Python script to upload a file to your S3 bucket. Replace the following placeholders with the appropriate values for your request:

[BUCKET-NAME]: The name of the bucket to upload the file to.
[LOCAL-FILE-PATH]: The path to the local file to upload.
[OBJECT-NAME]: The name of the object to upload. This is optional. If you don’t set a value, it will use the same string as your [LOCAL-FILE-PATH].

Upload a file to a bucket

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

# Create the S3 client
s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

# Define your bucket name and the file path
bucket_name = '[BUCKET-NAME]'
file_name = '[LOCAL-FILE-PATH]'
object_name = '[OBJECT-NAME]'  # This is optional. If you don't set a value, it will use the same string as your file_name.

# Upload file
s3.upload_file(file_name, bucket_name, object_name)

print(f'File {file_name} uploaded to {bucket_name}/{object_name}')

If this succeeds, a confirmation message is printed.

Example output

File /path/to/my-file uploaded to my-bucket-name/my-important-file

Verify object integrity

CoreWeave AI Object Storage supports checksum algorithms for verifying object integrity on upload and download. See Checksum algorithms for the full list of supported algorithms.

Exceeding quota limits

If you try to upload an object to a bucket in an Availability Zone where capacity quota limits have been reached, you will receive an error message: <Message>The account is write suspended.</Message> To resolve this, you can request a quota increase.

List buckets and their contents

You can list buckets and their contents using S3-compatible tools such as the AWS CLI, s3cmd, or Boto3. If you’re working with versioned buckets, you can use rclone to list buckets and their contents, including delete markers.

AWS CLI
s3cmd
Boto3

If you want to see all of your available buckets, use the ls command:

aws s3 ls

To list all the objects currently in a bucket, use the ls command to target a bucket path.

aws s3 ls s3://[BUCKET-NAME]

The terminal or command prompt will return a YAML file for your selected bucket, listing all objects within it, their sizes, and their last modified dates.

Example file listing output

2024-10-14 15:35:10 123456 my-first-file.txt
2024-10-14 16:45:22 234567 another-file-of-mine.txt

To list all your available buckets, use the ls command:

s3cmd ls

To see the contents of a specific bucket, target the bucket’s path using the s3:// scheme.

Example command

s3cmd ls s3://[BUCKET-NAME]/

This command will output a list of file paths including the bucket name and file name. For example:

Example output

2024-10-14 15:35 123456 s3://my-bucket-name/my-first-file.txt
2024-10-14 16:45 234567 s3://my-bucket-name/another-file-of-mine.txt

Set environment variables for your CoreWeave credentials:
```
export ACCESS_KEY_ID="[ACCESS-KEY-ID]"
export SECRET_ACCESS_KEY="[SECRET-ACCESS-KEY]"
```
Alternatively, configure your CoreWeave credentials to work with the AWS CLI. Using a separate profile for CoreWeave AI Object Storage is recommended to avoid conflicts with your other AWS profiles and S3-compatible services; if you do not set up this configuration, you may encounter errors when using AI Object Storage.
Configure CoreWeave credentials
1. Create a new credentials file and profile in your CoreWeave configuration directory.
  Create a new credentials file and profile
  AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
2. When prompted for information, provide the following values:
  - AWS Access Key ID: The Access Key ID of your CoreWeave AI Object Storage Access Key.
  - AWS Secret Access Key: The Secret Key of your CoreWeave AI Object Storage Access Key.
  - Default region name: Optional. To set a default region, refer to the CoreWeave Availability Zones.
  - Default output format: Use json for JSON output.
3. Set the default endpoint URL to the appropriate endpoint for your use case:
  - The primary endpoint, https://cwobject.com, for use when running outside of a CoreWeave cluster.
  - The LOTA endpoint, http://cwlota.com, for use when running inside a CoreWeave cluster. The LOTA endpoint routes to the LOTA path for best performance.
  Set the primary endpoint for local development
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
4. Set the S3 addressing_style to virtual:
  Set virtual addressing style
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set s3.addressing_style virtual --profile cw

List all your buckets programmatically using the following script:

List all buckets

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

response = s3.list_buckets()

print("Existing buckets:")
for bucket in response['Buckets']:
    print(f'  {bucket["Name"]}')

To view the contents of a specific bucket, replace [BUCKET-NAME] with the name of your bucket:

List objects in a bucket

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

bucket_name = '[BUCKET-NAME]'

response = s3.list_objects_v2(Bucket=bucket_name)

if 'Contents' in response:
    print(f"Objects in {bucket_name}:")
    for obj in response['Contents']:
        print(f'  {obj["Key"]} (Size: {obj["Size"]} bytes)')
else:
    print(f"No objects found in {bucket_name}.")

The resulting output looks similar to the following:

Example output

Objects in my-bucket-name:
my-first-file.txt (Size: 4500 bytes)
another-file-of-mine.txt (Size: 5400 bytes)

Delete an object from a bucket

AWS CLI
s3cmd
Boto3

To delete specific objects from a bucket, use the rm command with the AWS CLI.

aws s3 rm s3://[BUCKET-NAME]/[OBJECT-NAME]

When this succeeds, a confirmation message like this one is printed:

Example output

delete: s3://my-bucket-name/my-important-file.txt

To delete an object from a bucket, use the del command to delete things from a bucket in Object Storage.

Example command

s3cmd del s3://[BUCKET-NAME]/[OBJECT-NAME]

If this succeeds, a confirmation message is printed:

Example output

Object s3://my-bucket-name/my-important-file.txt deleted

Set environment variables for your CoreWeave credentials:
```
export ACCESS_KEY_ID="[ACCESS-KEY-ID]"
export SECRET_ACCESS_KEY="[SECRET-ACCESS-KEY]"
```
Alternatively, configure your CoreWeave credentials to work with the AWS CLI. Using a separate profile for CoreWeave AI Object Storage is recommended to avoid conflicts with your other AWS profiles and S3-compatible services; if you do not set up this configuration, you may encounter errors when using AI Object Storage.
Configure CoreWeave credentials
1. Create a new credentials file and profile in your CoreWeave configuration directory.
  Create a new credentials file and profile
  AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
2. When prompted for information, provide the following values:
  - AWS Access Key ID: The Access Key ID of your CoreWeave AI Object Storage Access Key.
  - AWS Secret Access Key: The Secret Key of your CoreWeave AI Object Storage Access Key.
  - Default region name: Optional. To set a default region, refer to the CoreWeave Availability Zones.
  - Default output format: Use json for JSON output.
3. Set the default endpoint URL to the appropriate endpoint for your use case:
  - The primary endpoint, https://cwobject.com, for use when running outside of a CoreWeave cluster.
  - The LOTA endpoint, http://cwlota.com, for use when running inside a CoreWeave cluster. The LOTA endpoint routes to the LOTA path for best performance.
  Set the primary endpoint for local development
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
4. Set the S3 addressing_style to virtual:
  Set virtual addressing style
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set s3.addressing_style virtual --profile cw

Use the following script to delete an object from a bucket. Replace [BUCKET-NAME] with the name of the bucket and [OBJECT-NAME] with the key of the object to delete.

Delete an object from a bucket

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

bucket_name = '[BUCKET-NAME]'
object_name = '[OBJECT-NAME]'

s3.delete_object(Bucket=bucket_name, Key=object_name)

print(f'Object {object_name} deleted from {bucket_name}')

If successful, a success message is printed:

Example output

Object my-important-file deleted from my-bucket-name

For more information about accessing and interacting with the contents of your buckets, see the official Amazon documentation for s3 buckets.

Rename an object

Make sure you have the following to rename an object:

s3:PutObject and s3:DeleteObject permissions.
A bucket that does not have versioning enabled, either currently or in the past.
A request scoped to renaming a single object within the same bucket. The source and destination keys must be in the same bucket.

See Rename Objects for more information.

AWS CLI
s3cmd
Boto3

To rename an object, use the aws s3api rename-object command with the AWS CLI.Replace the following placeholders with the appropriate values for your request:

[BUCKET-NAME]: The name of the bucket containing the object.
[SOURCE-OBJECT-NAME]: The current name of the object.
[DESTINATION-OBJECT-NAME]: The new name for the object.

Rename object

aws s3api rename-object \
  --bucket [BUCKET-NAME] \
  --key [DESTINATION-OBJECT-NAME] \
  --rename-source /[BUCKET-NAME]/[SOURCE-OBJECT-NAME]

When this succeeds, a confirmation message is printed:

Example output

rename: s3://my-bucket-name/my-important-file.txt -> s3://my-bucket-name/my-new-file.txt

To rename an object, use the mv command with s3cmd.

Copy and delete operationThe s3cmd mv command performs a copy and delete operation instead of an atomic rename operation. For atomic rename operations, use the AWS CLI or Boto3 with the RenameObject API.

To prevent accidental overwrites when uploading objects, use the If-None-Match: * header with PutObject. See Conditional writes for details.

Replace the following placeholders with the appropriate values for your request:

[BUCKET-NAME]: The name of the bucket containing the source object.
[SOURCE-OBJECT-NAME]: The name of the source object to rename.
[DESTINATION-OBJECT-NAME]: The new name for the object.

Rename object

s3cmd mv s3://[BUCKET-NAME]/[SOURCE-OBJECT-NAME] s3://[BUCKET-NAME]/[DESTINATION-OBJECT-NAME]

Set environment variables for your CoreWeave credentials:
```
export ACCESS_KEY_ID="[ACCESS-KEY-ID]"
export SECRET_ACCESS_KEY="[SECRET-ACCESS-KEY]"
```
Alternatively, configure your CoreWeave credentials to work with the AWS CLI. Using a separate profile for CoreWeave AI Object Storage is recommended to avoid conflicts with your other AWS profiles and S3-compatible services; if you do not set up this configuration, you may encounter errors when using AI Object Storage.
Configure CoreWeave credentials
1. Create a new credentials file and profile in your CoreWeave configuration directory.
  Create a new credentials file and profile
  AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
2. When prompted for information, provide the following values:
  - AWS Access Key ID: The Access Key ID of your CoreWeave AI Object Storage Access Key.
  - AWS Secret Access Key: The Secret Key of your CoreWeave AI Object Storage Access Key.
  - Default region name: Optional. To set a default region, refer to the CoreWeave Availability Zones.
  - Default output format: Use json for JSON output.
3. Set the default endpoint URL to the appropriate endpoint for your use case:
  - The primary endpoint, https://cwobject.com, for use when running outside of a CoreWeave cluster.
  - The LOTA endpoint, http://cwlota.com, for use when running inside a CoreWeave cluster. The LOTA endpoint routes to the LOTA path for best performance.
  Set the primary endpoint for local development
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
4. Set the S3 addressing_style to virtual:
  Set virtual addressing style
  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set s3.addressing_style virtual --profile cw

Use the rename_object method to rename an object. Replace [BUCKET-NAME], [SOURCE-OBJECT-NAME], and [DESTINATION-OBJECT-NAME] with the appropriate values for your request.

Rename an object

import os
import boto3
from botocore.client import Config

boto_config = Config(
    region_name='US-EAST-04A',
    s3={'addressing_style': 'virtual'}
)

s3 = boto3.client(
    's3',
    endpoint_url='https://cwobject.com',
    aws_access_key_id=os.environ['ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['SECRET_ACCESS_KEY'],
    config=boto_config
)

response = s3.rename_object(
    Bucket='[BUCKET-NAME]',
    Key='[DESTINATION-OBJECT-NAME]',
    RenameSource='/[BUCKET-NAME]/[SOURCE-OBJECT-NAME]'
)

print('Object renamed successfully.')

Storage

Documentation Index

​Prerequisites

​Add objects

​Verify object integrity

​Exceeding quota limits

​List buckets and their contents

​Delete an object from a bucket

​Rename an object

Prerequisites

Add objects

Verify object integrity

Exceeding quota limits

List buckets and their contents

Delete an object from a bucket

Rename an object