Manage Buckets

How to manage CoreWeave AI Object Storage buckets

Buckets are the primary organizational unit for Object Storage; they are an effective way of storing vast amounts of data in flat, non-hierarchical containers. Creating and deleting buckets are both straightforward operations, which users perform programmatically using the CoreWeave Storage API.

Prerequisites

To create a bucket using the CoreWeave AI Object Storage API, you will need:

• An active CoreWeave account with SAML/SSO configured.
  • Adequate permissions to create buckets in CoreWeave AI Object Storage (at minimum, S3:CreateBucket permissions are required).
• A valid API Access Token and Secret Token pair.
• Any Cloud SDK installed locally.

Install a Cloud SDK

Below are some examples of common Cloud SDKs and how to install them.

s3cmd

To install s3cmd on macOS, use Homebrew.

Example

$
brew install s3cmd

To install s3cmd on Linux, use the package manager. For example, on Debian-based distros:

Example

$
sudo apt install s3cmd

Note

Alternatively, download the latest release from the sources listed on the s3cmd website.

Boto3

To install boto3, check that Python and pip are installed.

Example

$
python3 --version

$
pip3 --version

If they are not installed, or are not up-to-date, install them using your package manager. Then, install boto3 using pip3.

Example

$
pip3 install boto3

Next, check that boto3 has been successfully installed.

Example

$
python3 -c "import boto3; print(boto3.__version__)"

Learn more

For more information on installing the Boto3 Amazon SDK for Python, see the official Amazon documentation.

Configure local S3 tools

Install the AWS CLI

For guidance on installing the latest version of the AWS CLI in macOS or Windows, see the official Amazon documentation.

To install the AWS CLI on Linux, first check that the package index is up to date.

Example

$
sudo apt update

Next, download the latest version of the AWS CLI from Amazon using curl command to download the .zip file:

Example

$
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

Once the file is downloaded, unzip it.

Example

$
unzip awscliv2.zip

Then, run the installer script to install the AWS CLI.

Example

$
./aws/install

After the installation completes, verify it worked by checking the version:

Example

$
aws --version

Learn more

For more information about configuring access to CoreWeave AI Object Storage, see: SAML Roles for Object Storage.

Configure credentials for AWS CLI and Boto3

To configure credentials for CoreWeave AI Object Storage using the AWS CLI, start by running aws configure:

Example

$
aws configure

You will be prompted for the following information, including your API Access Token and Secret Token:

• AWS Access Key ID
• AWS Secret Key
• Default Region Name

Provide each value accordingly. Once complete, the config files for AWS CLI are stored in the following default locations.

File	Linux and macOS	Windows
Credentials file	~/aws/credentials	C:\Users\USER\aws\credentials
Configuration file	~/aws/config	C:\Users\USER\.aws.config

Important

Once you have configured the AWS CLI, both the AWS CLI and Boto3 will automatically use the credentials in the config files indicated above. If you need to change the details of your Access Keys, you must reconfigure these files.

Configure credentials for s3cmd

Install s3cmd, then run --configure:

Command

$
s3cmd --configure

When prompted for information, provide the following to s3cmd:

Value	Definition
Access Key	Your CoreWeave Access Key is an alphanumeric string between 16 and 128 characters long, e.g. MY_ACCESS_KEYAKIAIOSFODNN7. For more information, see: Manage API Access Tokens.
Secret Key	Your CoreWeave Secret Token, e.g. MY_EXAMPLE_KEY_wJalrXUtnFEMI/K7MDENG/bPxRfiCY. For more information, see: Manage API Access Tokens.
Default Region	The default Region is set to us-east-03. Leave this as is.
S3 Endpoint	See the S3 reference documentation for more information on these options.
DNS-style bucket+hostname:port template for accessing a bucket:	%(bucket)s.object.us-east-03.cwobject.com
Encryption password	Leave this field blank.

After entering this information, s3cmd tests your connection and configurations. This process saves your config file to the location /home/$USER/.s3cfg.

Create a Bucket

Warning

Please bear the following stipulations in mind when creating buckets in CoreWeave AI Object Storage.

1. You cannot create a bucket without a valid API Access key. Anonymous requests are never allowed to create buckets.
2. The user who creates the bucket automatically becomes the bucket owner.
3. Unlike standard S3, you must use the LocationConstraint argument for every bucket you create.

Create a bucket using the AWS CLI

Use the aws s3api command to create a new bucket. For example:

Example

$
aws s3api create-bucket \
    --bucket my-bucket \
    --region REGION \
    --create-bucket-configuration LocationConstraint=us-east-03

Verify that the bucket was created successfully by listing all available buckets. The output should include the name of your newly created bucket.

Example

$
aws s3api list-buckets

Create a bucket with Boto3

With Boto3, the following method can be used to define bucket creation details.

Example

 response = client.create_bucket(
    Bucket='my-bucket',
    CreateBucketConfiguration={
        'LocationConstraint': 'REGION',
    },
)

print(response)

Create a bucket with s3cmd

Open your terminal or command prompt and enter the following command, where MY-BUCKET is the desired name of the bucket.

Example

$
s3cmd mb --bucket-location=REGION s3://MY-BUCKET

Delete a bucket

If you wish to delete a bucket, follow the steps below.

Info

Only empty buckets can be deleted. All object versions and delete markers must also be removed before bucket deletion.

Delete a bucket with AWS CLI

Target the bucket to delete using aws s3api delete-bucket:

Example

$
aws s3api delete-bucket --bucket my-bucket --region us-east-03

Verify that the bucket was deleted successfully by listing all available buckets. The output should not include the name of the bucket you have deleted.

Example

$
aws s3api list-buckets

Delete a bucket with Boto3

The following method may be used to remove a bucket:

Example

response = client.delete_bucket(
    Bucket='test-bucket',
)

print(response)

Delete a bucket with s3cmd

The s3cmd command rb can be used to remove a bucket.

Example

$
s3cmd rb s3://MY-BUCKET

Get bucket statistics

Inputting the following commands will return the usage statistics for your bucket.

AWS CLI

Example

$
aws s3 ls s3://bucket --recursive --human-readable --summarize

S3Cmd

Example

$
s3cmd du -H s3://bucket-name

Boto3

Example

        response = client.list_objects(
        Bucket='bucket_name',
        )['Contents']

        bucket_size = sum(obj['Size'] for obj in response)