Manage Buckets

How to manage CoreWeave AI Object Storage buckets with popular S3 clients.

In CoreWeave AI Object Storage, data is stored as objects consisting of the data itself, its metadata, and a unique identifier. These objects are organized into containers called buckets, which provide a namespace for objects to ensure that object names are unique within a bucket. Additionally, buckets can have specific configurations and policies to manage the data lifecycle and security.

Creating and deleting buckets are both straightforward operations that can be performed programmatically using the CoreWeave Storage API or various S3-compatible tools.

This guide explains how to create and delete CoreWeave AI Object Storage buckets using three popular clients: S3cmd, AWS CLI, and Boto3.

Prerequisites

This guide assumes that you have followed the steps in Get started with AI Object Storage. Before you begin, ensure that you have completed the following tasks:

1. Create Access Keys with either SAML/SSO assertions or Cloud Console tokens.
2. Create a bucket to store data.
3. Set policies to control access, lifecycle, and auditing.

About LOTA (Local Object Transfer Accelerator)

To use LOTA, clients must point their requests to the LOTA endpoint instead of the primary endpoint. No other changes are required for S3-compatible clients.

• The primary endpoint is https://cwobject.com
• The LOTA endpoint is http://cwlota.com

See Concepts: LOTA (Local Object Transfer Accelerator) to learn more.

Notes

Warning

Please take note of these important points before proceeding:

• All requests to CoreWeave AI Object Storage endpoints must be in DNS (virtual-hosted) style, where the bucket name is part of the domain name in the URL.
  • Use http://<bucket-name>.cwlota.com if using the LOTA endpoint.
  • Use http://<bucket-name>.cwobject.com if using the primary endpoint.
• The user who creates a bucket automatically becomes the bucket owner. The bucket owner has full control over the bucket and its contents.
• Bucket names must be globally unique across all CoreWeave locations. Attempting to create a bucket with a name that already exists in another location will result in an error.
• Bucket names must not begin with cw- or vip-. These prefixes are reserved for internal use. Attempting to create a bucket with a reserved name will result in an error.
• Only empty buckets can be deleted. Remove all objects, object versions, and delete markers before deleting a bucket.
• You cannot create a bucket without a valid API Access key. Anonymous requests are never allowed to create buckets. To obtain an API Access key, see How-To: Manage Access Keys.

Availability Zones

S3cmd, AWS CLI, and Boto3 use the term "region" to define the location of the object storage service. However, at CoreWeave, Regions are larger constructs that are organized into Availability Zones (AZs). When configuring your client, choose an Availability Zone from the table below in place of the traditional AWS "region".

General Access AZ	Location
RNO2A	Nevada, USA
US-EAST-02A	New Jersey, USA
US-EAST-04A	Virginia, USA
US-EAST-06A	Ohio, USA
US-WEST-01A	Nevada, USA
US-WEST-04A	Arizona, USA

When creating a new bucket, you must specify the Availability Zone in which the bucket will be created.

• When creating a bucket with S3cmd, use --bucket-location with the Availability Zone.
• When creating a bucket with AWS CLI or Boto3, use LocationConstraint.

The examples that follow demonstrate proper use.

Manage buckets with S3cmd

S3cmd is a command-line tool for managing objects in S3-compatible object storage services. It can be used to create, delete, and manage buckets, as well as upload, download, and manage objects within those buckets. S3cmd is available for Linux and macOS.

Install S3cmd

The preferred S3cmd installation method is to download the latest version from GitHub. Alternatively, you can install it with the package manager for your system. For example, macOS can use Homebrew.

macOS Example

$
brew install s3cmd

Linux distributions can use their native package manager. For example:

Debian example

$
sudo apt install s3cmd

The version of S3cmd available in these package managers may not be the latest version. If you require the latest version, download it from the S3cmd GitHub repository.

Configure S3cmd

S3cmd requires your API token, default Region, endpoint and other information to use CoreWeave AI Object Storage. To begin configuring S3cmd, run the following command:

Configure S3cmd

$
s3cmd --configure

When prompted for information, provide the following vales:

Field	Value
Access Key	The Access Key created by the CoreWeave AI Object Storage API.
Secret Key	The Secret Key created by the CoreWeave AI Object Storage API.
Default Region	The CoreWeave Availability Zone is found in the table above.
S3 Endpoint	• cwobject.com is the primary endpoint. • cwlota.com is the LOTA endpoint.
DNS-style template	• %(bucket)s.cwobject.com is the primary template. • %(bucket)s.cwlota.com is the LOTA template.

Leave the remaining fields blank or press Enter to accept the default values. Test and save the configuration when prompted.

By default, S3cmd stores its configuration in $HOME/.s3cfg.

Create a bucket with S3cmd

To create a bucket with S3cmd, use the mb (make bucket) command as shown below.

Replace AVAILABILITY_ZONE with a CoreWeave Availability Zone, and BUCKET_NAME with a globally-unique name.

Make a bucket with S3cmd

$
s3cmd mb --bucket-location=AVAILABILITY_ZONE s3://BUCKET_NAME

Important

• The --bucket-location parameter is required.
• The bucket name must be globally unique across all CoreWeave locations.
• Bucket names must not begin with cw- or vip-. These prefixes are reserved for internal use.

Delete a bucket with S3cmd

To delete a bucket with S3cmd, use the rb (remove bucket) command as shown below. Only empty buckets can be deleted. Remove all objects, object versions, and delete markers before deleting a bucket.

Replace BUCKET_NAME with the name of your bucket.

Remove a bucket with S3cmd

$
s3cmd rb s3://BUCKET_NAME

Report usage with S3Cmd

To report the usage of a bucket, use the du (disk usage) command. Use the --human-readable-sizes parameter to view information in an easier-to-understand format.

Replace BUCKET_NAME with the name of your bucket.

Report bucket usage with S3cmd

$
s3cmd du --human-readable-sizes s3://BUCKET_NAME

Manage buckets with AWS CLI

AWS CLI is a unified tool for managing AWS services from the command line. It provides a consistent interface for interacting with S3-compatible services, including CoreWeave AI Object Storage. Follow these steps to install and configure the AWS CLI, then use it to create and delete buckets in CoreWeave AI Object Storage.

Install the AWS CLI

AWS CLI supports macOS, Linux, and Windows. See the official installation procedures for your operating system in the AWS CLI User Guide.

Configure AWS CLI

To configure the AWS CLI location and credentials for CoreWeave AI Object Storage, run aws configure in a terminal.

Configure the AWS CLI

$
aws configure

When prompted for information, provide the following vales:

Field	Value
AWS Access Key ID	The Access Key created by the CoreWeave AI Object Storage API.
AWS Secret Key	The Secret Key created by the CoreWeave AI Object Storage API.
Default Region Name	The CoreWeave Availability Zone is found in the table above.

AWS CLI stores credentials and configuration in the following default locations, where $HOME is the user's home directory on macOS, Linux, or Windows.

• Credentials file: $HOME/.aws/credentials
• Configuration file: $HOME/.aws/config

Next, set the endpoint URL for AWS CLI. Choose your preferred method from the documentation to set this value in the AWS CLI configuration, on the command line, or in the environment.

Use the following endpoint URLs, depending on your use case:

Endpoint URL	Description
https://cwobject.com	The primary endpoint for CoreWeave AI Object Storage.
http://cwlota.com	The endpoint for LOTA. See Local Object Transfer Accelerator (LOTA).

Important

Boto3 shares this AWS CLI configuration, unless overridden by a Config object or the environment. See Boto3 Configuration for details.

Create a bucket with AWS CLI

Use the aws s3api command to create a new bucket. For example:

Replace the highlighted BUCKET_NAME and AVAILABILITY_ZONE below with your desired bucket name and a CoreWeave Availability Zone.

Create a bucket with AWS CLI

$
aws s3api create-bucket \
    --bucket BUCKET_NAME \
    --region AVAILABILITY_ZONE \
    --create-bucket-configuration LocationConstraint=AVAILABILITY_ZONE

Important

• The LocationConstraint parameter is required.
• The bucket name must be globally unique across all CoreWeave locations.
• Bucket names must not begin with cw- or vip-. These prefixes are reserved for internal use.

Delete a bucket with AWS CLI

To delete a bucket, use the delete-bucket sub-command as shown. Only empty buckets can be deleted. Remove all objects, object versions, and delete markers before deleting a bucket.

Replace the highlighted BUCKET_NAME below with the name of the bucket you wish to delete.

Delete a bucket with AWS CLI

$
aws s3api delete-bucket --bucket BUCKET_NAME

Report usage with AWS CLI

To report the usage of a bucket, use aws s3 with the ls sub-command. Use the --recursive, --human-readable, and --summarize parameters to view information in an easier-to-understand format.

Replace the highlighted BUCKET_NAME below with the name of your bucket.

Report bucket usage with AWS CLI

$
aws s3 ls s3://BUCKET_NAME --recursive --human-readable --summarize

Manage buckets with Boto3

Boto3 is the AWS SDK for Python. It allows you to interact with AWS services programmatically using Python. Follow these steps to install and configure Boto3, then use it to create and delete buckets in CoreWeave AI Object Storage.

Install Boto3

Boto3 supports any platform that can run Python. To install Boto3, follow the steps in the official documentation.

Configure Boto3

Boto3 uses the same configuration files as the AWS CLI. If you have already configured the AWS CLI, you can skip this step. Otherwise, please configure Boto3 by following the steps in the AWS CLI Configuration section above.

Boto3 can also use a Config object or the environment to override the AWS CLI configuration. See Boto3 Configuration for details.

Create a bucket with Boto3

To create a bucket with Boto3, use the create_bucket method. Replace the highlighted BUCKET_NAME and AVAILABILITY_ZONE below with your desired bucket name and a CoreWeave Availability Zone.

Create a bucket with Boto3

 response = client.create_bucket(
    Bucket='BUCKET_NAME',
    CreateBucketConfiguration={
        'LocationConstraint': 'AVAILABILITY_ZONE',
    },
)

print(response)

Important

• The LocationConstraint parameter is required.
• The bucket name must be globally unique across all CoreWeave locations.
• Bucket names must not begin with cw- or vip-. These prefixes are reserved for internal use.

Delete a bucket with Boto3

To delete a bucket with Boto3, use the delete_bucket method. Only empty buckets can be deleted. Remove all objects, object versions, and delete markers before deleting a bucket.

Replace the highlighted BUCKET_NAME below with the name of the bucket you wish to delete.

Delete a bucket with Boto3

response = client.delete_bucket(
    Bucket='BUCKET_NAME',
)

print(response)

Report usage with Boto3

To report the usage of a bucket, use the list_objects method. Sum the size of all objects in the bucket to determine the total size.

Report bucket usage with Boto3

        response = client.list_objects(
        Bucket='BUCKET_NAME',
        )['Contents']

        bucket_size = sum(obj['Size'] for obj in response)

Next steps

Now that you've installed and configured an S3-compatible client, and created a bucket, it's time to upload objects to the bucket. See How to: Add Files and Objects to Buckets for more information.