Create and Delete Buckets
How to manage CoreWeave AI Object Storage buckets with popular S3 clients.
In CoreWeave AI Object Storage, data is stored as objects consisting of the data itself, its metadata, and a unique identifier. These objects are organized into containers called buckets, which provide a namespace for objects to ensure that object names are unique within a bucket. Additionally, buckets can have specific configurations and policies to manage the data lifecycle and security.
Creating and deleting buckets are both straightforward operations that can be performed programmatically using the CoreWeave Storage API or various S3-compatible tools.
This guide explains how to create and delete CoreWeave AI Object Storage buckets using three popular clients: S3cmd, AWS CLI, and Boto3.
Prerequisites
This guide assumes that you have a CoreWeave account with permission to perform these prerequisite steps.
- Configure SAML/SSO for your organization, if it's not already set up.
- Create a valid API Access Token and Secret Token pair, if you haven't already.
- Ensure you have adequate permissions to create buckets in CoreWeave AI Object Storage. At a minimum,
S3:CreateBucket
is required. See Concepts: SAML for AI Object Storage and Reference: Bucket Policy to learn more.
Notes
Please take note of these important points when managing buckets with CoreWeave AI Object Storage.
-
You cannot create a bucket without a valid API Access key. Anonymous requests are never allowed to create buckets. To obtain an API Access key, see Manage API Access Tokens.
-
The user who creates a bucket automatically becomes the bucket owner. The bucket owner has full control over the bucket and its contents.
-
Bucket names must be globally unique across all CoreWeave locations. Attempting to create a bucket with a name that already exists in another location will result in an error.
-
Bucket names must not begin with
cw-
orvip-
. These prefixes are reserved for internal use. Attempting to create a bucket with a reserved name will result in an error. -
Only empty buckets can be deleted. Remove all objects, object versions, and delete markers before deleting a bucket.
Availability Zones
S3cmd, AWS CLI, and Boto3 use the term "region" to define the location of the object storage service. However, at CoreWeave, Regions are larger constructs that are organized into Availability Zones (AZs). When configuring your client, choose an Availability Zone from the table below in place of the traditional AWS "region".
General Access AZ | Location |
---|---|
RNO2A | Nevada, USA |
US-EAST-02A | New Jersey, USA |
US-EAST-04A | Virginia, USA |
US-EAST-06A | Ohio, USA |
US-WEST-01A | Nevada, USA |
US-WEST-04A | Arizona, USA |
When creating a new bucket, you must specify the Availability Zone in which the bucket will be created.
- When creating a bucket with S3cmd, use
--bucket-location
with the Availability Zone. - When creating a bucket with AWS CLI or Boto3, use
LocationConstraint
.
The examples that follow demonstrate proper use.
Manage buckets with S3cmd
S3cmd is a command-line tool for managing objects in S3-compatible object storage services. It can be used to create, delete, and manage buckets, as well as upload, download, and manage objects within those buckets. S3cmd is available for Linux and macOS.
Install S3cmd
The preferred S3cmd installation method is to download the latest version from GitHub. Alternatively, you can install it with the package manager for your system. For example, macOS can use Homebrew.
$brew install s3cmd
Linux distributions can use their native package manager. For example:
$sudo apt install s3cmd
The version of S3cmd available in these package managers may not be the latest version. If you require the latest version, download it from the S3cmd GitHub repository.
Configure S3cmd
S3cmd requires your API token, default Region, endpoint and other information to use CoreWeave AI Object Storage. To begin configuring S3cmd, run the following command:
$s3cmd --configure
When prompted for information, provide the following vales:
Field | Value |
---|---|
Access Key | The CoreWeave API Token is found in the Cloud Console. |
Secret Key | The CoreWeave Token Secret is shown in the Cloud Console when the token is generated. |
Default Region | The CoreWeave Availability Zone is found in the table above. |
S3 Endpoint | • cwobject.com is the primary endpoint.• cwlota.com is the LOTA endpoint. |
DNS-style template | • %(bucket)s.cwobject.com is the primary template.• %(bucket)s.cwlota.com is the LOTA template. |
Leave the remaining fields blank or press Enter
to accept the default values. Test and save the configuration when prompted.
By default, S3cmd stores its configuration in $HOME/.s3cfg
.
Create a bucket with S3cmd
To create a bucket with S3cmd, use the mb
(make bucket) command as shown below.
Replace AVAILABILITY_ZONE
with a CoreWeave Availability Zone, and BUCKET_NAME
with a globally-unique name.
$s3cmd mb --bucket-location=AVAILABILITY_ZONE s3://BUCKET_NAME
- The
--bucket-location
parameter is required. - The bucket name must be globally unique across all CoreWeave locations.
- Bucket names must not begin with
cw-
orvip-
. These prefixes are reserved for internal use.
Delete a bucket with S3cmd
To delete a bucket with S3cmd, use the rb
(remove bucket) command as shown below. Only empty buckets can be deleted. Remove all objects, object versions, and delete markers before deleting a bucket.
Replace BUCKET_NAME
with the name of your bucket.
$s3cmd rb s3://BUCKET_NAME
Report usage with S3Cmd
To report the usage of a bucket, use the du
(disk usage) command. Use the --human-readable-sizes
parameter to view information in an easier-to-understand format.
Replace BUCKET_NAME
with the name of your bucket.
$s3cmd du --human-readable-sizes s3://BUCKET_NAME
Manage buckets with AWS CLI
AWS CLI is a unified tool for managing AWS services from the command line. It provides a consistent interface for interacting with S3-compatible services, including CoreWeave AI Object Storage. Follow these steps to install and configure the AWS CLI, then use it to create and delete buckets in CoreWeave AI Object Storage.
Install the AWS CLI
AWS CLI supports macOS, Linux, and Windows. See the official installation procedures for your operating system in the AWS CLI User Guide.
Configure AWS CLI
To configure the AWS CLI location and credentials for CoreWeave AI Object Storage, run aws configure
in a terminal.
$aws configure
When prompted for information, provide the following vales:
Field | Value |
---|---|
AWS Access Key ID | The CoreWeave API Token is found in the Cloud Console. |
AWS Secret Key | The CoreWeave Token Secret is shown in the Cloud Console when the token is generated. |
Default Region Name | The CoreWeave Availability Zone is found in the table above. |
AWS CLI stores credentials and configuration in the following default locations, where $HOME
is the user's home directory on macOS, Linux, or Windows.
- Credentials file:
$HOME/.aws/credentials
- Configuration file:
$HOME/.aws/config
Next, set the endpoint URL for AWS CLI. Choose your preferred method from the documentation to set this value in the AWS CLI configuration, on the command line, or in the environment.
Use the following endpoint URLs, depending on your use case:
Endpoint URL | Description |
---|---|
https://cwobject.com | The primary endpoint for CoreWeave AI Object Storage. |
https://cwlota.com | The endpoint for LOTA. See Local Object Transfer Accelerator (LOTA). |
Boto3 shares this AWS CLI configuration, unless overridden by a Config
object or the environment. See Boto3 Configuration for details.
Create a bucket with AWS CLI
Use the aws s3api
command to create a new bucket. For example:
Replace the highlighted BUCKET_NAME
and AVAILABILITY_ZONE
below with your desired bucket name and a CoreWeave Availability Zone.
$aws s3api create-bucket \--bucket BUCKET_NAME \--region AVAILABILITY_ZONE \--create-bucket-configuration LocationConstraint=AVAILABILITY_ZONE
- The
LocationConstraint
parameter is required. - The bucket name must be globally unique across all CoreWeave locations.
- Bucket names must not begin with
cw-
orvip-
. These prefixes are reserved for internal use.
Delete a bucket with AWS CLI
To delete a bucket, use the delete-bucket
sub-command as shown. Only empty buckets can be deleted. Remove all objects, object versions, and delete markers before deleting a bucket.
Replace the highlighted BUCKET_NAME
below with the name of the bucket you wish to delete.
$aws s3api delete-bucket --bucket BUCKET_NAME
Report usage with AWS CLI
To report the usage of a bucket, use aws s3
with the ls
sub-command. Use the --recursive
, --human-readable
, and --summarize
parameters to view information in an easier-to-understand format.
Replace the highlighted BUCKET_NAME
below with the name of your bucket.
$aws s3 ls s3://BUCKET_NAME --recursive --human-readable --summarize
Manage buckets with Boto3
Boto3 is the AWS SDK for Python. It allows you to interact with AWS services programmatically using Python. Follow these steps to install and configure Boto3, then use it to create and delete buckets in CoreWeave AI Object Storage.
Install Boto3
Boto3 supports any platform that can run Python. To install Boto3, follow the steps in the official documentation.
Configure Boto3
Boto3 uses the same configuration files as the AWS CLI. If you have already configured the AWS CLI, you can skip this step. Otherwise, please configure Boto3 by following the steps in the AWS CLI Configuration section above.
Boto3 can also use a Config
object or the environment to override the AWS CLI configuration. See Boto3 Configuration for details.
Create a bucket with Boto3
To create a bucket with Boto3, use the create_bucket
method. Replace the highlighted BUCKET_NAME
and AVAILABILITY_ZONE
below with your desired bucket name and a CoreWeave Availability Zone.
response = client.create_bucket(Bucket='BUCKET_NAME',CreateBucketConfiguration={'LocationConstraint': 'AVAILABILITY_ZONE',},)print(response)
- The
LocationConstraint
parameter is required. - The bucket name must be globally unique across all CoreWeave locations.
- Bucket names must not begin with
cw-
orvip-
. These prefixes are reserved for internal use.
Delete a bucket with Boto3
To delete a bucket with Boto3, use the delete_bucket
method. Only empty buckets can be deleted. Remove all objects, object versions, and delete markers before deleting a bucket.
Replace the highlighted BUCKET_NAME
below with the name of the bucket you wish to delete.
response = client.delete_bucket(Bucket='BUCKET_NAME',)print(response)
Report usage with Boto3
To report the usage of a bucket, use the list_objects
method. Sum the size of all objects in the bucket to determine the total size.
response = client.list_objects(Bucket='BUCKET_NAME',)['Contents']bucket_size = sum(obj['Size'] for obj in response)
Next steps
Now that you've installed and configured an S3-compatible client, and created a bucket, it's time to upload objects to the bucket. See How to: Add Files and Objects to Buckets for more information.