Use Objects and Buckets

Interact with your buckets in Object Storage

This guide explains how to manage objects stored in CoreWeave AI Object Storage buckets using various tools, including the AWS CLI, s3cmd, and Boto3.

Prerequisites

This guide presumes you have the following:

• An active CoreWeave account with SAML/SSO configured.
• A valid API Access Key and Secret Token key pair.
• Adequate permissions to create buckets in CoreWeave AI Object Storage (at minimum, S3:CreateBucket permissions).
• An organization access policy set.
• A properly configured SDK of your choosing.
• Note that the primary endpoint, https://cwobject.com, requires TLS v1.3. Ensure your S3-compatible tools and OpenSSL use TLS v1.3.
• Make sure your S3 configuration uses the correct endpoint URL and has virtual-hosted addressing enabled.

Add objects

AWS CLI

Ensure you have the AWS CLI installed and configured.

Use the s3 cp command to copy a file into a bucket addressed using the s3:// scheme.

Example command

$
aws s3 cp /path/to/my-file s3://my-bucket-name

Example output

upload: ./my-important-file.txt to s3://my-bucket-name/my-important-file.txt

s3cmd

Ensure you have s3cmd installed, configured and correctly credentialed.

Use the put command to copy an object into the bucket.

Example command

$
s3cmd put /path/to/my-file s3://my-bucket-name/

This returns the following output, confirming your object has been stored in your bucket:

Example output

$
upload: 'my-important-file.txt' -> 's3://my-bucket-name/my-important-file.txt

Boto3

Ensure you have Boto3 installed, configured, and correctly credentialed before continuing. Open your Python environment and use the following script to upload a file to your S3 bucket:

Example

import boto3

# Create the S3 client
s3 = boto3.client('s3')

# Define your bucket name and the file path
bucket_name = 'my-bucket-name'
file_name = '/path/to/my-file`
object_name = 'my-important-file'  # This is optional. If you don't set a value, it will use the same string as your file_name.

# Upload file
s3.upload_file(file_name, bucket_name, object_name)

print(f'File {file_name} uploaded to {bucket_name}/{object_name}')

If this succeeds, a confirmation message is printed.

Example output

File /path/to/my-file uploaded to my-bucket-name/my-important-file

List buckets and their contents

AWS CLI

If you want to see all of your available buckets, use the ls command:

Example

$
aws s3 ls

To list all the objects currently in a bucket, use the ls command to target a bucket path.

Example

$
aws s3 ls s3://my-bucket-name

The terminal or command prompt will return a YAML file for your selected bucket, listing all objects within it, their sizes, and their last modified dates.

Example

2024-10-14 15:35:10 123456 my-first-file.txt
2024-10-14 16:45:22 234567 another-file-of-mine.txt

s3cmd

To list all your available buckets, use the ls command:

Example

$
s3cmd ls

To see the contents of a specific bucket, target the bucket's path using the s3:// scheme.

Example command

$
s3cmd ls s3://my-bucket-name/

This command will output a list of file paths including the bucket name and file name. For example:

Example output

2024-10-14 15:35 123456 s3://my-bucket-name/my-first-file.txt
2024-10-14 16:45 234567 s3://my-bucket-name/another-file-of-mine.txt

Boto3

List all your buckets programmatically in Boto3 using the following script:

Example

import boto3

# Create the S3 client
s3 = boto3.client('s3')

# List all your buckets
response = s3.list_buckets()

# Output your bucket names
print("Existing buckets:")
for bucket in response['Buckets']:
    print(f'  {bucket["Name"]}')

The following script may be used to view the contents of a specific bucket:

Example

import boto3

# Create the S3 client
s3 = boto3.client('s3')

# Define your bucket name
bucket_name = 'my-bucket-name'

# List the objects in the specified bucket
response = s3.list_objects_v2(Bucket=bucket_name)

# Output the object keys
if 'Contents' in response:
    print(f"Objects in {bucket_name}:")
    for obj in response['Contents']:
        print(f'  {obj["Key"]} (Size: {obj["Size"]} bytes)')
else:
    print(f"No objects found in {bucket_name}.")

The resulting output looks similar to the following:

Example output

Objects in my-bucket-name:
my-first-file.txt (Size: 4500 bytes)
another-file-of-mine.txt (Size: 5400 bytes)

Delete an object from a bucket

AWS CLI

To delete specific objects from a bucket, use the rm command with the AWS CLI.

Example

$
aws s3 rm s3://my-bucket-name/my-important-file.txt

When this succeeds, a confirmation message like this one is printed:

Example

$
delete: s3://my-bucket-name/my-important-file.txt

s3cmd

To delete an object from a bucket, use the del command to delete things from a bucket in Object Storage.

Example command

$
s3cmd del s3://my-bucket-name/my-important-file.txt

If this succeeds, a confirmation message is printed:

Example output

Object s3://my-bucket-name/my-important-file.txt deleted

Boto3

The following script may be used to delete an object from a bucket.

Example

import boto3

# Create the S3 client
s3 = boto3.client('s3')

# Define your bucket name and the object key you're deleting
bucket_name = 'my-bucket-name'
object_name = 'my-important-file'

# Delete the object
s3.delete_object(Bucket=bucket_name, Key=object_name)

print(f'Object {object_name} deleted from {bucket_name}')

If successful, a success message is printed:

Example output

Object my-important-file deleted from my-bucket-name

Learn more

For more information about accessing and interacting with the contents of your buckets, see the official Amazon documentation for s3 buckets.