CoreWeave
Search
K

Use s3cmd with CoreWeave Object Storage

S3cmd is a popular command-line tool used to manage and interact with CoreWeave Object Storage. This guide explains how to configure it and run basic commands. Before proceeding, you'll need a Linux or macOS machine with terminal access and a CoreWeave Object Storage Token.

Installation

To install s3cmd on macOS, use Homebrew.
$ brew install s3cmd
For Linux, use the system package manager. For example, on Ubuntu:
$ sudo apt-get install s3cmd
Alternatively, download the latest release from the sources listed on the s3cmd website.

Configuration

The default config file for s3cmd is located in the user's home directory at ~/.s3cfg. The configuration file can be automatically downloaded from CoreWeave, or manually created with the configuration wizard.
Tip
If desired, different configuration files can be used with the --config flag on the command-line.

Configure automatically

To automatically download the configuration, check the download box when creating the Object Storage token, or downloaded it later from the Web UI as described in the Object Storage documentation.
The downloaded file is similar to this:
cw-object-storage-config
[default]
access_key = REDACTED
secret_key = REDACTED
# The region for the host_bucket and host_base must be the same.
host_base = object.ord1.coreweave.com
host_bucket = %(bucket)s.object.ord1.coreweave.com
check_ssl_certificate = True
check_ssl_hostname = True
After downloading the file, move it to ~/.s3cfg:
$ mv ~/Downloads/cw-object-storage-config ~/.s3cfg

Configure manually

To manually configure s3cmd with a CoreWeave token, run the configuration wizard:
$ s3cmd --configure
When prompted by the wizard, enter the Access key and Secret key.
Leave the default value when prompted for the region. CoreWeave Object Storage does not use this value.
Enter an S3 endpoint when prompted.
Region
Endpoint
New York - LGA1
object.lga1.coreweave.com
Chicago - ORD1
object.ord1.coreweave.com
Las Vegas - LAS1
object.las1.coreweave.com
Enter the DNS-style template, which must be in the same region as the S3 endpoint.
Region
DNS-style template
New York - LGA1
%(bucket)s.object.lga1.coreweave.com
Chicago - ORD1
%(bucket)s.object.ord1.coreweave.com
Las Vegas - LAS1
%(bucket)s.object.las1.coreweave.com
Leave the Encryption password blank.
Encryption is optional, please refer to the s3cmd documentation for more information if you require encryption.
Take the default values for the rest of the prompts.
When complete, the session should look similar to this example.
Click to expand: s3cmd configuration session
$ s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: REDACTED
Secret Key: REDACTED
Default Region [US]:
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: object.ord1.coreweave.com
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket)s.object.ord1.coreweave.com
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:
New settings:
Access Key: REDACTED
Secret Key: REDACTED
Default Region: US
S3 Endpoint: object.ord1.coreweave.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.object.ord1.coreweave.com
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y
Configuration saved to '/home/username/.s3cfg'

Usage

The most common s3cmd commands for interacting with Object Storage are:

Create a new bucket

$ s3cmd mb s3://bucket-name

Remove a bucket

$ s3cmd rb s3://bucket-name

List buckets

$ s3cmd ls

List objects in a bucket

$ s3cmd ls s3://bucket-name

Delete all objects in a bucket

$ s3cmd del --recursive --force s3://bucket-name

Upload a file for private access

$ s3cmd put local-file.txt s3://bucket-name/remote-filename.txt

Upload a file for public access

$ s3cmd put -P local-file.txt s3://bucket-name/remote-filename.txt

Download a file

$ s3cmd get s3://bucket-name/remote-filename.txt local-file.txt

Delete a file

$ s3cmd rm s3://bucket-name/remote-filename.txt

Change file permission to public access

$ s3cmd setacl s3://bucket-name/remote-filename.txt --acl-public

Change file permission to private access

$ s3cmd setacl s3://bucket-name/remote-filename.txt --acl-private

Enable public directory listing for a bucket

$ s3cmd setacl s3://bucket-name/ --acl-public

Disable public directory listing for a bucket

$ s3cmd setacl s3://bucket-name/ --acl-private

Additional Resources

To learn more about s3cmd, please see the official documentation.