Comment on page
Object Storage
Use CoreWeave's S3-compatible Object Storage for flexible and efficient data storage
Coreweave Object Storage is an S3-compatible storage system that allows data to be used in a flexible and efficient way, with features that include:
Object Storage is easily deployed and configured with the Cloud UI. Advanced users that need more control, encryption, or fine-grained bucket policies should deploy with Kubernetes.
Use the Object Storage section of the Cloud UI to generate tokens and s3cmd configuration files. A token is a key-pair used by utilities such as s3cmd, s5cmd, rclone, Boto3, and more.
The Object Storage link is located on the left-hand menu on the Cloud UI
To create one, click Create a New Token. This brings up the Create Token modal, which prompts you to assign a name, default region (which can be changed later), and an access level.
Create an Object Storage Token
Once created, the Access and Secret keys are available on the Object Storage page. Click the icon to the right of each key to copy it to the clipboard.
The generated Object Storage token
The download icon on the far right generates a new s3cmd configuration, with an opportunity to set the default region.
Select a new S3 Region for s3cmd
When creating the token, choose the desired access level: Read, Write, Read/Write, and Full. To learn more about the capabilities of each level, please see Identity and Access Management (IAM) and access levels below.
Any access level may be chosen from this drop-down
From the Object Storage page, the Access Level field displays the key's current access level.
Access levels are displayed on the Object Storage page
CoreWeave provides Kubernetes Custom Resource Definitions (CRDs) for programmatic and automated methods of generating access to Object Storage clusters.
In most cases, a single user can be used. If separate credentials per user are needed, generate them by deploying a user CRD. More granular permissions can be created with bucket policies.
Deploying a user CRD creates a user with access the Object Storage clusters. Each user has an access key and a secret key, which are stored in a Kubernetes secret in the namespace.
The Secret's name can be controlled with
spec.secretName as shown below. If not specified, the secret is named <namespace>-<metadata.name>-obj-store-creds. The Secret is associated with the user, and deleted when the user is deleted.Tip
As a best practice, do not share a
secretName between multiple users. Because Secrets are associated with users, they are automatically deleted when the user is deleted, disrupting other users with the same Secret name. Unless setting the secretName is important for the use-case, consider leaving it blank to generate the default name.After creating the user CRD, it can be viewed and deleted with
kubectl. It can also be viewed and deleted in the Cloud UI as described above because these methods are compatible and work on the same resource.This CRD creates
my-example user with full permissions and my-example-secret.object-storage-crd.yaml
apiVersion: objectstorage.coreweave.com/v1alpha1
kind: User
metadata:
name: my-example
namespace: my-namespace
spec:
owner: my-namespace
access: full # Possible options are: full, readwrite, read, or write
secretName: my-example-secret
To create the user with
kubectl:$ kubectl apply -f object-storage-crd.yaml
user.objectstorage.coreweave.com/my-example created
To view the user:
$ kubectl get users.objectstorage.coreweave.com
NAME OWNER ACCESS OBJECT STORAGE ACCESS OBJECT STORAGE ID
my-example my-namespace full ACTIVE my-namespace:my-example
To view
my-example-secret:$ kubectl get secrets
NAME TYPE DATA AGE
my-example-secret Opaque 2 14s
$ kubectl get secrets my-example-secret \
-o jsonpath='{.data.accessKey}' | base64 --decode
EXAMPLEKEYDALCHZ
$ kubectl get secrets my-example-secret \
-o jsonpath='{.data.secretKey}' | base64 --decode
EXAMPLEKEYueqR0hf5nbxJJRSD9gqVYqAou1Qgp8J2Z
To delete the user with
kubectl:$ kubectl delete users.objectstorage.coreweave.com my-example
user.objectstorage.coreweave.com "my-example" deleted
The corresponding Secret is deleted automatically.
When retrieving files with HTTPS, use the endpoint for the region where the bucket is located.
Region | Endpoint |
|---|---|
New York - LGA1 | https://object.lga1.coreweave.com/ |
Chicago - ORD1 | https://object.ord1.coreweave.com/ |
Las Vegas - LAS1 | https://object.las1.coreweave.com/ |
Each endpoint represents an independent, exclusive object store. This means that objects stored in
ORD1buckets are not accessible from the LAS1 region, and so on.Bucket names must be unique per region. It is a recommended practice to include the region name (such as
ord1) in the name of the bucket.Users may use any regional Object Storage endpoint and create and use buckets as they wish, but each region comes with its own quota limit. The default quota limit is
30TB of data per region.Note
CoreWeave also offers Accelerated Object Storage, a series of Anycasted NVMe-backed storage caches that provide blazing fast download speeds. Accelerated Object Storage is best suited for frequently accessed data that doesn't change often, such as model weights and training data.
One of the biggest advantages of Anycasted Object Storage Caches is that data can be pulled from across data center regions, then be cached in the data center in which your workloads are located.
For example, if your models are hosted in
ORD1 (Chicago), but have a deployment scale to all regions (ORD1, LAS1, LGA1), your call to https://accel-object.ord1.coreweave.com will be routed to a cache located closest to the workload - that is to say, if you are calling from LGA1, it will hit the cache in LGA1; if you are calling from LAS1, it will hit the cache in LAS1. This drastically reduces spin up times for workloads where scaling is a concern.Note
When using Accelerated Object Storage, there's no need to change the endpoint for every region your application is deployed in - this is the beauty of it!
Use of CoreWeave's Accelerated Object Storage is available at no additional cost. To use Accelerated Object Storage, simply modify your Object Storage endpoint to one of the addresses that corresponds to your Data Center region.
Region | Endpoint |
|---|---|
New York - LGA1 | accel-object.las1.coreweave.com |
Chicago - ORD1 | accel-object.lga1.coreweave.com |
Las Vegas - LAS1 | accel-object.ord1.coreweave.com |
Warning
Accelerated endpoints should only be used to get objects.
Do not use accelerated endpoints for any operation that lists, puts, manipulates, updates, or otherwise changes objects.
Note
CoreWeave supports Server Side Encryption via customer-provided encryption keys. The client passes an encryption key along with each request to read or write encrypted data. No modifications to your bucket need to be made to enable Server Side Encryption (SSE-C); simply specify the required encryption headers in your requests.
Important
It is the client’s responsibility to manage all keys, and to remember which key is used to encrypt each object.
The following headers are utilized to specify SSE-C customizations.
Name | Description |
|---|---|
x-amz-server-side-encryption-customer-algorithm | Use this header to specify the encryption algorithm. The header value must be AES256. |
x-amz-server-side-encryption-customer-key | Use this header to provide the 256-bit, base64-encoded encryption key to encrypt or decrypt your data. |
x-amz-server-side-encryption-customer-key-MD5 | Use this header to provide the base64-encoded, 128-bit MD5 digest of the encryption key according to RFC 1321. This header is used for a message integrity check to ensure that the encryption key was transmitted without error or interference. |
The following example demonstrates using an S3 tool to configure Server Side Encryption for Object Storage.
Note
Because SSE with static keys is not supported by
s3cmdat this time, the AWS CLI tool is used for this example. For a full explanation of the parameters used with the s3 tool in this example, review the AWS CLI s3 documentation.First, run
aws configure to set up access and to configure your Secret Keys.$ aws configure
Separately, generate a key using your preferred method. In this case, we use OpenSSL to print a new key to the file
sse.key.$ openssl rand 32 > sse.key
Important
The generated key must be 32 bytes in length.
Once the process of
aws configure is complete and your new key has been configured for use, run the following s3 commands to upload a file with Server Side Encryption.$ aws s3 --endpoint-url=https://object.las1.coreweave.com \
cp your-file.txt s3://your-bucket/your-file.txt \
--sse-c-key=fileb://sse.key \
--sse-c AES256
Finally, to retrieve the file, pass the path of the encryption key used (
sse-customer-key) to aws s3 to decrypt the file:$ aws s3 --endpoint-url=https://object.las1.coreweave.com \
cp s3://your-bucket/your-file.txt your-file.txt \
--sse-c-key=fileb://sse.key \
--sse-c AES256
When an initial key pair is created for Object Storage access, that key pair is given the permissions specified on creation in order to read, write, and modify policies of the buckets which it owns. Each key pair is considered an individual user for access, and can be used to provide granular access to applications or users.
Permission levels that may be granted are:
Permission level | CRD key | Description |
|---|---|---|
Read | read | Gives access to only read from buckets you own and have created |
Write | write | Gives access to only write to buckets you own and have created |
Read/Write | readwrite | Grants access to both read and write to buckets you own and have created |
Full | full | Grant Write/Read access, as well as admin access to create buckets and apply policies to buckets |
Currently, CoreWeave Cloud supports the following IAM bucket policy actions:
s3:AbortMultipartUploads3:CreateBuckets3:DeleteBucketPolicys3:DeleteBuckets3:DeleteBucketWebsites3:DeleteObjects3:DeleteObjectVersions3:DeleteReplicationConfigurations3:GetAccelerateConfigurations3:GetBucketACLs3:GetBucketCORSs3:GetBucketLocations3:GetBucketLoggings3:GetBucketNotifications3:GetBucketPolicys3:GetBucketRequestPayments3:GetBucketTaggings3:GetBucketVersionings3:GetBucketWebsites3:GetLifecycleConfigurations3:GetObjectAcls3:GetObjects3:GetObjectTorrents3:GetObjectVersionAcls3:GetObjectVersions3:GetObjectVersionTorrents3:GetReplicationConfigurations3:ListAllMyBucketss3:ListBucketMultipartUploadss3:ListBuckets3:ListBucketVersionss3:ListMultipartUploadPartss3:PutAccelerateConfigurations3:PutBucketAcls3:PutBucketCORSs3:PutBucketLoggings3:PutBucketNotifications3:PutBucketPolicys3:PutBucketRequestPayments3:PutBucketTaggings3:PutBucketVersionings3:PutBucketWebsites3:PutLifecycleConfigurations3:PutObjectAcls3:PutObjects3:PutObjectVersionAcls3:PutReplicationConfigurations3:RestoreObject
Important
CoreWeave Cloud does not yet support setting policies on users, groups, or roles. Currently, account owners need to grant access directly to individual users. Granting an account access to a bucket grants access to all users in that account.
For all requests, the condition keys CoreWeave currently supports are:
aws:CurrentTimeaws:EpochTimeaws:PrincipalTypeaws:Refereraws:SecureTransportaws:SourceIpaws:UserAgentaws:username
Certain S3 condition keys for bucket and object requests are also supported. In the following tables,
<perm> may be replaced withreadwrite/read-acp- or
write-acp/full-control
for read, write/read, or full control access, respectively.
Permission | Condition Keys |
|---|---|
s3:createBucket | s3:x-amz-acl, s3:x-amz-grant-<perm> |
s3:ListBucket | s3:<prefix> |
s3:ListBucketVersions | N/A |
s3:delimiter | N/A |
s3:max-keys | N/A |
s3:PutBucketAcl | s3:x-amz-acl s3:x-amz-grant-<perm> |
Permission | Condition Keys |
|---|---|
s3:PutObject | s3:x-amz-acl and s3:x-amz-grant-<perm> |
s3:x-amz-copy-source | N/A |
s3:x-amz-server-side-encryption | N/A |
s3:x-amz-server-side-encryption-aws-kms-key-id | N/A |
s3:x-amz-metadata-directive | Use PUT and COPY to overwrite or preserve metadata in COPY requests, respectively |
s3:RequestObjectTag/<tag-key> | N/A |
s3:PutObjectAcl | s3:x-amz-acl and s3-amz-grant-<perm> |
s3:PutObjectVersionAcl | s3:x-amz-acl and s3-amz-grant-<perm> |
s3:ExistingObjectTag/<tag-key> | N/A |
s3:PutObjectTagging | s3:RequestObjectTag/<tag-key> |
s3:PutObjectVersionTagging | s3:RequestObjectTag/<tag-key> |
s3:ExistingObjectTag/<tag-key> | N/A |
s3:GetObject | s3:ExistingObjectTag/<tag-key> |
s3:GetObjectVersion | s3:ExistingObjectTag/<tag-key> |
s3:GetObjectAcl | s3:ExistingObjectTag/<tag-key> |
s3:GetObjectVersionAcl | s3:ExistingObjectTag/<tag-key> |
s3:GetObjectTagging | s3:ExistingObjectTag/<tag-key> |
s3:GetObjectVersionTagging | s3:ExistingObjectTag/<tag-key> |
s3:DeleteObjectTagging | s3:ExistingObjectTag/<tag-key> |
s3:DeleteObjectVersionTagging | s3:ExistingObjectTag/<tag-key> |
Note
When using AWS SDKs, the variable
AWS_REGION is defined within the V4 signature headers. The object storage region for CoreWeave is named default.Another access control mechanism is bucket policies, which are managed through standard S3 operations. A bucket policy may be set or deleted by using
s3cmd, as shown below.
In this example, a bucket policy is created to make the bucket downloads public: $ cat > examplepol
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::happybucket/*"
]
}
]
}
The policy is then applied using
s3cmd setpolicy:$ s3cmd setpolicy examplepol s3://happybucket
Once the policy is applied, the data in your bucket may be accessed without credentials, for example, by using
curl:$ curl -v https://happybucket.object.las1.coreweave.com/my-new-file.txt
Finally, the policy is deleted using
s3cmd delpolicy:$ s3cmd delpolicy s3://happybucket
Note
Bucket policies do not yet support string interpolation.
Yes. All Object Storage tokens for an organization can access all buckets in the organization. Tokens can have different access levels.
No. Even when all tokens in the organization are deleted, the data is untouched. To delete all data, use a tool like
s3cmd or rclone to purge the buckets. The current price for Object Storage is
$0.03 per GB per month.Last modified 1mo ago