Get Started with AI Object Storage

Set up AI Object Storage authentication, policies, and storage access for enterprise workloads

This guide provides a high-level overview of the steps required to set up and use AI Object Storage.

Prerequisites

Administrative access to your CoreWeave organization
A Cloud Console token with admin permissions

Configure authentication

Access keys are required for users and workloads to access AI Object Storage. For testing and validation, you can get started in minutes with Cloud Console tokens.

Create an access key using Cloud Console tokens:

You must be logged in with an account that has admin permissions.

In the Cloud Console, navigate to Object Storage > Keys.
Click Create Key.
In the dialog:
- For a Temporary key, enter the desired duration in seconds and click Create.
- For a Permanent key, select Enable Permanent Key and click Create.

Save the Access Key ID and Secret Key for later use.

For the most secure setup in production, use SAML assertions with Workload Identity Federation to generate workload Access Keys.

After setting up access keys for authentication, you need to set a policy for access control.

Set an organization access policy

You must create an organization access policy in order to access the Object Storage S3-compatible API. This sets a baseline level of access for your organization to the Object Storage S3-compatible API.

For testing and validation, you can start with a simple policy that grants full access to all S3 buckets and objects to all SAML admins. Create a policy using the Cloud Console:

In the Cloud Console, navigate to Object Storage > Organization Access Policies.
In the upper right corner, click Create Policy.
In the Create Policy page, enter the Policy Name. For example, "admin-test-policy".
Within the policy, add one statement. In the Statement section, enter a descriptive Name for the statement. For example, "full-admin-access".
For Access, select Allow access.
For Principals, enter role/Admin.
For both Actions and Resources, enter *.
Click Submit to create the policy.

Alternatively, you can create a policy programmatically using the AI Object Storage API or the CoreWeave Terraform provider. See examples of organization access policies.

After setting up an organization access policy, you can configure your development environment and then set up buckets and objects.

Set CoreWeave-specific configurations

Before you create your first bucket in AI Object Storage, ensure the following settings are applied in your AWS configuration file or environment variables. This is a one-time configuration to add CoreWeave-specific settings to your development environment.

Create a new credentials file and profile in your CoreWeave configuration directory. Using a separate profile for CoreWeave AI Object Storage is recommended to avoid conflicts with your other AWS profiles and S3-compatible services.

Create a new credentials file and profile
```
$
AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
```
When prompted for information, provide the following values:
- AWS Access Key ID: The Access Key ID of your CoreWeave AI Object Storage Access Key.
- AWS Secret Access Key: The Secret Key of your CoreWeave AI Object Storage Access Key.
- Default region name: Optional. To set a default region, refer to the CoreWeave Availability Zones.
- Default output format: Use json for JSON output.
Set the default endpoint URL to the appropriate endpoint for your use case:
- The primary endpoint, https://cwobject.com, for use when running outside of a CoreWeave cluster.
- The LOTA endpoint, http://cwlota.com, for use when running inside a CoreWeave cluster. The LOTA endpoint routes to the LOTA cache for best performance.
Set the primary endpoint for local development
```
$
AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
```

Set the S3 addressing_style to virtual:

Set virtual addressing style

$
AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set default.s3.addressing_style virtual --profile cw

For more details and full configuration examples, including Boto3, see Configure Endpoints.

Create a bucket

After creating access keys, an organization access policy, and setting your global configuration, create a bucket to store data. Buckets are the primary organizational unit used by CoreWeave AI Object Storage to store object data.

To create a bucket with the CoreWeave Cloud Console, follow these steps:

In the Cloud Console, navigate to Object Storage > Buckets.
Click Create Bucket.
In the dialog, enter a Bucket Name. (Bucket names must be globally-unique and not begin with cw- or vip-, which are reserved for internal use.)
Choose an Availability Zone from the dropdown menu.
Click Create.

Alternatively, you can create buckets programmatically using popular S3-compatible tools.

Use objects and buckets

AI Object Storage is compatible with the S3 API, so you can use any S3-compatible tool to manage your data. See How-To: Manage Objects to learn how to use AI Object Storage with s3cmd, Boto3, or the AWS CLI.

For example, use the AWS CLI to copy a file into a bucket addressed using the s3:// scheme:

Example command

$
aws s3 cp ./my-important-file.txt s3://my-bucket-name --profile cw

Example output

upload: ./my-important-file.txt to s3://my-bucket-name/my-important-file.txt

Alternatively, you can use Cyberduck to manage your buckets and objects in a graphical interface. Currently, the Cloud Console does not support managing objects.

CoreWeave AI Object Storage has default quota limits. See Manage quota limits for more information.

Endpoint selection for best performance

AI Object Storage works with LOTA (Local Object Transport Accelerator), a first-of-its-kind intelligent proxy that provides a highly efficient, local gateway to AI Object Storage. Always use the LOTA endpoint when running inside a CoreWeave cluster to improve performance. See How to use LOTA for best practices.

Endpoint	URL	Notes
Primary endpoint	`https://cwobject.com`	Uses HTTPS and requires TLS v1.3. Ensure your S3-compatible tools and OpenSSL use TLS v1.3. Use this endpoint when running outside of a CoreWeave cluster.
LOTA endpoint	`http://cwlota.com`	Uses HTTP protocol, not HTTPS. For best performance, the LOTA cache routes to the LOTA endpoint. Always use this endpoint when running inside a CoreWeave cluster.

Next steps

Now that you've created a bucket and uploaded data, you can set up authentication for production workloads and configure bucket access policies and versioned buckets.

Set a bucket access policy

You may want to set a bucket access policy to refine access control for each bucket. Bucket access policies are used to control access to data, control lifecycles, and configure auditing. Bucket access policies are evaluated after organization access policies, and can be used with a broader organization access policy to narrow permissions on specific buckets. You can also use bucket access policies to allow users from other organizations to access your bucket.

You can set bucket access policies programmatically using S3-compatible tools like the AWS CLI, s3cmd, or Boto3, and also using the CoreWeave Terraform provider. See how to manage bucket access policies.

Manage versioned buckets

Versioned buckets provide a way to keep multiple variants of an object in the same bucket. When enabled, all objects are assigned a unique version ID. This allows you to retrieve, restore, or delete specific versions of an object. Versioning is disabled by default.

You can use rclone and the AWS CLI to manage versioned buckets with AI Object Storage. See Versioned Buckets for more information.

Set up authentication for production workloads

This section covers how to implement secure, automated credential management for production workloads.

Workload Identity Federation provides a more secure, controlled way to generate time-limited Access Keys using SAML assertions, and is recommended for granting workloads access to AI Object Storage in production environments.

Before you can generate Access Keys for AI Object Storage using Workload Identity Federation, you must first configure Workload Identity Federation for your organization.

After you configure Workload Identity Federation, you can use it to generate access keys for your workloads by submitting a CreateAccessKeysFromSAML request to the AI Object Storage API.

This request contains:

A SAML assertion (usually generated by an IdP),
a configId, generated from the Workload Identity Federation Console, and
a duration parameter to specify the maximum lifespan of the Access Key.

Example

{
  "durationSeconds": 300,
  "orgId": "abc123",
  "configId": "<WORKLOAD_FEDERATION_CONFIG_ID>",
  "samlResponse": "<BASE64_ENCODED_SAML_RESPONSE>"
}

Learn more about creating access keys with Workload Identity Federation.

API reference and compatibility information

For API reference for AI Object Storage, refer to the API documentation.
For S3 API compatibility reference, refer to the S3 compatibility documentation.

Prerequisites​

Configure authentication​

Set an organization access policy​

Set CoreWeave-specific configurations​

Create a bucket​

Use objects and buckets​

Endpoint selection for best performance​

Next steps​

Set a bucket access policy​

Manage versioned buckets​

Set up authentication for production workloads​

API reference and compatibility information​