Skip to main content

Intro to AI Object Storage

Highly reliable, performant Object Storage designed for AI workloads

Object Storage designed for ML/AI​

LOTA​

CoreWeave AI Object Storage is a hyper-performant S3-compatible Object Storage gateway, which leverages CoreWeave's Local Object Transfer Accelerator (LOTA), a first-of-its-kind, Node-local connection to Object Storage, designed to create a hyper-efficient path for object data to the GPU, which also caches data on GPU Nodes to reduce load times.

CoreWeave AI Object Storage is designed from the ground up for storing as well as loading training code, training checkpoints, and model weights. It efficiently serves data to both machine learning models and to model serializers, such as CoreWeave Tensorizer. LOTA then optimize retrieval speeds of cached data, ensuring quick access to the first byte. This provides a level of performance critical to the efficiency of data-intensive applications.

AI Object Storage features​

Security and availability​

AI Object Storage is designed to safeguard information that requires secure, reliable containment for long periods of time. AI Object Storage offers superior reliability by providing high redundancy, versioning, and bucket security policies.

  • Versioning for backups and archiving: CoreWeave AI Object Storage supports versioning, allowing multiple versions of data to be stored. This is useful for data recovery, and is an effective method for tracking changes over time. If required, clients can also revert to previous versions of data. AI Object Storage is ideal for archiving, backups, and use cases where data requires long-term, low-risk resiliency.
  • Bucket and lifecycle policies: AI Object Storage mitigates the risk of unauthorized access and misuse via bucket and lifecycle policies. These policies can maintain efficient access control, consistent retention scheduling, and secure data management.
  • Platform independent: Clients can use CoreWeave AI Object Storage independently of Cloud provider infrastructure. As such, it's a flexible option for various use cases and environments. CoreWeave AI Object Storage can also be connected to multiple storage backends across Cloud providers, offering a high degree of availability.

High performance​

  • Immediately accessible data: CoreWeave's AI Object Storage makes data extremely accessible as soon as it has been stored: after writing data to CoreWeave AI Object Storage, clients can re-access and read that data immediately.
  • "Read-after-write": CoreWeave AI Object Storage provides a "read-after-write" function, meaning any read operation after a write operation will retrieve updated data. This allows for secure parallel access, higher performance, and improved scalability.

Request paths​

Requests to LOTA must be made in path-style (i.e. cwobject.com/bucket).

Metrics and logging​

CoreWeave plans to offer audit logging and other metrics in the near future, but they are currently unavailable for this release. If you have specific questions about observability, please contact CoreWeave support.

Last updated on