Fast SSD and cost effective HDD storage are available as both block storage and shared filesystem types. All data is replicated for High Availability. Storage is allocated using Kubernetes Persistent Volume Claims. Volumes are automatically provisioned when a Persistent Volume Claim is created.
Block Storage provides the best performance, and is the recommended storage access method whenever possible. Block Storage is exposed via the Kubernetes
ReadWriteOnce access mode. Block volumes can only be attached to a single physical node at any one time.
data-hdd-pvc.yamlapiVersion: v1kind: PersistentVolumeClaimmetadata:name: dataspec:storageClassName: ceph-hdd-2-replicaaccessModes:- ReadWriteOnceresources:requests:storage: 10Gi
data-ssd-pvc.yamlapiVersion: v1kind: PersistentVolumeClaimmetadata:name: dataspec:storageClassName: ceph-ssd-2-replicaaccessModes:- ReadWriteOnceresources:requests:storage: 10Gi
Unlike block volumes a shared filesystem can be accessed by multiple nodes at the same time. This storage type is useful for parallel tasks, i.e. reading assets for CGI rendering or loading ML models for parallel inference. A shared filesystem is accessed similarly to block storage. The access mode changes to
ReadWriteMany and the storage class names are different.
shared-data-hdd-pvc.yamlapiVersion: v1kind: PersistentVolumeClaimmetadata:name: shared-dataspec:storageClassName: sharedfs-hdd-replicatedaccessModes:- ReadWriteManyresources:requests:storage: 10Gi
shared-data-ssd-pvc.yamlapiVersion: v1kind: PersistentVolumeClaimmetadata:name: shared-dataspec:storageClassName: sharedfs-ssd-replicatedaccessModes:- ReadWriteManyresources:requests:storage: 10Gi
Storage is billed per gigabyte of allocated (requested) space as an average over a billing cycle.
Volumes can be expanded by simply increasing the
storage request and re-applying the manifest.
ReadWriteMany volumes are re-sized online without disruption the workload. For
ReadWriteOnce volumes you will need to stop or restart all workloads that are attaching the volume for the reisize to take effect.
All physical nodes are equipped with SSD or NVMe ephemeral (local) storage. Ephemeral storage available ranges between 100GB to 2TB depending upon node type. No volume claims are needed to allocate ephemeral storage, simply write anywhere in the container filesystem. If a larger amount (above 20GB) of ephemeral storage is used, it is recommended to include this in the workloads resource request.
spec:containers:- name: exampleresources:limits:cpu: 3memory: 16Ginvidia.com/gpu: 1ephemeral-storage: 20Gi