About Distributed File Storage

High-performance distributed file storage with snapshot and efficient deletion features for CKS workloads

Distributed File Storage delivers high write speeds, high IOPS for datapath-related operations, as well as higher read speeds in cases where block size and queue depth are low. Distributed File Storage is ideal for operations that require a lot of synchronization between Pods, and performance bottlenecks caused by contention are minimized.

Scope of Distributed File Storage

Storage access across Regions or Availability Zones is not supported. A Distributed File Storage volume can only be accessed from within the cluster where it is deployed.

Features

Encryption

All data stored in Distributed File Storage volumes is automatically encrypted at rest using AES-XTS 256-bit encryption with FIPS 140-3 validated encryption keys. Encryption is enabled by default with no configuration required.

Encryption keys are automatically generated and managed by VAST and are applied at the cluster level. Keys are not accessible to users or operators.

Distributed File Storage does not currently support encryption for data in transit between pods and the storage backend.

.snapshot folder

Every Distributed File Storage volume features a folder called .snapshot, which contains a snapshot in time of the volume. This folder is accessible in any directory on the volume.

Snapshots of Distributed File Storage volumes are taken every 6 hours, and persist for 72 hours.

Trash directory

The trash directory allows asynchronous deletion of files and folders in bulk. Serial rm -rf on large directories is a slow and possibly blocking operation; moving files marked for removal to the trash folder (.vast_trash) is much faster.

Finding the trash directory

The trash directory (.vast_trash) is located at the root of each volume's mount point. For example, if your volume is mounted at /mnt/vast, the trash directory is at /mnt/vast/.vast_trash.

.vast_trash is not available when using Kubernetes subPath mounts, because subPath only exposes a subdirectory of the volume, not the root where .vast_trash exists.

If you need .vast_trash access but are currently using a subPath mount, you can:

• Mount the full volume at a different location and then update your workloads to use the new mount point.
• Use symlinks to organize subdirectories within a full volume mount instead of using subPath.

To check if you're using a subPath mount, run the following command:

Example

# Check if you're using a subPath mount
$
kubectl describe pod <your-pod> | grep -A5 "Mounts:"

Finding .vast_trash

.vast_trash does not appear in directory listings (ls). To verify it exists:

Example

# Replace /mnt/vast with your actual volume mount path
$
cd /[your-volume-mount-path]/.vast_trash

If this is successful, you have trash access. If you get an error like No such file or directory, you likely have a subPath mount and need to use the full volume mount.

Permissions for the trash directory

The root user can delete files by moving them to trash:

Example

$
sudo mv /path/to/remove /mnt/vast/.vast_trash

If you need to update the ownership of the trash directory, contact CoreWeave Support.

Example: Diagnosing mixed subPath and full volume mounts

You may have a configuration where some directories are full volume mounts and others are subPath mounts. This can cause confusion when trying to use .vast_trash.

In this example, you have a full volume mount at /mnt/datasets and a subPath mount at /mnt/experiments:

Example

/mnt/datasets          # Full volume mount
├── .vast_trash/       # ✅ Trash accessible here
├── training-data/
└── validation/

/mnt/experiments       # subPath mount (pointing to experiments/ in another volume)
├── run-001/           # ❌ No .vast_trash visible
├── run-002/
└── checkpoints/

When you try to delete files in this setup, you will see different behavior depending on the mount type:

• Deleting from the full volume mount at /mnt/datasets works:

  Example

  # This works - trash is accessible
  $
  sudo mv /mnt/datasets/training-data/old-dataset /mnt/datasets/.vast_trash/

• Deleting from the subPath mount at /mnt/experiments fails:

  Example

  # This fails - no .vast_trash at subPath mount
  $
  sudo mv /mnt/experiments/run-001 /mnt/experiments/.vast_trash/
  # Error: No such file or directory

To verify your mount configuration, run the following command:

Example

$
kubectl describe pod [your-pod-name] | grep -A5 "Mounts:"

Look for entries with SubPath: - these mounts won't have access to the trash directory.

If you want to delete files from a subPath, like the experiments directory, you have two options:

1. Use standard deletion: rm -rf /mnt/experiments/run-001
2. Migrate to a full volume mount to gain trash access (see workarounds above).