Delete Files with VAST trash

Delete files from Distributed File Storage using the VAST trash directory

You can delete files from Distributed File Storage using either the VAST trash directory or the rm command.

The VAST trash directory allows asynchronous deletion of files and folders in bulk. Serial rm -rf on large directories is a slow and possibly blocking operation; moving files marked for removal to the trash folder (.vast_trash) is much faster. This guide covers using the VAST trash directory.

Prerequisites

• You have an active CKS cluster with a Distributed File Storage volume mounted.
• You have root user access to the cluster.

Accessing VAST trash

The VAST trash directory (.vast_trash) is located at the root of each volume's mount point. For example, if your volume is mounted at /mnt/vast, the VAST trash directory is at /mnt/vast/.vast_trash.

.vast_trash does not appear in directory listings (ls). To verify it exists:

Example

# Replace /mnt/vast with your actual volume mount path
$
cd /[your-volume-mount-path]/.vast_trash

If this is successful, you have trash access. If you get an error like No such file or directory, you likely have a subPath mount, which means there is no .vast_trash directory. You need to use the full volume mount instead.

Moving files to VAST trash

The root user can delete files by moving them to the VAST trash directory:

Example

$
sudo mv /[path/to/remove] /mnt/vast/.vast_trash

If you need to update the ownership of the trash directory, contact CoreWeave Support.

If you see an error like mv: cannot overwrite non-directory '.vast_trash/<name>' with directory '<name>', this can occur if the same directory name was previously deleted and VAST is still finalizing metadata cleanup. Try a different directory name or wait a few seconds before retrying.

Once moved to .vast_trash, files are permanently deleted and cannot be recovered. Attempting to list the contents of the trash directory will return an empty list:

Example

$
ls -la /mnt/vast/.vast_trash
# Output:
# total 0

Working with subPath mounts

.vast_trash is not available when using Kubernetes subPath mounts, because subPath only exposes a subdirectory of the volume, not the root where .vast_trash exists.

If you need .vast_trash access but are currently using a subPath mount, you can:

• Mount the full volume at a different location and then update your workloads to use the new mount point.
• Use symlinks to organize subdirectories within a full volume mount instead of using subPath.

To check if you're using a subPath mount, run the following command:

Example

# Check if you're using a subPath mount
$
kubectl describe -n [your-namespace] pod [your-pod-name] | grep -A6 "Mounts:"

In the Mounts section, look for entries with path="...": this indicates a subPath mount.

Diagnosing mixed subPath and full volume mounts

You may have a configuration where some directories are full volume mounts and others are subPath mounts. This can cause confusion when trying to use .vast_trash.

In this example, you have a full volume mount at /mnt/datasets and a subPath mount at /mnt/experiments:

Example

/mnt/datasets          # Full volume mount
├── .vast_trash/       # ✅ Trash accessible here
├── training-data/
└── validation/

/mnt/experiments       # subPath mount (pointing to experiments/ in another volume)
├── run-001/           # ❌ No .vast_trash visible
├── run-002/
└── checkpoints/

To verify your mount configuration, run the following command:

Example

$
kubectl describe -n [your-namespace] pod [your-pod-name] | grep -A6 "Mounts:"

You'll see output like this:

Example

    Mounts:
      /mnt/datasets from [your-volume-name] (rw)
      /mnt/experiments from [your-volume-name] (rw,path="experiments")

In the Mounts section, look for entries with path="...": this indicates a subPath mount. In this example, /mnt/experiments is a subPath mount, so it does not have its own VAST trash directory.

When you try to delete files in this setup, you will see different behavior depending on the mount type:

• Deleting from the full volume mount at /mnt/datasets works:

  Example

  # This works - trash is accessible
  $
  sudo mv /mnt/datasets/training-data/old-dataset /mnt/datasets/.vast_trash/

• Deleting from the subPath mount at /mnt/experiments fails:

  Example

  # This fails - no .vast_trash at subPath mount
  $
  sudo mv /mnt/experiments/run-001 /mnt/experiments/.vast_trash/
  # Error: No such file or directory

If you want to delete files from a subPath, like the experiments directory, you have two options:

1. Use standard deletion: rm -rf /mnt/experiments/run-001
2. Migrate to a full volume mount to gain trash access (see Working with subPath mounts above).