cache-dropper sidecar to flush the Linux page cache between exclusive Slurm jobs on a SUNK compute node. It’s intended for cluster administrators and job authors who need to mitigate memory fragmentation in CPU-intensive training workloads.
Between exclusive jobs on a node, dropping the page cache can improve performance and lower memory access times caused by memory fragmentation. Memory fragmentation can lead to Out of Memory (OOM) errors and slowdowns in CPU-intensive training jobs.
SUNK includes a sidecar container, cache-dropper, to handle page cache flushes. compute.cacheDropper.enabled defaults to true, so the sidecar is on unless an administrator disables it. The sidecar runs in privileged mode, which lets it drop the cache without requiring the main slurmd container to run as privileged. It checks for the presence of a specific trigger file and drops the page cache if that file appears. The cache-dropper sidecar writes to the drop_caches sysctl file to free both page cache and reclaimable slab objects.
The cache-dropper can’t free shmem or tmpfs memory. If node pressure comes from pinned shmem (Shmem tracks Cached in /proc/meminfo), use Diagnose unexplained memory pressure on a SUNK node instead.
Dropping the cache is a non-destructive operation, but it can incur additional CPU and I/O overhead as dropped objects are recreated.
Configure the cache-dropper sidecar
The sidecar ships enabled by default. To disable it:
- In the Slurm
values.yamlfile, find thecompute.cacheDropper.enabledparameter. - Set
.compute.cacheDropper.enabledtofalse. - Save the file, and apply the changes to the cluster.
cache-dropper is present in every compute pod. It periodically checks for the existence of the /run/enroot/drop_caches file as a signal to proceed. If this file exists, it triggers the cache drop operation.
Drop the page cache with a Slurm job
Individual Slurm jobs trigger a cache drop by creating the trigger file the sidecar watches for. To usecache-dropper, add the following touch command to a Slurm job script: