Skip to main content

Use Docker in SUNK

This guide explains how to use Docker inside a SUNK cluster, using a technique known as Docker-in-Docker (DinD). This approach enables automated evaluations and allows you to run benchmarks, such as SWE-bench, within your SUNK cluster.

Warning

The solution presented in this guide requires enabling privileged pods and disabling the AppArmor profile recommended for use in SUNK. This process grants elevated kernel capabilities and weakens isolation guarantees. See the known security risks section for more details.

It is your responsibility to verify that third-party code is safe to execute alongside your other workloads.

Enable Docker-in-Docker​

To disable AppArmor and run privileged pods, edit the values.yaml file of the Slurm Helm chart.

In the compute section of the slurm Helm chart:

  1. Create an s6 block to install and enable Docker.

  2. Add a volume mount for Docker using the volumeMount and volumes blocks.

  3. Under pyxis, apply the following settings:

    • Set pyxis.enabled to true.
    • Set pyxis.appArmorProfile to null to disable the AppArmor profile.
    • Set pyxis.podSecurityContext to null to remove the pod security context.

  4. Under securityContext, set securityContext.privileged to true.

    The result should resemble the following:

    Example
    compute:

      s6:

        docker:

          type: longrun

          script: |

            #!/bin/sh

            curl -fsSL https://get.docker.com | sh

            dockerd

      volumeMounts:

      - mountPath: /var/lib/docker

        name: docker-storage

      volumes:

        - name: docker-storage

          emptyDir: {}

      pyxis:

        enabled: true

        appArmorProfile: null

        podSecurityContext: null

      securityContext:

        privileged: true

  5. To verify that the AppArmor profile is correctly disabled, run the following command:

    Example
    $
    kubectl describe po <slurmd-node-name> | grep LocalhostProfile

    This should return no output. If it outputs LocalhostProfile: profiles/enroot, then the configuration is not applied correctly.

Grant users Docker access​

After disabling AppArmor and enabling privileged pods, you will need to grant users access to Docker.

If not running as root, users must belong to the sudoers group. If the SUNK cluster manages users with SUNK User Provisioning (SUP), set nsscache.sudoGroups to the groups that should have sudo privileges.

For SUP-based clusters, configure nsscache: to provision access via SCIM, and list the sudo-enabled groups under sudoGroups:. Group names must exactly match the group names in CoreWeave IAM or your upstream IdP. Refer to the SUP documentation on creating user groups

Example
 nsscache:

  sudoGroups:

   - slurm-sudo

   - additional-group

SSSD-based clusters​

For SSSD-based clusters, set the sudoGroups: configuration in directoryService: to a list of Unix groups from all directories with sudo privileges.

  • The group names are not fully-qualified for the default directory. For example, the default directory's group name should be group1 instead of [email protected].
  • The group names are fully-qualified for any additional directories.
Example
directoryService:

  sudoGroups: ["slurm-sudo", "[email protected]"]

Known security risks​

Kubernetes provides a comprehensive list of pod security standards. Below, we have outlined the subset more relevant to SUNK, but all of the standards listed in the Kubernetes documentation should be considered when assessing risk.

The setup described in this article:

  • Runs Docker inside a pod
  • Requires privileged mode
  • Disables AppArmor protections
Danger

This setup will grant full access to the /proc and sys directories, and will allow kernel-level operations, including mounting, pivoting root, and process tracing.

Item at risk: cgroups​

cgroups handle management and monitoring of resources used by Slurm jobs. In privileged mode, users could bypass cgroup restrictions, reconfigure the resource limits set by Slurm, and move processes outside of their assigned cgroups.

Item at risk: System and Security​

Disabling AppArmor on a privileged pod removes additional security features, including protections against kernel exploits. This allows unrestricted access to the /proc and /sys directories. Without AppArmor protections in place, users could gain access to and tamper with other containers in the pod, or access other devices outside the intended allocation.

Last updated on