Deploy Dragonfly on CKS

​

Prerequisites

• kubectl installed and configured for your cluster.
• Helm version 3.8+.
• Git.

​

Step 1. Verify your system configuration

1. Verify that you can access your cluster with kubectl. For example:
   kubectl cluster-info
   
   You should see something similar to:
   Kubernetes control plane is running at...
   CoreDNS is running at...
   node-local-dns is running at...
   
2. Verify your cluster has at least one CPU Node. GPU Nodes are also supported, but CPU Nodes are preferred since Dragonfly cannot use GPUs for any of its functionality. For example:
   kubectl get nodes -o=custom-columns="NAME:metadata.name,CLASS:metadata.labels['node\.coreweave\.cloud\/class']"
   
   You should see something similar to the following:
   NAME      CLASS
   g137a10   gpu
   g5424e0   cpu
   g77575e   cpu
   gd926d4   gpu
   

​

Step 2. Deploy Dragonfly

1. Install the Dragonfly Operator. See the Operator installation guide for more details.
   kubectl apply -f https://raw.githubusercontent.com/dragonflydb/dragonfly-operator/main/manifests/dragonfly-operator.yaml
   
   This installs the Dragonfly Custom Resource Definition (CRD), which defines Dragonfly clusters, along with the Operator that manages them. It creates a new namespace called dragonfly-operator-system for the Operator itself. You can add dragonfly-operator as a dependency to the CoreWeave chart, which you download in the next step. However, installing it in a separate namespace from the database is preferred. The Operator can manage multiple Dragonfly clusters in different namespaces.
2. Clone the CoreWeave Dragonfly chart repository. It’s located at https://github.com/coreweave/reference-architecture/tree/main/tooling/vector_dbs/cw-dragonfly.
3. Edit the chart’s values.yaml with your details. None of the values in values.yaml must be changed, but you can adjust them for your specific use case. Keep the following principles in mind:
   • Dragonfly allocates 80% of the limit memory.
   • If the CPU limit is set, the I/O threads are equal to it.
   • If the CPU limit is not set, all visible cores are used.
   • If the CPU limit is not set and the proactor threads parameter is set, the parameter is used.
   • Ensure you have 256MiB memory per thread.
   • See CoreWeave CPU Instances for details about the number of cores and memory per Node.
   The CoreWeave chart handles the following items:
   • Provisioning a secret for the database password. You can also specify one of your own through the dbPassword attribute in values.yaml, or provide an existing secret containing the password through existingDbPasswordSecretName.
   • Setting Node affinities for the Dragonfly Pods to CPU Nodes. Pods will be scheduled onto GPU Nodes if no CPU Nodes are available.
   For example:
   affinity:
     # prefer running on CPU nodes, if available
     nodeAffinity:
       preferredDuringSchedulingIgnoredDuringExecution:
         - weight: 100
           preference:
             matchExpressions:
             - key: node.coreweave.cloud/class
               operator: In
               values:
               - cpu
   
   • Configuring snapshotting to PVC backed by VAST.
   You can control the cron expression for scheduling the job, as well as the volume size, through the following block in values.yaml.
   snapshot:
     cron: "30 7 * * *"
     enableOnMasterOnly: false
     persistentVolumeClaimSpec:
         storageClassName: shared-vast
         accessModes:
         - ReadWriteMany
         resources:
           requests:
             storage: 2Gi
   
   By default, the chart configures Dragonfly to maintain a single snapshot file and provisions a sidecar container that copies the snapshot to a persistent volume. You can control the scheduling of the snapshot copy job through snapshotMoveCron.
   Snapshots will accumulateSnapshots are not pruned, regardless of which mechanism you use (timestamped snapshots or scheduled snapshot copies). They accumulate on your volume unless you clean out unneeded ones periodically. Alternatively, you can specify a database file name as an argument. The effect is that a single snapshot is kept with that name. The cron expression governs the timing of creating that snapshot.

​

Step 3. Install the chart

1. Change to the chart directory:
   cd reference-architecture/tooling/vector_dbs/cw-dragonfly
   
2. Install the chart in a new namespace, for example, dragonfly:
   helm install -n dragonfly --create-namespace cw-dragonfly .
   
3. Check the status of the custom resource. This may take several minutes to complete, as the Operator sets up the database.
   kubectl -n dragonfly describe dragonfly cw-dragonfly
   
   The Status block shows the status of the database. Once everything is set up, that block should look like this:
   Status:
     Phase:  Ready
   

​

Step 4. Access the database

1. Forward a local port to the database service.
   kubectl -n dragonfly port-forward --address 0.0.0.0 service/cw-dragonfly 27017:6379
   
   You should see output similar to:
   Forwarding from 0.0.0.0:27017 -> 6379
   
2. Connect to the database at localhost:27017. You can use any Redis client, such as the Redis CLI, to connect:
   redis-cli -h localhost -p 27017
   localhost:27017> GET 1
   (error) NOAUTH Authentication required.
   localhost:27017> AUTH [YOUR-PASSWORD]
   OK
   localhost:27017> GET 1
   (nil)
   
   If you didn’t specify your own password, you can get the password by looking at the cw-dragonfly-db-password secret in the dragonfly namespace. Note the password is base64 encoded.

​

Additional resources