Deploy an open-source LLM on CKS

Documentation Index

Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt

Use this file to discover all available pages before exploring further.

• Create a cluster in CKS
• Create a Node pool
• Interact with clusters and Pods using kubectl
• Deploy and interact with an LLM using Open WebUI

​

Before you begin

• kubectl is installed on your machine. kubectl is the command-line tool for interacting with Kubernetes clusters. If needed, see the kubectl installation instructions.
• Access to the CoreWeave Cloud Console. For more information, see Activate and sign in to your CoreWeave organization.
• A Hugging Face access token. See the Hugging Face instructions at User access tokens. Be sure to copy and store the token in a secure location. You will need it later in this guide.
• Access to the Llama-3.1-8B-Instruct model at Hugging Face. Go to meta-llama/Llama-3.1-8B-Instruct and request access. Note that approval for restricted models can take a few hours or longer.

​

Create a CKS cluster and Node Pool

1. Log in to the Cloud Console and navigate to the Clusters page.
2. Click the Create Cluster button.
3. In the Create a Cluster dialog, give the cluster a name, select the latest Kubernetes version, and verify the box is checked for “Enable access to the Kubernetes API via the Internet”. Click Next.
4. Create the cluster where you have GPU quota available. Verify the box is checked for “Create a default VPC,” and then click Next.
5. Leave the authentication boxes unchecked and click Next.
6. On the deploy page, click Submit.
7. On the Success! dialog box, click Create a Node Pool.
8. Verify the cluster you just created is selected, and do the following:
   • Name the Node Pool
   • Pick a GPU instance
   • Set Target Nodes to 1
   • Leave all other fields empty
   • Click Submit

​

Do not install the NVIDIA GPU Operator on CKS clusters

​

Generate a CoreWeave access token

1. In the Cloud Console, navigate to the Tokens page and click the Create Token button.
2. Enter a name and expiration and then click Create.
3. In the Create API Token dialog, select the cluster you just created from the “Select current-context” dropdown menu, and then click Download.

​

Use kubectl with your cluster

1. Make a KUBECONFIG environment variable that points to the kubeconfig file you just downloaded, for example:
   export KUBECONFIG=~/Downloads/<CWKubeconfig_file_name>
   
2. Confirm you can connect to the cluster with the following command:
   kubectl cluster-info
   
   You should see cluster information like the following:
   Kubernetes Control Plane is running at https://****.k8s.us-east-02a.coreweave.com
   CoreDNS is running at https://****.k8s.us-east-02a.coreweave.com/api/v1/namespaces/kube-system/services/coredns:dns/proxy
   node-local-dns is running at https://****.k8s.us-east-02a.coreweave.com/api/v1/namespaces/kube-system/services/node-local-dns:dns/proxy
   

​

Create a Hugging Face secret

1. Run the following command to create a Hugging Face secret:
   kubectl create secret generic hf-token-secret --from-literal=api_token=<Hugging Face token>
   
   • <Hugging Face token>: This is the token Hugging Face provides you. For more information about creating a Hugging Face token, see User access tokens.

​

Download and apply a YAML configuration file

1. Use kubectl to apply the file by running the following command:
   Before running the command, confirm you have been granted access to the Llama-3.1-8B model. Visit the meta-llama/Llama-3.1-8B-Instruct page to verify you have access.
   kubectl apply -f https://raw.githubusercontent.com/coreweave/doc-examples/refs/heads/main/cks/llm-on-cks.yaml
   
2. Confirm Kubernetes deployed the resources by running the following commands:
   kubectl get pods
   
   Verify all Pods are ready and running. The output should look like the following:
   NAME                                       READY   STATUS    RESTARTS   AGE
   llama-3-1-8b-deployment-77f4559f9f-wdvpj   1/1     Running   0          2m53s
   open-webui-5b464664d8-942cg                1/1     Running   0          2m53s
   
3. Verify the services are working by running the following commands:
   kubectl logs <llama-pod-name>
   
   • <llama-pod-name>: The Pod name beginning with llama-* that was returned by kubectl get pods.
   • In the logs, you should see the following line: INFO: Application startup complete.

​

Get Open WebUI endpoint

1. Run the following command to forward local port 8080 to the Open WebUI service:
   kubectl port-forward svc/open-webui-svc 8080:80
   
2. Leave the command running and open http://localhost:8080 in your browser.

​

Next steps

• For more information about CKS clusters, see Introduction to clusters.
• For more information about Node Pools, see Introduction to Node Pools.