Skip to main content
This tutorial walks you through deploying Meta’s Llama 3.1 8B Instruct open source LLM on CoreWeave Kubernetes Service (CKS), so you can run inference against a hosted model from your own cluster. It’s for developers and ML practitioners who are new to CKS and want a complete, end-to-end example of provisioning GPU infrastructure and serving a model. You’ll complete the following steps:
  • Create a cluster in CKS.
  • Create a Node Pool.
  • Interact with clusters and Pods using kubectl.
  • Deploy and interact with an LLM using Open WebUI.

​
Before you begin

Before completing the steps in this guide, you must have the following:
  • Access to the Llama-3.1-8B-Instruct model at Hugging Face. Go to meta-llama/Llama-3.1-8B-Instruct and request access. Approval for restricted models can take a few hours or longer, so request access before you start the rest of this guide.
  • kubectl installed on your machine. kubectl is the command-line tool for interacting with Kubernetes clusters. If needed, see the kubectl installation instructions.
  • Access to the CoreWeave Cloud Console. For more information, see Activate and sign in to your CoreWeave organization.
  • A Hugging Face access token. See the Hugging Face instructions at User access tokens. Be sure to copy and store the token in a secure location. You need it later in this guide.
Cost and security disclaimer
  • Using resources, such as compute, incurs charges. Monitor your resource usage to avoid unexpected charges.
  • CoreWeave is not responsible for the security of the Llama model provided by Hugging Face or the Open Web UI container image.

​
Create a CKS cluster and Node Pool

CKS clusters and Node Pools are the core infrastructure for running and managing workloads. To create a cluster and Node Pool, complete the following steps:
  1. Log in to the Cloud Console and navigate to the Clusters page.
  2. Click the Create Cluster button.
  3. In the Create a Cluster dialog, give the cluster a name, select the latest Kubernetes version, and verify the box is checked for Enable access to the Kubernetes API via the Internet. Click Next. Create a cluster in the console.
  4. Create the cluster where you have GPU quota available. Verify the box is checked for Create a default VPC, and then click Next. Select the region where you have GPU quota.
  5. Leave the authentication boxes unchecked and click Next. Cluster authentication options in the create cluster flow.
  6. On the deploy page, click Submit.
  7. On the Success! dialog box, click Create a Node Pool.
  8. Verify the cluster you just created is selected, and do the following:
    • Name the Node Pool.
    • Pick a GPU instance.
    • Set Target Nodes to 1.
    • Leave all other fields empty.
    • Click Submit.
Node Pool creation can be delayed while the cluster is being created. Then, Node Pool provisioning can take up to 30 minutes. When the Node Pool status is Healthy, your cluster has GPU capacity ready to serve the model, and you can continue to the following steps.

​
Do not install the NVIDIA GPU Operator on CKS clusters

CoreWeave manages the NVIDIA GPU Operator on your behalf. Do not install the NVIDIA GPU Operator on CKS clusters. Doing so conflicts with the platform-managed deployment and is not supported.

​
Generate a CoreWeave access token

Access tokens let you authenticate to your Kubernetes resources through kubectl. You must create one for the cluster you just provisioned before you can run commands against it. To create an access token, complete the following steps:
  1. In the Cloud Console, navigate to the Tokens page and click the Create Token button.
  2. Enter a name and expiration and then click Create.
  3. In the Create API Token dialog, select the cluster you just created from the Select current-context dropdown menu, and then click Download. Create API Token dialog with current-context and download.

​
Use kubectl with your cluster

To communicate with your cluster using kubectl, complete the following steps:
  1. Make a KUBECONFIG environment variable that points to the kubeconfig file you just downloaded, for example: 
    export KUBECONFIG=~/Downloads/[CW-KUBECONFIG-FILENAME]
  2. Confirm you can connect to the cluster with the following command: 
    kubectl cluster-info
    You should see cluster information like the following: 
    Kubernetes Control Plane is running at https://****.k8s.us-east-02a.coreweave.com
CoreDNS is running at https://****.k8s.us-east-02a.coreweave.com/api/v1/namespaces/kube-system/services/coredns:dns/proxy
node-local-dns is running at https://****.k8s.us-east-02a.coreweave.com/api/v1/namespaces/kube-system/services/node-local-dns:dns/proxy

​
Create a Hugging Face secret

For CKS to download the llama-3.1-8B-Instruct model from Hugging Face, you must create a Kubernetes secret that holds your Hugging Face access token. The model deployment in the next section reads this secret at runtime to authenticate with Hugging Face. Complete the following steps to create the secret:
  1. Run the following command to create a Hugging Face secret: 
    kubectl create secret generic hf-token-secret --from-literal=api_token=[HUGGING-FACE-TOKEN]
    • [HUGGING-FACE-TOKEN]: This is the token Hugging Face provides you. For more information about creating a Hugging Face token, see User access tokens.

​
Download and apply a YAML configuration file

Kubernetes uses YAML files to configure resources. The following example YAML file defines the model deployment and the Open WebUI service so you can deploy both with a single command. To deploy the Llama-3.1-8B-Instruct model using this example, complete the following steps:
  1. Use kubectl to apply the file by running the following command:
    Before running the command, confirm you have access to the Llama-3.1-8B model. Visit the meta-llama/Llama-3.1-8B-Instruct page to verify your access.
    kubectl apply -f https://raw.githubusercontent.com/coreweave/doc-examples/refs/heads/main/cks/llm-on-cks.yaml
  2. Confirm Kubernetes deployed the resources by running the following commands: 
    kubectl get pods
    Verify all Pods are ready and running. The output should look like the following: 
    NAME                                       READY   STATUS    RESTARTS   AGE
llama-3-1-8b-deployment-77f4559f9f-wdvpj   1/1     Running   0          2m53s
open-webui-5b464664d8-942cg                1/1     Running   0          2m53s
  3. Verify the services are working by running the following commands: 
    kubectl logs [LLAMA-POD-NAME]
    • [LLAMA-POD-NAME]: The Pod name beginning with llama-* that kubectl get pods returns.
    • In the logs, look for the following line: INFO: Application startup complete.

​
Get the Open WebUI endpoint

The Open WebUI service is not exposed to the internet. To access Open WebUI from your machine, use port-forwarding:
  1. Run the following command to forward local port 8080 to the Open WebUI service: 
    kubectl port-forward svc/open-webui-svc 8080:80
  2. Leave the command running and open http://localhost:8080 in your browser.
The UI and model remain accessible only from the machine running the port-forward, not from the internet. You should now see the Open WebUI site, where you can chat with the deployed Llama 3.1 model: Open WebUI site

​
Next steps

You’ve deployed an LLM on CKS.
Last modified on June 10, 2026