- **True**: online
- **False**: not online
- **OK**: The GPUs peer correctly.
- **X**: The GPU peers with itself. This is ignored.
- **NS**: not supported
- **Passed**: The most recent run completed successfully.
- **Failed**: The most recent run failed.
- **Not Run**: A check hasn't run yet. This is common for newly delivered Nodes in a CKS cluster, since the verification data is stored on the Kubernetes host, not the Node itself.
- **None**
- **Pending**
- **Firing**
- **Yes**: The Node is running a workload.
- **No**: The Node isn't running a workload.
- Electrical current
- Power
- Power consumption (server-wide)
- Voltage
- Temperature
- Fan speed
- Fan speed ratio
- System Event Log (SEL) free space
- System Event Log (SEL) log count
- Chassis power state
- Electrical current sensor state
- Fan speed sensor state
- Power sensor state
- Cable/interconnect VGA cable presence
- Power supply status
- Serial cable status
- PCIe slot critical interrupt
- Physical security intrusion
- Fan redundancy state
- CMOS battery state
- Temperature sensor state
- Voltage sensor state
- Read Bandwidth
- Write Bandwidth
- Average Request Time by Operation
- Queue Time by Operation and VIP
- Timeouts by Operation
- Average Response Time by Operation
- Transmissions
- Retransmissions
- Mean Queue Time by Mount Address
- `node_infiniband_port_data_received_bytes_total`
- `node_infiniband_port_data_transmitted_bytes_total`
- `node_infiniband_port_transmit_wait_total`
- `node_infiniband_rate_bytes_per_second`
- Log metrics and rich media during training and fine-tuning.
- Track and compare experiments.
- Analyze runs and results, such as loss curves and accuracy scores.
- Tune hyperparameters.
- Trace agentic AI applications.
- Analyze traces and classify failure modes.
- Evaluate models and agents with labeled datasets.
- Run online evaluations for production monitoring.
- **Internal secret manager** (default): `k8s-secretmanager://wandb-secret`
- **AWS Secret Manager**: `aws-secretmanager`
- **Google Cloud Secret Manager**: `gcp-secretmanager`
- **Azure**: `az-secretmanger`
- [Instance-level BYOB](/products/wandb/platform/hosting/data-security/secure-storage-connector): Synced to instance-level bucket (BYOB) every 10 minutes. Also available with [the API](#fetch-audit-logs).
- Default instance-level storage: Available only with [the API](#fetch-audit-logs).
mysql:
ingress:
annotations:
```
4. Start the Operator with your custom configuration so that it can install, configure, and manage the W\&B Server application:
```shell theme={"system"}
kubectl apply -f operator.yaml
```
Wait until the deployment completes. This takes a few minutes.
5. To verify the installation using the web UI, create the first admin user account, then follow the verification steps outlined in [Verify the installation](#verify-the-installation).
After these steps complete, you have a W\&B Kubernetes Operator running in the `wandb-cr` namespace and a W\&B Server application that the operator manages from your `operator.yaml` custom resource.
**Terraform**
Deploy Weights & Biases using Terraform for infrastructure-as-code deployments. Choose between:
* **Helm Terraform Module**: Deploys the operator to existing Kubernetes infrastructure.
* **Cloud Terraform Modules**: Complete infrastructure plus application deployment for AWS, Google Cloud, and Azure.
For deployment-specific considerations, see [Environment-specific considerations](#environment-specific-considerations) and [Deploy with Terraform on public cloud](#deploy-with-terraform-on-public-cloud). For disconnected environments, see [Deploy on Air-Gapped Kubernetes](/products/wandb/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped).
#### Helm Terraform Module
This method supports customized deployments tailored to specific requirements, using Terraform's infrastructure-as-code approach for consistency and repeatability. The official W\&B [Helm-based Terraform Module](https://registry.terraform.io/modules/wandb/wandb/helm/latest) is available on the Terraform Registry.
Use the following code as a starting point. It includes all necessary configuration options for a production grade deployment:
```hcl theme={"system"}
module "wandb" {
source = "wandb/wandb/helm"
spec = {
values = {
global = {
host = "https://"
license = "eyJhbGnUzaH...j9ZieKQ2x5GGfw"
bucket = {
}
mysql = {
}
}
ingress = {
annotations = {
"a" = "b"
"x" = "y"
}
}
}
}
}
```
The configuration options are the same as described in [Configuration Reference](#configuration-reference-for-wb-server), but the syntax must follow the HashiCorp Configuration Language (HCL). The Terraform module creates the Weights & Biases custom resource definition (CRD).
To see how Weights & Biases themselves use the Helm Terraform module to deploy Dedicated Cloud installations for customers, follow these links:
* [AWS](https://github.com/wandb/terraform-aws-wandb/blob/45e1d746f53e78e73e68f911a1f8cad5408e74b6/main.tf#L225)
* [Azure](https://github.com/wandb/terraform-azurerm-wandb/blob/170e03136b6b6fc758102d59dacda99768854045/main.tf#L155)
* [Google Cloud](https://github.com/wandb/terraform-google-wandb/blob/49ddc3383df4cefc04337a2ae784f57ce2a2c699/main.tf#L189)
#### Cloud Terraform Modules
Weights & Biases provides a set of Terraform Modules for AWS, Google Cloud, and Azure. These modules deploy the complete infrastructure, including Kubernetes clusters, load balancers, and MySQL databases, along with the W\&B Server application. The W\&B Kubernetes Operator is included in these official Weights & Biases cloud-specific Terraform Modules with the following versions:
| Terraform Registry | Source Code | Version |
| - | - | - |
| [AWS](https://registry.terraform.io/modules/wandb/wandb/aws/latest) | [https://github.com/wandb/terraform-aws-wandb](https://github.com/wandb/terraform-aws-wandb) | v4.0.0+ |
| [Azure](https://github.com/wandb/terraform-azurerm-wandb) | [https://github.com/wandb/terraform-azurerm-wandb](https://github.com/wandb/terraform-azurerm-wandb) | v2.0.0+ |
| [Google Cloud](https://github.com/wandb/terraform-google-wandb) | [https://github.com/wandb/terraform-google-wandb](https://github.com/wandb/terraform-google-wandb) | v2.0.0+ |
These modules install the W\&B Kubernetes Operator as part of the deployment, so you can use it to manage W\&B Server in your cloud environment without additional setup.
For detailed instructions on using these cloud-specific modules, see [Deploy with Terraform on public cloud](#deploy-with-terraform-on-public-cloud).
### Verify the installation
To verify the installation, W\&B recommends using the [W\&B CLI](/products/wandb/ref/cli). The `wandb verify` command runs tests that confirm components and configurations work as expected.
> [!NOTE]
>
> This procedure assumes that you create the first admin user account in a browser.
To verify the installation:
1. Install the W\&B CLI:
```bash theme={"system"}
pip install wandb
```
2. Log in to W\&B:
```bash theme={"system"}
wandb login --host=https://YOUR_DNS_DOMAIN
```
For example:
```bash theme={"system"}
wandb login --host=https://wandb.company-name.com
```
3. Verify the installation:
```bash theme={"system"}
wandb verify
```
After the command runs, a successful installation displays the following output:
```console theme={"system"}
Default host selected: https://wandb.company-name.com
Find detailed logs for this test at: /var/folders/pn/b3g3gnc11_sbsykqkm3tx5rh0000gp/T/tmpdtdjbxua/wandb
Checking if logged in...................................................✅
Checking signed URL upload..............................................✅
Checking ability to send large payloads through proxy...................✅
Checking requests to base url...........................................✅
Checking requests made over signed URLs.................................✅
Checking CORs configuration of the bucket...............................✅
Checking wandb package version is up to date............................✅
Checking logged metrics, saving and downloading a file..................✅
Checking artifact save and download workflows...........................✅
```
Contact W\&B Support if you encounter errors.
## Enable the MCP server
The W\&B MCP Server, covered in [Use Weights & Biases with AI Assistants](/products/wandb/platform/ai-assistants), ships as an optional subchart in `operator-wandb`. When enabled, the operator deploys an in-cluster MCP server exposed through your existing ingress at `/mcp`, so any MCP-compatible client can connect using a Forge API Key. This is the same server Weights & Biases runs as the hosted offering at `https://mcp.withwandb.com/mcp`, but pointed at your deployment's data.
For end-user client configuration and the tool catalog, see [Use Weights & Biases with AI Assistants](/products/wandb/platform/ai-assistants). This section only covers the operator-side enablement.
### Prerequisites
Make sure your deployment meets the following requirements before you enable the MCP server:
* **Chart version**: `operator-wandb` `0.42.3` or later. The `mcp-server` subchart was introduced in `0.42.1`, but the Datadog and privacy fields used in the following example were added later.
* **Weave Traces enabled**: the MCP server depends on Weave Traces for trace tools and for the `WF_TRACE_SERVER_URL` default. Set `weave-trace.install: true`. If Weave Traces isn't enabled, the Helm render fails with `mcp-server requires weave-trace.install=true`.
* **Reachable ingress**: `global.host` must already resolve and route to the Weights & Biases ingress. The MCP pod reads `WANDB_BASE_URL` from `global.host` and is available at `/mcp`.
* **Node capacity**: the MCP pod requests `500m` CPU and `1Gi` memory by default (limits `2` CPU and `4Gi` memory). Confirm your node pool has enough headroom before you enable the subchart.
### Enable the subchart
Enable the `mcp-server` subchart so that the operator deploys an in-cluster MCP server and extends your existing Weights & Biases ingress with a `/mcp` route. Add the following to the `spec.values` block of your existing `WeightsAndBiases` custom resource (CR), alongside your existing `global`, `ingress`, and other overrides. The Datadog block is optional, but recommended when a Datadog Agent DaemonSet already collects pod logs and traces in your cluster.
```yaml theme={"system"}
spec:
values:
weave-trace:
install: true
mcp-server:
install: true
image:
repository: us-docker.pkg.dev/wandb-production/public/wandb/mcp-server
tag: "0.3.3"
datadog:
enabled: true
mode: "agent"
service: "wandb-mcp-server-"
env: ""
deploymentType: "self-managed"
customer: ""
extraTags:
- "region:"
- "tier:"
privacy:
logLevel: "standard"
```
Configure each block:
* **`weave-trace.install: true`**: required unless you set `mcp-server.env.WF_TRACE_SERVER_URL` yourself.
* **`datadog.mode: "agent"`**: use for Kubernetes deployments where the Datadog Agent DaemonSet owns log and trace collection. In agent mode, the MCP pod doesn't need a Datadog API key.
* **`datadog.service`, `env`, `deploymentType`, `customer`, `extraTags`**: set these to match your deployment's observability naming conventions. Set `customer` to an empty string if you don't want a customer tag.
* **`privacy.logLevel`**: use `"standard"` for most self-managed Kubernetes installations. This redacts free-text parameter values in logs while preserving deployment identifiers that operators commonly use for debugging. Use `"strict"` when entity, project, run, or user identifiers should not remain in plaintext logs. Use `"off"` only when you explicitly want plaintext logging for those values.
Apply the change to trigger reconciliation:
```bash theme={"system"}
kubectl apply -f operator.yaml
```
The operator creates a `wandb-mcp-server` deployment and service in the release namespace, and extends the Weights & Biases ingress with a `/mcp` path.
### Verify the MCP server
Wait for the pod to become `Running`, then check the health endpoint in-cluster and through the ingress:
```bash theme={"system"}
kubectl get pod -l app.kubernetes.io/component=mcp-server
kubectl port-forward svc/wandb-mcp-server 8080:8080
curl -s http://localhost:8080/mcp/health
curl -s "https:///mcp/health"
```
Both requests should return `200 OK`. The in-cluster check confirms the pod is healthy. The ingress check confirms routing. If the in-cluster check returns `200 OK` but the ingress check returns `404 Not Found`, see [Troubleshooting](#troubleshooting). If you enabled Datadog, MCP server logs should also appear in Datadog with the configured `mcp-server.datadog.service` and `mcp-server.datadog.env` values.
### Connect a client
After the MCP server is healthy, configure your MCP client to use `https:///mcp` with an API Key as the bearer token. For IDE and agent configurations, see [Use Weights & Biases with AI Assistants](/products/wandb/platform/ai-assistants).
### Troubleshooting
| Symptom | Cause and fix |
| - | - |
| `helm render` fails with `mcp-server requires weave-trace.install=true` | Add `weave-trace.install: true` to `spec.values`. The MCP server depends on Weave Traces for trace tools. |
| `wandb-mcp-server` pod stuck in `Pending` with `Insufficient cpu` or `Insufficient memory` | Add node capacity, or lower `mcp-server.resources.requests` in your CR. Defaults are `500m` CPU and `1Gi` memory. |
| `curl https:///mcp/health` returns 404 | The chart renders the `/mcp` ingress path only when `mcp-server.install: true`. Reapply the CR and wait for the ingress controller to propagate the new path. |
| MCP logs don't appear in Datadog | Confirm `mcp-server.datadog.enabled: true`, `mcp-server.datadog.mode: "agent"`, and that the Datadog Agent DaemonSet collects pod stdout. Search Datadog with the configured `service` and `env` values. |
| MCP logs include more user-supplied text than expected | Set `mcp-server.privacy.logLevel` to `"standard"` or `"strict"`. Use `"strict"` when identifiers such as entity, project, run, or user names should not remain in plaintext logs. |
| `wandb-mcp-server` pod in `ImagePullBackOff` in an air-gapped or mirrored cluster | Mirror the image to your registry and override `mcp-server.image.repository` in your CR, the same pattern used for other Weights & Biases component images in air-gapped installs. See [Deploy on Air-Gapped Kubernetes](/products/wandb/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped). |
## Environment-specific considerations
Kubernetes is the same whether it runs on-premises or in the cloud. The main differences are in naming and managed services (for example, MySQL compared to RDS, or S3 compared to on-premises object storage). This section covers considerations that vary by environment.
### On-premises and bare metal
When deploying to on-premises or bare-metal Kubernetes, pay attention to the following.
#### Load balancer configuration
On-premises Kubernetes clusters typically require manual load balancer configuration. Options include:
* **External load balancer**: Configure an existing hardware or software load balancer, such as F5 or HAProxy.
* **Nginx Ingress Controller**: Deploy nginx-ingress-controller with NodePort or host networking.
* **MetalLB**: For bare-metal Kubernetes clusters, MetalLB provides load balancer services.
For detailed load balancer configuration examples, see the [Reference Architecture networking section](/products/wandb/platform/hosting/self-managed/ref-arch#networking).
#### Persistent storage
Ensure your Kubernetes cluster has a StorageClass configured for persistent volumes. Weights & Biases components might require persistent storage for caching and temporary data.
Common on-premises storage options include:
* NFS-based storage classes
* Ceph/Rook storage
* Local persistent volumes
* Enterprise storage solutions such as NetApp or Pure Storage
#### DNS and certificate management
For on-premises deployments, complete the following tasks:
* Configure internal DNS records to point to your Weights & Biases hostname.
* Provision SSL/TLS certificates from your internal Certificate Authority (CA).
* If using self-signed certificates, configure the operator to trust your CA certificate.
See the [SSL/TLS requirements](/products/wandb/platform/hosting/self-managed/requirements#ssl-tls) for certificate configuration details.
#### OpenShift deployments
Weights & Biases fully supports deployment on OpenShift Kubernetes clusters. OpenShift deployments require additional security context configurations due to OpenShift's stricter security policies.
For OpenShift-specific configuration details, see [OpenShift Kubernetes clusters](#openshift-kubernetes-clusters). For OpenShift examples in air-gapped environments, see [Deploy on Air-Gapped Kubernetes](/products/wandb/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped#openshift-configuration).
#### Object storage for on-premises and S3-compatible
After provisioning your object storage bucket (see [Object storage provisioning](/products/wandb/platform/hosting/data-security/secure-storage-connector)), configure it in your W\&B Custom Resource.
**AWS S3 (on-premises)**
For on-premises AWS S3 (through Outposts or compatible storage):
```yaml theme={"system"}
bucket:
kmsKey: # Optional KMS key for encryption
name: # Example: wandb
path: "" # Keep as empty string
provider: s3
region: # Example: us-east-1
```
**S3-compatible storage such as MinIO, Ceph, or NetApp**
For S3-compatible storage systems:
```yaml theme={"system"}
bucket:
kmsKey: null
name: # Example: s3.example.com:9000
path: # Example: wandb
provider: s3
region: # Example: us-east-1
```
To enable TLS for S3-compatible storage, append `?tls=true` to the bucket path:
```yaml theme={"system"}
bucket:
path: "wandb?tls=true"
```
> [!WARNING]
>
> The certificate must be trusted. Self-signed certificates require additional configuration. See the [SSL/TLS requirements](/products/wandb/platform/hosting/self-managed/requirements#ssl-tls) for details.
**Important considerations for on-premises object storage**
When running your own object storage, consider:
1. **Storage capacity and performance**: Monitor disk capacity carefully. Average Weights & Biases usage results in tens to hundreds of gigabytes. Heavy usage can result in petabytes of storage consumption.
2. **Fault tolerance**: At minimum, use RAID arrays for physical disks. For S3-compatible storage, use distributed or highly available configurations.
3. **Availability**: Configure monitoring to ensure the storage remains available.
**MinIO considerations**
> [!WARNING]
>
> MinIO Open Source is in [maintenance mode](https://github.com/minio/minio) with no active development. Pre-compiled binaries are no longer provided, and only critical security fixes are considered case-by-case. For production deployments, CoreWeave recommends using managed object storage services or [MinIO Enterprise (AIStor)](https://min.io/product/aistor).
Enterprise alternatives for on-premises object storage include:
* [Amazon S3 on Outposts](https://aws.amazon.com/s3/outposts/)
* [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid/)
* MinIO Enterprise (AIStor)
* [Dell ObjectScale](https://www.dell.com/en-us/shop/cty/sf/objectscale)
If you are using an existing MinIO deployment or MinIO Enterprise, you can create a bucket using the MinIO client:
```bash theme={"system"}
mc config host add local http://$MINIO_HOST:$MINIO_PORT "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY" --api s3v4
mc mb --region=us-east-1 local/wandb-files
```
### Public cloud with Terraform
For full infrastructure-plus-application deployment on AWS, Google Cloud, or Azure, see [Deploy with Terraform on public cloud](#deploy-with-terraform-on-public-cloud).
## Deploy with Terraform on public cloud
> [!NOTE]
>
> CoreWeave recommends fully managed deployment options such as [W\&B Multi-tenant Cloud](/products/wandb/platform/hosting/hosting-options/multi_tenant_cloud) or [W\&B Dedicated Cloud](/products/wandb/platform/hosting/hosting-options/dedicated-cloud) deployment types. Fully managed services require little or no configuration.
Weights & Biases provides Terraform modules for deploying the platform on public cloud providers. These modules automate the provisioning of infrastructure and installation of W\&B Server, so you can stand up a complete environment without manually creating each cloud resource.
Before you start, CoreWeave recommends that you choose one of the [remote backends](https://developer.hashicorp.com/terraform/language/backend) available for Terraform to store the [State File](https://developer.hashicorp.com/terraform/language/state). The State File is the necessary resource to roll out upgrades or make changes in your deployment without recreating all components.
Select your cloud provider:
**AWS**
CoreWeave recommends using the [W\&B Server AWS Terraform Module](https://registry.terraform.io/modules/wandb/wandb/aws/latest) to deploy the platform on AWS.
The Terraform Module deploys the following mandatory components:
* Load Balancer
* AWS Identity & Access Management (IAM)
* AWS Key Management System (KMS)
* Amazon Aurora MySQL
* Amazon VPC
* Amazon S3
* Amazon Route53
* Amazon Certificate Manager (ACM)
* Amazon Elastic Load Balancing (ALB)
* Amazon Secrets Manager
Optional components include:
* Elastic Cache for Redis
* SQS
### Prerequisite permissions
The account that runs Terraform must be able to create all components listed in the preceding section and have permission to create **IAM Policies** and **IAM Roles** and assign roles to resources.
### General steps
The steps in this section are common for any deployment option.
1. Prepare the development environment.
* Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
* CoreWeave recommends creating a Git repository for version control.
2. Create the `terraform.tfvars` file.
Customize the `tvfars` file content according to the installation type. The minimum recommended content looks like the following example.
```bash theme={"system"}
namespace = "wandb"
license = "xxxxxxxxxxyyyyyyyyyyyzzzzzzz"
subdomain = "wandb-aws"
domain_name = "wandb.ml"
zone_id = "xxxxxxxxxxxxxxxx"
allowed_inbound_cidr = ["0.0.0.0/0"]
allowed_inbound_ipv6_cidr = ["::/0"]
eks_cluster_version = "1.29"
```
Define variables in your `tvfars` file before you deploy because the `namespace` variable is a string that prefixes all resources created by Terraform.
The combination of `subdomain` and `domain` forms the FQDN for your Weights & Biases instance. In the preceding example, the Weights & Biases FQDN is `wandb-aws.wandb.ml` and the DNS `zone_id` is where Terraform creates the FQDN record.
Both `allowed_inbound_cidr` and `allowed_inbound_ipv6_cidr` also require setting. In the module, this is a mandatory input. The following example permits access from any source to the Weights & Biases installation.
3. Create the file `versions.tf`.
This file contains the Terraform and Terraform provider versions required to deploy Weights & Biases in AWS:
```bash theme={"system"}
provider "aws" {
region = "eu-central-1"
default_tags {
tags = {
GithubRepo = "terraform-aws-wandb"
GithubOrg = "wandb"
Enviroment = "Example"
Example = "PublicDnsExternal"
}
}
}
```
Refer to the [Terraform Official Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#provider-configuration) to configure the AWS provider.
CoreWeave recommends that you also add the [remote backend configuration](https://developer.hashicorp.com/terraform/language/backend) mentioned at the beginning of this documentation.
4. Create the file `variables.tf`
For every option configured in the `terraform.tfvars` Terraform requires a correspondent variable declaration.
```hcl theme={"system"}
variable "namespace" {
type = string
description = "Name prefix used for resources"
}
variable "domain_name" {
type = string
description = "Domain name used to access instance."
}
variable "subdomain" {
type = string
default = null
description = "Subdomain for accessing the Weights & Biases UI."
}
variable "license" {
type = string
}
variable "zone_id" {
type = string
description = "Domain for creating the Weights & Biases subdomain on."
}
variable "allowed_inbound_cidr" {
description = "CIDRs allowed to access wandb-server."
nullable = false
type = list(string)
}
variable "allowed_inbound_ipv6_cidr" {
description = "CIDRs allowed to access wandb-server."
nullable = false
type = list(string)
}
variable "eks_cluster_version" {
description = "EKS cluster kubernetes version"
nullable = false
type = string
}
```
### Recommended deployment
This is the most straightforward deployment option configuration that creates all mandatory components and installs in the Kubernetes Cluster the latest version of Weights & Biases.
1. Create the `main.tf`
In the same directory where you created the files in the General Steps, create a file `main.tf` with the following content:
```hcl theme={"system"}
module "wandb_infra" {
source = "wandb/wandb/aws"
version = "~>7.0"
namespace = var.namespace
domain_name = var.domain_name
license = var.license
subdomain = var.subdomain
zone_id = var.zone_id
allowed_inbound_cidr = var.allowed_inbound_cidr
allowed_inbound_ipv6_cidr = var.allowed_inbound_ipv6_cidr
public_access = true
external_dns = true
kubernetes_public_access = true
kubernetes_public_access_cidrs = ["0.0.0.0/0"]
eks_cluster_version = var.eks_cluster_version
}
data "aws_eks_cluster" "eks_cluster_id" {
name = module.wandb_infra.cluster_name
}
data "aws_eks_cluster_auth" "eks_cluster_auth" {
name = module.wandb_infra.cluster_name
}
provider "kubernetes" {
host = data.aws_eks_cluster.eks_cluster_id.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_id.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.eks_cluster_auth.token
}
provider "helm" {
kubernetes {
host = data.aws_eks_cluster.eks_cluster_id.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster_id.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.eks_cluster_auth.token
}
}
output "url" {
value = module.wandb_infra.url
}
output "bucket" {
value = module.wandb_infra.bucket_name
}
```
2. Deploy Weights & Biases
To deploy Weights & Biases, execute the following commands:
```bash theme={"system"}
terraform init
terraform apply -var-file=terraform.tfvars
```
### Enable Redis
To use Redis to cache SQL queries and speed up the application response when loading metrics, add the option `create_elasticache_subnet = true` to the `main.tf` file:
```hcl theme={"system"}
module "wandb_infra" {
source = "wandb/wandb/aws"
version = "~>7.0"
namespace = var.namespace
domain_name = var.domain_name
subdomain = var.subdomain
zone_id = var.zone_id
create_elasticache_subnet = true
}
[...]
```
### Enable message broker (queue)
To enable an external message broker using SQS, add the option `use_internal_queue = false` to the `main.tf` file:
> [!NOTE]
>
> This is optional because Weights & Biases includes an embedded broker. This option doesn't bring a performance improvement.
```hcl theme={"system"}
module "wandb_infra" {
source = "wandb/wandb/aws"
version = "~>7.0"
namespace = var.namespace
domain_name = var.domain_name
subdomain = var.subdomain
zone_id = var.zone_id
use_internal_queue = false
[...]
}
```
### Additional resources
* [AWS Terraform Module documentation](https://registry.terraform.io/modules/wandb/wandb/aws/latest)
* [AWS Terraform Module source code](https://github.com/wandb/terraform-aws-wandb)
* [Migrate to operator-based AWS Terraform modules](#migrate-to-operator-based-aws-terraform-modules)
**Google Cloud**
CoreWeave recommends using the [W\&B Server Google Cloud Terraform Module](https://registry.terraform.io/modules/wandb/wandb/google/latest) to deploy the platform on Google Cloud.
The module documentation lists all available options.
Before you start, CoreWeave recommends that you choose one of the [remote backends](https://developer.hashicorp.com/terraform/language/backend/remote) available for Terraform to store the [State File](https://developer.hashicorp.com/terraform/language/state). The State File is the necessary resource to roll out upgrades or make changes in your deployment without recreating all components.
The Terraform Module deploys the following mandatory components:
* VPC
* Cloud SQL for MySQL
* Cloud Storage Bucket
* Google Kubernetes Engine
* Memorystore for Redis
* KMS Crypto Key
* Load Balancer
Optional components include:
* Pub/Sub messages system
### Prerequisite permissions
The account that runs Terraform must have the role `roles/owner` in the Google Cloud project used.
### General steps
The steps in this section are common for any deployment option.
1. Prepare the development environment.
* Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli).
* CoreWeave recommends creating a Git repository for your code, but you can keep your files locally.
* Create a project in [Google Cloud Console](https://console.cloud.google.com/).
* Authenticate with Google Cloud (make sure to [install gcloud](https://cloud.google.com/sdk/docs/install) first) using `gcloud auth application-default login`.
2. Create the `terraform.tfvars` file.
Customize the `tvfars` file content according to the installation type. The minimum recommended content looks like the following example.
```bash theme={"system"}
project_id = "wandb-project"
region = "europe-west2"
zone = "europe-west2-a"
namespace = "wandb"
license = "xxxxxxxxxxyyyyyyyyyyyzzzzzzz"
subdomain = "wandb-gcp"
domain_name = "wandb.ml"
```
You must decide the values for these variables before deployment. The `namespace` variable is a string that prefixes all resources created by Terraform.
The combination of `subdomain` and `domain` forms the FQDN that configures Weights & Biases. In the preceding example, the Weights & Biases FQDN is `wandb-gcp.wandb.ml`.
3. Create the file `variables.tf`.
For every option configured in the `terraform.tfvars` Terraform requires a correspondent variable declaration.
```hcl theme={"system"}
variable "project_id" {
type = string
description = "Project ID"
}
variable "region" {
type = string
description = "Google region"
}
variable "zone" {
type = string
description = "Google zone"
}
variable "namespace" {
type = string
description = "Namespace prefix used for resources"
}
variable "domain_name" {
type = string
description = "Domain name for accessing the Weights & Biases UI."
}
variable "subdomain" {
type = string
description = "Subdomain for access the Weights & Biases UI."
}
variable "license" {
type = string
description = "W&B License"
}
```
### Recommended deployment
This is the most straightforward deployment option configuration that creates all mandatory components and installs in the Kubernetes Cluster the latest version of Weights & Biases.
1. Create the `main.tf`
In the same directory where you created the files in the General Steps, create a file `main.tf` with the following content:
```hcl theme={"system"}
provider "google" {
project = var.project_id
region = var.region
zone = var.zone
}
provider "google-beta" {
project = var.project_id
region = var.region
zone = var.zone
}
data "google_client_config" "current" {}
provider "kubernetes" {
host = "https://${module.wandb.cluster_endpoint}"
cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
token = data.google_client_config.current.access_token
}
provider "helm" {
kubernetes {
host = "https://${module.wandb.cluster_endpoint}"
cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
token = data.google_client_config.current.access_token
}
}
# Spin up all required services
module "wandb" {
source = "wandb/wandb/google"
version = "~> 10.0"
namespace = var.namespace
license = var.license
domain_name = var.domain_name
subdomain = var.subdomain
}
# You'll want to update your DNS with the provisioned IP address
output "url" {
value = module.wandb.url
}
output "address" {
value = module.wandb.address
}
output "bucket_name" {
value = module.wandb.bucket_name
}
```
2. Deploy Weights & Biases.
To deploy Weights & Biases, execute the following commands:
```bash theme={"system"}
terraform init
terraform apply -var-file=terraform.tfvars
```
### Enable Redis
To use Redis to cache SQL queries and speed up the application response when loading metrics, add the option `create_redis = true` to the `main.tf` file:
```hcl theme={"system"}
[...]
module "wandb" {
source = "wandb/wandb/google"
version = "~> 10.0"
namespace = var.namespace
license = var.license
domain_name = var.domain_name
subdomain = var.subdomain
create_redis = true
}
[...]
```
### Enable message broker (queue)
To enable an external message broker using Pub/Sub, add the option `use_internal_queue = false` to the `main.tf` file:
> [!NOTE]
>
> This is optional because Weights & Biases includes an embedded broker. This option doesn't bring a performance improvement.
```hcl theme={"system"}
[...]
module "wandb" {
source = "wandb/wandb/google"
version = "~> 10.0"
namespace = var.namespace
license = var.license
domain_name = var.domain_name
subdomain = var.subdomain
use_internal_queue = false
}
[...]
```
### Additional resources
* [Google Cloud Terraform Module documentation](https://registry.terraform.io/modules/wandb/wandb/google/latest)
* [Google Cloud Terraform Module source code](https://github.com/wandb/terraform-google-wandb)
**Azure**
CoreWeave recommends using the [W\&B Server Azure Terraform Module](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest) to deploy the platform on Azure.
The module documentation lists all available options.
The Terraform Module deploys the following mandatory components:
* Azure Resource Group
* Azure Virtual Network (VPC)
* Azure MySQL Flexible Server
* Azure Storage Account and Blob Storage
* Azure Kubernetes Service
* Azure Application Gateway
Optional components include:
* Azure Cache for Redis
* Azure Event Grid
### Prerequisite permissions
The simplest way to configure the AzureRM provider is through the [Azure CLI](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/azure_cli). For automation, you can also use an [Azure Service Principal](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret).
Regardless of the authentication method used, the account that runs Terraform must be able to create all components listed in the preceding section.
### General steps
The steps in this section are common for any deployment option.
1. Prepare the development environment.
* Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
* CoreWeave recommends creating a Git repository for your code, but you can keep your files locally.
2. Create the `terraform.tfvars` file.
Customize the `tvfars` file content according to the installation type. The minimum recommended content looks like the following example.
```bash theme={"system"}
namespace = "wandb"
wandb_license = "xxxxxxxxxxyyyyyyyyyyyzzzzzzz"
subdomain = "wandb-azure"
domain_name = "wandb.ml"
location = "westeurope"
```
You must decide the values for these variables before deployment. The `namespace` variable is a string that prefixes all resources created by Terraform.
The combination of `subdomain` and `domain` forms the FQDN that configures Weights & Biases. In the preceding example, the Weights & Biases FQDN is `wandb-azure.wandb.ml`.
3. Create the file `versions.tf`.
This file contains the Terraform and Terraform provider versions required to deploy Weights & Biases in Azure:
```bash theme={"system"}
terraform {
required_version = "~> 1.3"
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~> 3.17"
}
}
}
```
Refer to the [Terraform Official Documentation](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs) to configure the Azure provider.
CoreWeave recommends that you also add the [remote backend configuration](https://developer.hashicorp.com/terraform/language/backend) mentioned at the beginning of this documentation.
4. Create the file `variables.tf`
For every option configured in the `terraform.tfvars` Terraform requires a correspondent variable declaration.
```bash theme={"system"}
variable "namespace" {
type = string
description = "String used for prefix resources."
}
variable "location" {
type = string
description = "Azure Resource Group location"
}
variable "domain_name" {
type = string
description = "Domain for accessing the Weights & Biases UI."
}
variable "subdomain" {
type = string
default = null
description = "Subdomain for accessing the Weights & Biases UI. Default creates record at Route53 Route."
}
variable "license" {
type = string
description = "Your wandb/local license"
}
```
### Recommended deployment
This is the most straightforward deployment option configuration that creates all mandatory components and installs in the Kubernetes Cluster the latest version of Weights & Biases.
1. Create the `main.tf`
In the same directory where you created the files in the General Steps, create a file `main.tf` with the following content:
```bash theme={"system"}
provider "azurerm" {
features {}
}
provider "kubernetes" {
host = module.wandb.cluster_host
cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
client_key = base64decode(module.wandb.cluster_client_key)
client_certificate = base64decode(module.wandb.cluster_client_certificate)
}
provider "helm" {
kubernetes {
host = module.wandb.cluster_host
cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
client_key = base64decode(module.wandb.cluster_client_key)
client_certificate = base64decode(module.wandb.cluster_client_certificate)
}
}
# Spin up all required services
module "wandb" {
source = "wandb/wandb/azurerm"
version = "~> 1.2"
namespace = var.namespace
location = var.location
license = var.license
domain_name = var.domain_name
subdomain = var.subdomain
deletion_protection = false
tags = {
"Example" : "PublicDns"
}
}
output "address" {
value = module.wandb.address
}
output "url" {
value = module.wandb.url
}
```
2. Deploy Weights & Biases
To deploy Weights & Biases, execute the following commands:
```bash theme={"system"}
terraform init
terraform apply -var-file=terraform.tfvars
```
### Enable Redis
To use Redis to cache SQL queries and speed up the application response when loading metrics, add the option `create_redis = true` to the `main.tf` file:
```bash theme={"system"}
# Spin up all required services
module "wandb" {
source = "wandb/wandb/azurerm"
version = "~> 1.2"
namespace = var.namespace
location = var.location
license = var.license
domain_name = var.domain_name
subdomain = var.subdomain
create_redis = true
[...]
}
```
### Enable message broker (queue)
To enable an external message broker using Azure Event Grid, add the option `use_internal_queue = false` to the `main.tf` file:
> [!NOTE]
>
> This is optional because Weights & Biases includes an embedded broker. This option doesn't bring a performance improvement.
```bash theme={"system"}
# Spin up all required services
module "wandb" {
source = "wandb/wandb/azurerm"
version = "~> 1.2"
namespace = var.namespace
location = var.location
license = var.license
domain_name = var.domain_name
subdomain = var.subdomain
use_internal_queue = false
[...]
}
```
### Additional resources
* [Azure Terraform Module documentation](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest)
* [Azure Terraform Module source code](https://github.com/wandb/terraform-azurerm-wandb)
### Other deployment options
You can combine multiple deployment options by adding all configurations to the same file. Each Terraform module provides several options that can be combined with the standard options and the minimal configuration found in the recommended deployment section.
Refer to the module documentation for your cloud provider for the full list of available options:
* [AWS Module documentation](https://registry.terraform.io/modules/wandb/wandb/aws/latest)
* [Google Cloud Module documentation](https://registry.terraform.io/modules/wandb/wandb/google/latest)
* [Azure Module documentation](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest)
## Access the Weights & Biases management console
The W\&B Kubernetes Operator comes with a management console where you can review deployment status, view component metrics, and adjust operator-level settings. It's available at `${HOST_URI}/console`, for example `https://wandb.company-name.com/console`.
You can log in to the management console in two ways:
**Option 1 (Recommended)**
1. Open Forge in the browser and log in. Log in to Forge with `${HOST_URI}/`, for example `https://wandb.company-name.com/`
2. Access the console. Click the icon in the top right corner and then click **System console**. Only users with admin privileges can see the **System console** entry.
**Option 2**
> [!NOTE]
>
> CoreWeave recommends you access the console using the following steps only if Option 1 doesn't work.
1. Open the console application in your browser. Open the URL described in the preceding section, which redirects you to the login screen:
2. Retrieve the password from the Kubernetes secret that the installation generates:
```shell theme={"system"}
kubectl get secret wandb-password -o jsonpath='{.data.password}' | base64 -d
```
Copy the password.
3. Log in to the console. Paste the copied password, then click **Login**.
## Update the W\&B Kubernetes Operator
This section describes how to update the W\&B Kubernetes Operator itself. Update the operator periodically so that you get bug fixes and new reconciliation features.
> [!NOTE]
>
> * Updating the W\&B Kubernetes Operator doesn't update the W\&B Server application.
> * If you use a Helm chart that doesn't use the W\&B Kubernetes Operator, see the [migration instructions](#migrate-self-managed-instances-to-wb-operator) before following the steps in this section to update the W\&B Operator.
Copy and paste the following code snippets into your terminal.
1. Update the repo with [`helm repo update`](https://helm.sh/docs/helm/helm_repo_update/):
```shell theme={"system"}
helm repo update
```
2. Update the Helm chart with [`helm upgrade`](https://helm.sh/docs/helm/helm_upgrade/):
```shell theme={"system"}
helm upgrade operator wandb/operator -n wandb-cr --reuse-values
```
## Update the W\&B Server application
You no longer need to update W\&B Server application if you use the W\&B Kubernetes Operator.
The operator automatically updates your W\&B Server application when a new version of the software of Weights & Biases is released.
## Upgrade MySQL to 8.4.x
MySQL 8.0.x is end of life. Self-Managed deployments must run a supported MySQL version that receives security patches and critical bug fixes. If you run community MySQL, install or upgrade to **MySQL 8.4.x**. If you use a managed service, run an engine version your provider documents as supported and patched (for example Amazon RDS for MySQL, Google Cloud SQL for MySQL, or Azure Database for MySQL). Weights & Biases has validated the platform against MySQL 8.4.0 and current 8.4.x releases.
These steps describe the sequence from Weights & Biases's perspective. For how to upgrade MySQL itself, including backups and version paths, follow your MySQL distribution's or cloud provider's documentation. The same sequence applies to [standard](/products/wandb/platform/hosting/self-managed/operator) and [air-gapped](/products/wandb/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped) Operator deployments. In air-gapped environments, obtain MySQL 8.4.x software through your internal distribution process before you upgrade the database.
> [!NOTE]
>
> Plan a maintenance window and notify users before you begin. Contact [Customer Support](mailto:forge-support@coreweave.com) or your Weights & Biases team if you have questions about compatibility or your deployment topology.
1. Review MySQL release notes and documentation for your target version and any intervening versions for requirements and other details.
2. Prepare for maintenance.
Before you start the upgrade, you can run the [MySQL Shell upgrade checker](https://dev.mysql.com/doc/mysql-shell/8.4/en/mysql-shell-utilities-upgrade.html) against your database to surface and fix compatibility issues for your target version. Resolve any errors or warnings in the checker output before you proceed. Refer to the documentation for your MySQL distribution.
3. Shut down MySQL and take a full backup of your MySQL database according to your MySQL distribution's documentation.
MySQL will be unavailable during the upgrade. While the database is unavailable, Weights & Biases client applications can't connect and will experience temporary errors.
4. Upgrade MySQL to 8.4.x according to your MySQL distribution's documentation.
5. Restart MySQL and verify that it is operational.
6. When MySQL is up, run `wandb verify` to validate your Weights & Biases deployment. The command runs a series of checks and reports results to `STDOUT`. If it reports problems, make adjustments and run it again. For setup and login steps, see [Verify the installation](#verify-the-installation).
7. After validation completes, users can resume normal operations.
## ClickHouse compatibility for upgrades
Self-Managed deployments that use an external ClickHouse cluster must verify ClickHouse compatibility before upgrading W\&B Server.
### Supported ClickHouse versions
W\&B Weave requires a supported version of both ClickHouse Server and ClickHouse Keeper.
* Weave supports ClickHouse 25.8 through 25.12, and 26.3 or newer.
* Weave does not work with ClickHouse 26.1 or 26.2.
***Before* you upgrade W\&B Self-Managed**, verify that both ClickHouse Server and ClickHouse Keeper run a version that Weave supports. If you must change ClickHouse versions, upgrade both components together.
If your W\&B Self-Managed deployment does not use Weave, ClickHouse is not required and this requirement does not apply to your upgrade.
Dedicated Cloud and Multi-tenant Cloud deployments already run a compatible ClickHouse version and are not affected.
For version-specific release notes, see [Supported W\&B Server releases](/release-notes/server-releases).
## Migrate self-managed instances to W\&B Operator
The following section describes how to migrate from self-managing your own W\&B Server installation to using the W\&B Operator to do this for you. Migrating lets the operator handle reconciliation and W\&B Server upgrades automatically, so you no longer have to coordinate manifest changes or Helm upgrades for the application. The migration process depends on how you installed W\&B Server:
> [!NOTE]
>
> The W\&B Operator is the default and recommended installation method for W\&B Server. Reach out to [Customer Support](mailto:forge-support@coreweave.com) or your Weights & Biases team if you have any questions.
* If you used the official W\&B Cloud Terraform Modules, navigate to the appropriate documentation and follow the steps there:
* [AWS](#migrate-to-operator-based-aws-terraform-modules)
* [Google Cloud](#migrate-to-operator-based-google-cloud-terraform-modules)
* [Azure](#migrate-to-operator-based-azure-terraform-modules)
* If you used the [W\&B Non-Operator Helm chart](https://github.com/wandb/helm-charts/tree/main/charts/wandb), see [Migrate to operator-based Helm chart](#migrate-to-operator-based-helm-chart).
* If you used the [W\&B Non-Operator Helm chart with Terraform](https://registry.terraform.io/modules/wandb/wandb/kubernetes/latest), see [Migrate to operator-based Terraform Helm chart](#migrate-to-operator-based-terraform-helm-chart).
* If you created the Kubernetes resources with manifests, see [Migrate to operator-based Helm chart](#migrate-to-operator-based-helm-chart).
### Migrate to operator-based AWS Terraform modules
For a detailed description of the migration process, see the [operator-wandb chart documentation](https://github.com/wandb/helm-charts/tree/main/charts/operator-wandb).
### Migrate to operator-based Google Cloud Terraform modules
Reach out to [Support](mailto:forge-support@coreweave.com) or your CoreWeave team if you have any questions or need assistance.
### Migrate to operator-based Azure Terraform modules
Reach out to [Support](mailto:forge-support@coreweave.com) or your CoreWeave team if you have any questions or need assistance.
### Migrate to operator-based Helm chart
Follow these steps to migrate to the operator-based Helm chart:
1. Get the current Weights & Biases configuration. If you deployed Weights & Biases with a non-operator-based version of the Helm chart, export the values like this:
```shell theme={"system"}
helm get values wandb
```
If you deployed Weights & Biases with Kubernetes manifests, export the values like this:
```shell theme={"system"}
kubectl get deployment wandb -o yaml
```
You now have all the configuration values you need for the next step.
2. Create a file called `operator.yaml`. Follow the format described in the [Configuration Reference](#configuration-reference-for-wb-operator). Use the values from step 1.
3. Scale the current deployment to 0 pods. This step stops the current deployment.
```shell theme={"system"}
kubectl scale --replicas=0 deployment wandb
```
4. Update the Helm chart repo:
```shell theme={"system"}
helm repo update
```
5. Install the new Helm chart:
```shell theme={"system"}
helm upgrade --install operator wandb/operator -n wandb-cr --create-namespace
```
6. Configure the new Helm chart and trigger Weights & Biases application deployment. Apply the new configuration.
```shell theme={"system"}
kubectl apply -f operator.yaml
```
The deployment takes a few minutes to complete.
7. Verify the installation. Make sure that everything works by following the steps in [Verify the installation](#verify-the-installation).
8. Remove the old installation. Uninstall the old Helm chart or delete the resources that you created with manifests.
### Migrate to operator-based Terraform Helm chart
Follow these steps to migrate to the operator-based Helm chart:
1. Prepare Terraform config. Replace the Terraform code from the old deployment in your Terraform config with the code described in [Deploy Weights & Biases with Helm Terraform module](#deploy-wb-with-helm-terraform-module). Set the same variables as before. Do not change the `.tfvars` file if you have one.
2. Execute Terraform run. Execute `terraform init`, `terraform plan`, and `terraform apply`.
3. Verify the installation. Make sure that everything works by following the steps in [Verify the installation](#verify-the-installation).
4. Remove the old installation. Uninstall the old Helm chart or delete the resources that you created with manifests.
## Configuration reference for W\&B Server
This section is a reference for the configuration options that you set in your `WeightsAndBiases` custom resource. Use it to look up the YAML schema for a specific subsystem (for example, MySQL, Redis, ingress, or OIDC) as you build or update your `operator.yaml` file.
This section describes the configuration options for W\&B Server application. The application receives its configuration as custom resource definition named [WeightsAndBiases](#how-it-works). Some configuration options are exposed with the following configuration. You must set others as environment variables.
The documentation has two lists of environment variables: [basic](/products/wandb/platform/hosting/env-vars) and [advanced](/products/wandb/platform/hosting/iam/advanced_env_vars). Only use environment variables if the configuration option that you need is not exposed using the Helm chart.
### Basic example
This example defines the minimum set of values required for Weights & Biases. For a more realistic production example, see [Complete example](#complete-example).
This YAML file defines the desired state of your Weights & Biases deployment, including the version, environment variables, external resources like databases, and other necessary settings.
```yaml theme={"system"}
apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
labels:
app.kubernetes.io/name: weightsandbiases
app.kubernetes.io/instance: wandb
name: wandb
namespace: default
spec:
values:
global:
host: https://
license: eyJhbGnUzaH...j9ZieKQ2x5GGfw
bucket:
mysql:
ingress:
annotations:
```
Find the full set of values in the [W\&B Helm repository](https://github.com/wandb/helm-charts/blob/main/charts/operator-wandb/values.yaml). **Change only those values you need to override**.
### Complete example
This example configuration deploys Weights & Biases to Google Cloud Anthos using Google Cloud Storage:
```yaml theme={"system"}
apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
labels:
app.kubernetes.io/name: weightsandbiases
app.kubernetes.io/instance: wandb
name: wandb
namespace: default
spec:
values:
global:
host: https://abc-wandb.sandbox-gcp.wandb.ml
bucket:
name: abc-wandb-moving-pipefish
provider: gcs
mysql:
database: wandb_local
host: 10.218.0.2
name: wandb_local
password: 8wtX6cJHizAZvYScjDzZcUarK4zZGjpV
port: 3306
user: wandb
redis:
host: redis.example.com
port: 6379
password: password
api:
enabled: true
glue:
enabled: true
executor:
enabled: true
license: eyJhbGnUzaHgyQjQyQWhEU3...ZieKQ2x5GGfw
ingress:
annotations:
ingress.gcp.kubernetes.io/pre-shared-cert: abc-wandb-cert-creative-puma
kubernetes.io/ingress.class: gce
kubernetes.io/ingress.global-static-ip-name: abc-wandb-operator-address
```
### Host
```yaml theme={"system"}
# Provide the FQDN with protocol
global:
# example host name, replace with your own
host: https://wandb.example.com
```
### Object storage (bucket)
**AWS**
```yaml theme={"system"}
global:
bucket:
provider: "s3"
name: ""
kmsKey: ""
region: ""
```
**Google Cloud**
```yaml theme={"system"}
global:
bucket:
provider: "gcs"
name: ""
```
**Azure**
```yaml theme={"system"}
global:
bucket:
provider: "az"
name: ""
secretKey: ""
```
**Other providers (Minio, Ceph, and other S3-compatible storage)**
For other S3 compatible providers, set the bucket configuration as follows:
```yaml theme={"system"}
global:
bucket:
# Example values, replace with your own
provider: s3
name: storage.example.com
kmsKey: null
path: wandb
region: default
accessKey: 5WOA500...P5DK7I
secretKey: HDKYe4Q...JAp1YyjysnX
```
For S3-compatible storage hosted outside of AWS, `kmsKey` must be `null`.
To reference `accessKey` and `secretKey` from a secret:
```yaml theme={"system"}
global:
bucket:
# Example values, replace with your own
provider: s3
name: storage.example.com
kmsKey: null
path: wandb
region: default
secret:
secretName: bucket-secret
accessKeyName: ACCESS_KEY
secretKeyName: SECRET_KEY
```
### MySQL
```yaml theme={"system"}
global:
mysql:
# Example values, replace with your own
host: db.example.com
port: 3306
database: wandb_local
user: wandb
password: 8wtX6cJH...ZcUarK4zZGjpV
```
To reference the `password` from a secret:
```yaml theme={"system"}
global:
mysql:
# Example values, replace with your own
host: db.example.com
port: 3306
database: wandb_local
user: wandb
passwordSecret:
name: database-secret
passwordKey: MYSQL_WANDB_PASSWORD
```
### License
```yaml theme={"system"}
global:
# Example license, replace with your own
license: eyJhbGnUzaHgyQjQy...VFnPS_KETXg1hi
```
To reference the `license` from a secret:
```yaml theme={"system"}
global:
licenseSecret:
name: license-secret
key: CUSTOMER_WANDB_LICENSE
```
### Ingress
See [How to identify the Kubernetes ingress class](#how-to-identify-the-kubernetes-ingress-class).
**Without TLS**
```yaml theme={"system"}
global:
# IMPORTANT: Ingress is on the same level in the YAML as 'global' (not a child)
ingress:
class: ""
```
**With TLS**
Create a secret that contains the certificate
```console theme={"system"}
kubectl create secret tls wandb-ingress-tls --key wandb-ingress-tls.key --cert wandb-ingress-tls.crt
```
Reference the secret in the ingress configuration
```yaml theme={"system"}
global:
# IMPORTANT: Ingress is on the same level in the YAML as 'global' (not a child)
ingress:
class: ""
annotations:
{}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
tls:
- secretName: wandb-ingress-tls
hosts:
-
```
For Nginx, you might have to add the following annotation:
```yaml theme={"system"}
ingress:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 0
```
### Custom Kubernetes service accounts
Specify custom Kubernetes service accounts to run the Weights & Biases pods.
The following snippet creates a service account as part of the deployment with the specified name:
```yaml theme={"system"}
app:
serviceAccount:
name: custom-service-account
create: true
parquet:
serviceAccount:
name: custom-service-account
create: true
global:
...
```
The subsystems "app" and "parquet" run under the specified service account. The other subsystems run under the default service account.
If the service account already exists on the cluster, set `create: false`:
```yaml theme={"system"}
app:
serviceAccount:
name: custom-service-account
create: false
parquet:
serviceAccount:
name: custom-service-account
create: false
global:
...
```
You can specify service accounts on different subsystems such as app, parquet, console, and others:
```yaml theme={"system"}
app:
serviceAccount:
name: custom-service-account
create: true
console:
serviceAccount:
name: custom-service-account
create: true
global:
...
```
The service accounts can be different between the subsystems:
```yaml theme={"system"}
app:
serviceAccount:
name: custom-service-account
create: false
console:
serviceAccount:
name: another-custom-service-account
create: true
global:
...
```
### External Redis
```yaml theme={"system"}
redis:
install: false
global:
redis:
host: ""
port: 6379
password: ""
parameters: {}
caCert: ""
```
To reference the `password` from a secret:
```console theme={"system"}
kubectl create secret generic redis-secret --from-literal=redis-password=supersecret
```
Reference it in the following configuration:
```yaml theme={"system"}
redis:
install: false
global:
redis:
host: redis.example
port: 9001
auth:
enabled: true
secret: redis-secret
key: redis-password
```
### LDAP
> [!WARNING]
>
> LDAP configuration support in the current Helm chart is limited. Contact Weights & Biases Support or your AISE for assistance configuring LDAP.
Configure LDAP by setting environment variables in `global.extraEnv`:
```yaml theme={"system"}
global:
extraEnv:
LDAP_ADDRESS: ldaps://ldap.company.example.com
LDAP_BASE_DN: cn=accounts,dc=company,dc=example,dc=com
LDAP_USER_BASE_DN: cn=users,cn=accounts,dc=company,dc=example,dc=com
LDAP_GROUP_BASE_DN: cn=groups,cn=accounts,dc=company,dc=example,dc=com
LDAP_BIND_DN: uid=ldapbind,cn=sysaccounts,cn=etc,dc=company,dc=example,dc=com
LDAP_BIND_PW: ********************
LDAP_ATTRIBUTES: email=mail,name=cn
LDAP_TLS_ENABLE: "true"
LDAP_LOGIN: "true"
LDAP_USER_OBJECT_CLASS: user
LDAP_GROUP_OBJECT_CLASS: group
```
This legacy approach is no longer recommended. This section is provided for reference.
**Without TLS**
```yaml theme={"system"}
global:
ldap:
enabled: true
# LDAP server address including "ldap://" or "ldaps://"
host:
# LDAP search base to use for finding users
baseDN:
# LDAP user to bind with (if not using anonymous bind)
bindDN:
# Secret name and key with LDAP password to bind with (if not using anonymous bind)
bindPW:
# LDAP attribute for email and group ID attribute names as comma separated string values.
attributes:
# LDAP group allow list
groupAllowList:
# Enable LDAP TLS
tls: false
```
**With TLS**
The LDAP TLS cert configuration requires a config map pre-created with the certificate content.
To create the config map you can use the following command:
```console theme={"system"}
kubectl create configmap ldap-tls-cert --from-file=certificate.crt
```
And use the config map in the YAML like the following example.
```yaml theme={"system"}
global:
ldap:
enabled: true
# LDAP server address including "ldap://" or "ldaps://"
host:
# LDAP search base to use for finding users
baseDN:
# LDAP user to bind with (if not using anonymous bind)
bindDN:
# Secret name and key with LDAP password to bind with (if not using anonymous bind)
bindPW:
# LDAP attribute for email and group ID attribute names as comma separated string values.
attributes:
# LDAP group allow list
groupAllowList:
# Enable LDAP TLS
tls: true
# ConfigMap name and key with CA certificate for LDAP server
tlsCert:
configMap:
name: "ldap-tls-cert"
key: "certificate.crt"
```
### OIDC SSO
```yaml theme={"system"}
global:
auth:
sessionLengthHours: 720
oidc:
clientId: ""
secret: ""
# Only include if your IdP requires it.
authMethod: ""
issuer: ""
```
`authMethod` is optional.
### SMTP
```yaml theme={"system"}
global:
email:
smtp:
host: ""
port: 587
user: ""
password: ""
```
### Environment variables
```yaml theme={"system"}
global:
extraEnv:
GLOBAL_ENV: "example"
```
### Set rate limits
For Self-Managed deployments that use the Operator, you can optionally configure [rate limits](/products/wandb/platform/hosting/self-managed/rate-limits) by setting environment variables in `spec.values.global.extraEnv`. Rate limits are off by default: if you don't set them, the instance doesn't enforce any limits. Set only the limits you want to enforce.
This example sets `GORILLA_LIMITER` and each filestream and run update limit to the value that CoreWeave recommends for typical production workloads:
```yaml theme={"system"}
spec:
values:
global:
extraEnv:
GORILLA_LIMITER: "redis://[REDIS-HOST]:6379?ttlInSeconds=900"
GORILLA_DEFAULT_RATE_LIMITS_FILESTREAM_COUNT: "500"
GORILLA_DEFAULT_RATE_LIMITS_FILESTREAM_SIZE: "120"
GORILLA_DEFAULT_RATE_LIMITS_FILESTREAM_PER_RUN_COUNT: "2"
GORILLA_DEFAULT_RATE_LIMITS_RUN_UPDATE_COUNT: "80"
```
> [!WARNING]
>
> To enforce any rate limit, you must set `GORILLA_LIMITER` to the connection string of your Redis server. If `GORILLA_LIMITER` isn't set, the instance ignores the other rate limit variables. Setting `GORILLA_LIMITER` in `global.extraEnv` works with any version of the `operator-wandb` Helm chart, and takes precedence over the value the chart sets.
Refer to the following table for more detail:
| Environment variable | Recommended value | Description |
| - | - | - |
| `GORILLA_LIMITER` | None | Connection string of the Redis server that tracks rate limit usage. Required to enforce any rate limit. |
| `GORILLA_DEFAULT_RATE_LIMITS_FILESTREAM_COUNT` | `500` | Filestream requests per second, per project. |
| `GORILLA_DEFAULT_RATE_LIMITS_FILESTREAM_SIZE` | `120` | Filestream data ingested per second, per project, in MB. |
| `GORILLA_DEFAULT_RATE_LIMITS_FILESTREAM_PER_RUN_COUNT` | `2` | Filestream requests per second, per run. |
| `GORILLA_DEFAULT_RATE_LIMITS_RUN_UPDATE_COUNT` | `80` | Run creation and metadata update requests per second, per project. |
| `GORILLA_DEFAULT_RATE_LIMITS_SDK_GRAPHQL_QUERY_SECONDS` | Not set | Database time, in seconds, that GraphQL queries from the SDK can use per second, per project. |
Each limit is measured over a 15-second window, so a project can briefly exceed the per-second value as long as its average over the window stays within the limit. A value of `0`, or leaving a variable unset, turns that limit off.
> [!WARNING]
>
> Don't set a rate limit to a negative value. A negative value can block all requests of that type instead of removing the limit.
### Custom certificate authority
`customCACerts` is a list and can take many certificates. Certificate authorities specified in `customCACerts` only apply to the W\&B Server application.
```yaml theme={"system"}
global:
customCACerts:
- |
-----BEGIN CERTIFICATE-----
MIIBnDCCAUKgAwIBAg.....................fucMwCgYIKoZIzj0EAwIwLDEQ
MA4GA1UEChMHSG9tZU.....................tZUxhYiBSb290IENBMB4XDTI0
MDQwMTA4MjgzMFoXDT.....................oNWYggsMo8O+0mWLYMAoGCCqG
SM49BAMCA0gAMEUCIQ.....................hwuJgyQRaqMI149div72V2QIg
P5GD+5I+02yEp58Cwxd5Bj2CvyQwTjTO4hiVl1Xd0M0=
-----END CERTIFICATE-----
- |
-----BEGIN CERTIFICATE-----
MIIBxTCCAWugAwIB.......................qaJcwCgYIKoZIzj0EAwIwLDEQ
MA4GA1UEChMHSG9t.......................tZUxhYiBSb290IENBMB4XDTI0
MDQwMTA4MjgzMVoX.......................UK+moK4nZYvpNpqfvz/7m5wKU
SAAwRQIhAIzXZMW4.......................E8UFqsCcILdXjAiA7iTluM0IU
aIgJYVqKxXt25blH/VyBRzvNhViesfkNUQ==
-----END CERTIFICATE-----
```
CA certificates can also be stored in a ConfigMap:
```yaml theme={"system"}
global:
caCertsConfigMap: custom-ca-certs
```
The ConfigMap must look like this:
```yaml theme={"system"}
apiVersion: v1
kind: ConfigMap
metadata:
name: custom-ca-certs
data:
ca-cert1.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
ca-cert2.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
```
> [!NOTE]
>
> If using a ConfigMap, each key in the ConfigMap must end with `.crt` (for example, `my-cert.crt` or `ca-cert1.crt`). This naming convention is required for `update-ca-certificates` to parse and add each certificate to the system CA store.
### Custom security context
Each Weights & Biases component supports custom security context configurations of the following form:
```yaml theme={"system"}
pod:
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 0
fsGroup: 1001
fsGroupChangePolicy: Always
seccompProfile:
type: RuntimeDefault
container:
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
allowPrivilegeEscalation: false
```
> [!NOTE]
>
> The only valid value for `runAsGroup:` is `0`. Any other value is an error.
For example, to configure the application pod, add a section `app` to your configuration:
```yaml theme={"system"}
global:
...
app:
pod:
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 0
fsGroup: 1001
fsGroupChangePolicy: Always
seccompProfile:
type: RuntimeDefault
container:
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
allowPrivilegeEscalation: false
```
The same concept applies to `console`, `weave`, `weave-trace`, and `parquet`.
## Configuration reference for W\&B Operator
This section describes configuration options for W\&B Kubernetes Operator (`wandb-controller-manager`). The operator receives its configuration in the form of a YAML file.
By default, the W\&B Kubernetes Operator doesn't need a configuration file. Create a configuration file if required. For example, you might need a configuration file to specify custom certificate authorities, deploy in an air gap environment, and so forth.
Find the full list of spec customization [in the Helm repository](https://github.com/wandb/helm-charts/blob/main/charts/operator/values.yaml).
### Custom CA
A custom certificate authority (`customCACerts`) is a list and can take many certificates. Those certificate authorities, when added, only apply to the W\&B Kubernetes Operator (`wandb-controller-manager`).
```yaml theme={"system"}
customCACerts:
- |
-----BEGIN CERTIFICATE-----
MIIBnDCCAUKgAwIBAg.....................fucMwCgYIKoZIzj0EAwIwLDEQ
MA4GA1UEChMHSG9tZU.....................tZUxhYiBSb290IENBMB4XDTI0
MDQwMTA4MjgzMFoXDT.....................oNWYggsMo8O+0mWLYMAoGCCqG
SM49BAMCA0gAMEUCIQ.....................hwuJgyQRaqMI149div72V2QIg
P5GD+5I+02yEp58Cwxd5Bj2CvyQwTjTO4hiVl1Xd0M0=
-----END CERTIFICATE-----
- |
-----BEGIN CERTIFICATE-----
MIIBxTCCAWugAwIB.......................qaJcwCgYIKoZIzj0EAwIwLDEQ
MA4GA1UEChMHSG9t.......................tZUxhYiBSb290IENBMB4XDTI0
MDQwMTA4MjgzMVoX.......................UK+moK4nZYvpNpqfvz/7m5wKU
SAAwRQIhAIzXZMW4.......................E8UFqsCcILdXjAiA7iTluM0IU
aIgJYVqKxXt25blH/VyBRzvNhViesfkNUQ==
-----END CERTIFICATE-----
```
CA certificates can also be stored in a ConfigMap:
```yaml theme={"system"}
caCertsConfigMap: custom-ca-certs
```
The ConfigMap must look like this:
```yaml theme={"system"}
apiVersion: v1
kind: ConfigMap
metadata:
name: custom-ca-certs
data:
ca-cert1.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
ca-cert2.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
```
> [!NOTE]
>
> Each key in the ConfigMap must end with `.crt` (for example, `my-cert.crt` or `ca-cert1.crt`). This naming convention is required for `update-ca-certificates` to parse and add each certificate to the system CA store.
## FAQ
### Purpose and role of each pod
A W\&B Server deployment includes the following pods:
* **`wandb-app`**: the core of Weights & Biases, including the GraphQL API and frontend application. It powers most of the Weights & Biases platform's functionality.
* **`wandb-console`**: the administration console, accessed through `/console`.
* **`wandb-otel`**: the OpenTelemetry agent, which collects metrics and logs from resources at the Kubernetes layer for display in the administration console.
* **`wandb-prometheus`**: the Prometheus server, which captures metrics from various components for display in the administration console.
* **`wandb-parquet`**: a backend microservice separate from the `wandb-app` pod that exports database data to object storage in Parquet format.
* **`wandb-weave`**: another backend microservice that loads query tables in the UI and supports various core app features.
* **`wandb-weave-trace`**: a framework for tracking, experimenting with, evaluating, deploying, and improving LLM-based applications. The framework is accessed through the `wandb-app` pod.
### How to get the W\&B Operator Console password
See [Access the Weights & Biases management console](#access-the-weights-&-biases-management-console).
### How to access the W\&B Operator Console if Ingress doesn't work
Execute the following command on a host that can reach the Kubernetes cluster:
```console theme={"system"}
kubectl port-forward svc/wandb-console 8082
```
Access the console in the browser with `https://localhost:8082/` console.
For how to get the password (Option 2), see [Access the Weights & Biases management console](#access-the-weights-&-biases-management-console).
### How to view W\&B Server logs
The application pod is named **wandb-app-xxx**.
```console theme={"system"}
kubectl get pods
kubectl logs wandb-XXXXX-XXXXX
```
### How to identify the Kubernetes ingress class
You can get the ingress class installed in your cluster by running
```console theme={"system"}
kubectl get ingressclass
```
# Rate limits
Source: https://docs.coreweave.com/products/wandb/platform/hosting/self-managed/rate-limits
Optional rate limits on Self-Managed instances for stability
You can configure rate limits on your W\&B Self-Managed instance to maintain stability and prevent one user or workload from affecting others. These limits are optional. If you don't set them, the instance doesn't enforce any limits.
## Default limits and notification
The following default limits help maintain platform stability:
| Limit | Default | Scope |
| - | - | - |
| Filestream requests per second | 500 | Project |
| Filestream ingestion per second | 120 MB | Project |
| Filestream requests per second | 2 | Run |
| Run creation requests per second | 80 | Project |
When a limit is exceeded, the W\&B SDK returns HTTP response `429`, and the message `HTTP 429: rate limit exceeded` appears in the SDK logs. How the SDK responds depends on which limit was exceeded:
* Filestream rate limits don't cause logging to crash or fail. When the SDK receives a `429` response on a filestream request, it backs off and retries the rate-limited request as-is, while subsequent updates accumulate.
* Run creation rate limits block further training.
```text theme={"system"}
HTTP 429: rate limit exceeded
```
CoreWeave recommends these defaults for typical production workloads. If your workloads consistently exceed these limits, you can adjust them (or leave them unset) according to your instance size and shared usage. See [Set rate limits](/products/wandb/platform/hosting/self-managed/operator#set-rate-limits). Contact [Support](mailto:forge-support@coreweave.com) if you need help sizing these limits for your deployment.
# Reference architecture
Source: https://docs.coreweave.com/products/wandb/platform/hosting/self-managed/ref-arch
Review self-managed Weights & Biases architecture covering Kubernetes, MySQL, object storage, and networking.
This page describes a reference architecture for a Weights & Biases deployment and outlines the recommended infrastructure and resources to support a production deployment of the platform. Use it as a planning guide to size, provision, and integrate the components required for a reliable Self-Managed installation.
This page is intended for platform engineers, site reliability engineers, and infrastructure administrators who deploy and operate Weights & Biases on their own infrastructure.
Depending on your chosen deployment environment for Weights & Biases, different services can help to enhance the resiliency of your deployment.
For instance, major cloud providers offer managed database services that help to reduce the complexity of database configuration, maintenance, high availability, and resilience.
This reference architecture addresses common deployment scenarios and shows how you can integrate your Weights & Biases deployment with cloud vendor services for performance and reliability.
## Before you start
Running any application in production comes with its own set of challenges, and Weights & Biases is no exception. Although Weights & Biases aims to streamline the process, complexities may arise depending on your architecture and design decisions. Typically, managing a production deployment involves overseeing components including hardware, operating systems, networking, storage, security, the Weights & Biases platform itself, and other dependencies. This responsibility extends to both the initial setup of the environment and its ongoing maintenance.
Consider carefully whether a Self-Managed approach with Weights & Biases is suitable for your team and your requirements.
A strong understanding of how to run and maintain production-grade application is an important prerequisite before you deploy Self-Managed Weights & Biases. If your team needs assistance, the CoreWeave Professional Services team and partners offer support for implementation and optimization.
To learn more about managed solutions for running Weights & Biases instead of managing it yourself, refer to [W\&B Multi-tenant Cloud](/products/wandb/platform/hosting/hosting-options/multi_tenant_cloud) and [W\&B Dedicated Cloud](/products/wandb/platform/hosting/hosting-options/dedicated-cloud).
## Infrastructure
A Weights & Biases deployment consists of an application layer and a storage layer. The following diagram shows how these layers fit together, and the subsections that follow describe each one.
### Application layer
The application layer consists of a multi-node Kubernetes cluster, with resilience against node failures. The Kubernetes cluster runs and maintains the Weights & Biases pods.
### Storage layer
The storage layer consists of a MySQL database and object storage. The MySQL database stores metadata and the object storage stores artifacts such as models and datasets.
## Infrastructure requirements
The following sections detail requirements for a Weights & Biases deployment, including Kubernetes cluster details, MySQL, Redis, object storage, software versions, networking, DNS, load balancer and ingress, SSL/TLS, and supported CPU architectures. Confirm that your environment meets each of these requirements before you begin a deployment.
### Kubernetes
Weights & Biases deploys the W\&B Server application as a [Kubernetes Operator](/products/wandb/platform/hosting/self-managed/operator) that deploys multiple pods. For this reason, Weights & Biases requires a Kubernetes cluster with:
* A fully configured and functioning ingress controller.
* The capability to provision Persistent Volumes.
Weights & Biases supports deployment on [OpenShift Kubernetes clusters](https://www.redhat.com/en/technologies/cloud-computing/openshift) in cloud, on-premises, and air-gapped environments. For specific configuration instructions, see the [OpenShift section](/products/wandb/platform/hosting/self-managed/operator#openshift-kubernetes-clusters) in the Operator guide.
### MySQL
MySQL 8.0.x reached end of life in April 2026. W\&B Self-Managed deployments must run a supported MySQL version that receives security patches and critical bug fixes. If you run community MySQL, install or upgrade to **MySQL 8.4.x**. If you use a managed service, run an engine version your provider documents as supported and patched (for example Amazon RDS for MySQL, Google Cloud SQL for MySQL, or Azure Database for MySQL). W\&B has validated the platform against MySQL 8.4.0 and current 8.4.x releases. If you are still on MySQL 8.0.x, plan an upgrade using the steps in [Upgrade MySQL to 8.4.x](/products/wandb/platform/hosting/self-managed/operator#upgrade-mysql-to-84x).
Weights & Biases stores metadata in a MySQL database. The database's performance and storage requirements depend on the shapes of the model parameters and related metadata. For example, the database grows in size as you track more training runs, and load on the database increases based on queries in run tables, user workspaces, and reports.
**Weights & Biases strongly recommends using managed database services** (such as AWS RDS Aurora MySQL, Google Cloud SQL for MySQL, or Azure Database for MySQL) for production deployments. Managed services provide automated backups, monitoring, high availability, and patching, and reduce operational complexity. See the [Cloud provider instance recommendations](#cloud-provider-instance-recommendations) section for specific service recommendations.
If you choose to deploy a self-managed MySQL database, consider the following:
* **Backups**: Periodically back up the database to a separate facility. Weights & Biases recommends daily backups with at least 1 week of retention.
* **Performance**: The database requires fast storage hardware, such as SSD or accelerated NAS.
* **Monitoring**: The database requires adequate CPU resources. Monitor the database server's CPU load. If CPU usage is sustained at > 90% of the system for more than 5 minutes, consider adding CPU capacity.
* **Availability**: To meet your availability and durability requirements, Weights & Biases recommends configuring a hot standby deployment on a separate machine. The standby streams all updates in real time from the primary deployment and is ready to fail over if the primary server crashes, becomes corrupted, or experiences sustained downtime.
#### MySQL topology
For production, a managed MySQL service is the simplest path to high availability because the cloud provider handles failover, backups, and patching. Use the provider's high availability option, for example, Aurora Multi-AZ on AWS.
If you run self-managed MySQL, use a primary database with a hot standby that receives a real-time replication stream and can take over on failure. Weights & Biases doesn't support a multi-primary topology or read-only replicas for the application database.
#### MySQL database creation
For instructions to manually create the MySQL database and user, see the [bare-metal guide MySQL database section](/products/wandb/platform/hosting/self-managed/operator#mysql-database).
#### MySQL configuration parameters
These parameters tune MySQL for the write patterns and schema changes that Weights & Biases performs at scale. If you're running your own MySQL instance, configure MySQL with these settings:
```ini theme={"system"}
binlog_format = 'ROW'
binlog_row_image = 'MINIMAL'
innodb_flush_log_at_trx_commit = 1
innodb_online_alter_log_max_size = 268435456
max_prepared_stmt_count = 1048576
sort_buffer_size = '67108864'
sync_binlog = 1
```
Weights & Biases has validated these settings for performance and reliability.
### Redis
Weights & Biases depends on a single-node Redis 7.x deployment that Weights & Biases components use for job queuing and data caching. For convenience during testing and development of proofs of concept, W\&B Self-Managed includes a local Redis deployment that isn't appropriate for production deployments.
Weights & Biases can connect to a Redis instance in the following environments:
* [AWS Elasticache](https://aws.amazon.com/elasticache/).
* [Google Cloud Memory Store](https://cloud.google.com/memorystore?hl=en).
* [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache).
* Redis deployment hosted in your cloud or on-premises infrastructure.
### Object storage
Weights & Biases requires object storage with pre-signed URL and CORS support, deployed in one of:
* [CoreWeave AI Object Storage](/products/storage/object-storage) is an S3-compatible object storage service optimized for AI workloads.
* [Amazon S3](https://aws.amazon.com/s3/) is an object storage service that provides scalability, data availability, security, and performance.
* [Google Cloud Storage](https://cloud.google.com/storage) is a managed service for storing unstructured data at scale.
* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) is a cloud-based object storage solution for storing unstructured data like text, binary data, images, videos, and logs.
* S3-compatible storage such as [MinIO Enterprise (AIStor)](https://www.min.io/product/aistor), [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid/), or other enterprise-grade solutions hosted in your cloud or on-premises infrastructure.
### Versions
| Software | Minimum version |
| - | - |
| Kubernetes | v1.34 or newer ([Supported Kubernetes versions](https://kubernetes.io/releases/patch-releases/)) |
| Helm | v3.x |
| MySQL | W\&B Self-Managed deployments must run a supported MySQL version that receives security patches and critical bug fixes. Install or upgrade to **MySQL 8.4.x**, or use a managed service version that your provider documents as supported and patched. Aurora MySQL version strings differ from community MySQL versions. Use `SELECT version()` to see the full engine version string and `SELECT aurora_version()` to see the Aurora version. [Aurora MySQL version 3](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.MySQL80.html) is compatible with MySQL 8.0.x and is still supported. See [Amazon Aurora versioning](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.VersionPolicy.Versioning.html) and your cloud provider's documentation when you choose a target version. |
| Redis | v7.x |
| ClickHouse | Required for Weave on Self-Managed deployments. Not required if your deployment does not use Weave. See [ClickHouse compatibility for upgrades](/products/wandb/platform/hosting/self-managed/operator#clickhouse-compatibility-for-upgrades) and [Supported W\&B Server releases](/release-notes/server-releases). |
### Networking
For a networked deployment, allow egress to these endpoints during both installation and runtime:
* `https://deploy.wandb.ai`
* `https://charts.wandb.ai`
* `https://quay.io` (used for Prometheus images)
> [!NOTE]
>
> Additional container registries might be required depending on your deployment configuration:
>
> * `https://gcr.io` when you deploy Bufstream and etcd for Weave online evaluations.
To learn about air-gapped deployments, refer to [Kubernetes operator for air-gapped instances](/products/wandb/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped).
Grant the training infrastructure and each experiment tracking system access to W\&B and the object storage.
### DNS
The fully qualified domain name (FQDN) of the Weights & Biases deployment must resolve to the IP address of the ingress or load balancer using an `A` record.
### Load balancer and ingress
The W\&B Kubernetes Operator can expose services using a Kubernetes ingress controller, which routes to service endpoints based on URL paths with different ports. The ingress controller must be accessible by all machines that execute machine learning payloads or access the service through web browsers.
#### Ingress controller requirements
Your Kubernetes cluster must have an `IngressClass` available. Common ingress controller options include:
* [Nginx Ingress Controller](https://kubernetes.github.io/ingress-nginx/).
* [Istio](https://istio.io).
* [Traefik](https://traefik.io/).
* Cloud provider ingress controllers (AWS ALB, GCP Ingress, and Azure Application Gateway).
#### Weights & Biases service routing
The W\&B Operator routes requests automatically to multiple backend services based on path:
| Path | Service | Default port | Purpose |
| - | - | - | - |
| `/` | `wandb-app` | 8080 | Main web application UI |
| `/api` | `wandb-api` | 8081 | API service |
| `/graphql` | `wandb-api` | 8081 | GraphQL API endpoint |
| `/graphql2` | `wandb-api` | 8081 | GraphQL API v2 endpoint |
| `/console` | `wandb-console` | 8082 | System Console |
| `/traces` | `wandb-weave-trace` | 8722 | Weave tracing service (if enabled) |
#### Example ingress configuration
The following shows an example ingress resource created by the W\&B Operator:
```yaml theme={"system"}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: wandb
namespace: wandb
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
spec:
ingressClassName: nginx
rules:
- host: wandb.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: wandb-app
port:
number: 8080
- path: /api
pathType: Prefix
backend:
service:
name: wandb-api
port:
number: 8081
- path: /graphql
pathType: Prefix
backend:
service:
name: wandb-api
port:
number: 8081
- path: /graphql2
pathType: Prefix
backend:
service:
name: wandb-api
port:
number: 8081
- path: /console
pathType: Prefix
backend:
service:
name: wandb-console
port:
number: 8082
tls:
- hosts:
- wandb.example.com
secretName: wandb-tls
```
> [!NOTE]
>
> The W\&B Operator creates and manages the ingress configuration automatically. You typically don't need to create ingress resources manually. Make sure your cluster has a functioning ingress controller and the appropriate `IngressClass` configured.
### SSL/TLS
W\&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress or load balancer. The W\&B Server application doesn't terminate SSL or TLS connections.
> [!WARNING]
>
> W\&B doesn't support self-signed certificates or custom CAs. Self-signed certificates cause challenges for users and aren't supported.
If possible, use a service like [Let's Encrypt](https://letsencrypt.org) to provide trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you.
If your security policies require SSL communication within your trusted networks, consider using a tool like Istio and [sidecar containers](https://istio.io/latest/docs/reference/config/networking/sidecar/).
### Supported CPU architectures
Weights & Biases runs on Intel and AMD 64-bit architecture. ARM isn't supported.
## Deployment method
After your infrastructure meets the preceding requirements, choose how to install Weights & Biases and provision the underlying resources. The following sections describe the recommended deployment method and the recommended approach for infrastructure provisioning.
### W\&B Kubernetes Operator with Helm
The recommended installation method for W\&B Self-Managed uses the **W\&B Kubernetes Operator**, deployed through Helm. This approach provides:
* Automated updates and management of Weights & Biases components.
* Simplified configuration and deployment.
* Support for all deployment scenarios (cloud, on-premises, and air-gapped).
For detailed installation instructions, see:
* [Deploy W\&B Platform On-premises](/products/wandb/platform/hosting/self-managed/operator) - Primary installation guide.
* [Kubernetes operator for air-gapped instances](/products/wandb/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped) - For disconnected environments.
### Infrastructure provisioning
Terraform is the recommended way to provision infrastructure for Weights & Biases production deployments. With Terraform, you define the required resources, their references to other resources, and their dependencies. Weights & Biases provides Terraform modules for the major cloud providers. For details, refer to [Deploy W\&B Server within Self-Managed cloud accounts](/products/wandb/platform/hosting/hosting-options/self-managed#deploy-wb-server-within-Self-Managed-cloud-accounts).
## Sizing
Use the following guidelines as a starting point when planning a deployment. Weights & Biases recommends that you monitor all components of a deployment closely and that you make adjustments based on observed usage patterns. Continue to monitor production deployments over time and make adjustments as needed to maintain performance.
When you plan capacity, you size two core components: a Kubernetes cluster for the W\&B Operator workload and a MySQL database for metadata. Recommendations vary by **environment** (Test/Dev or Production) and, for Kubernetes only, by **product mix** (Models only, Weave only, or Models and Weave). Weights & Biases recommends starting with a minimum of 3 worker nodes for both Test/Dev and Production, and enabling cluster autoscaling in Production.
The following sections give per-node sizing recommendations for the Kubernetes cluster and the MySQL database.
### Kubernetes sizing
**Models only**
| Environment | CPU | Memory | Disk |
| - | - | - | - |
| Test/Dev | 2 cores | 16 GB | 100 GB |
| Production | 8 cores | 64 GB | 100 GB |
Numbers are per Kubernetes worker node.
**Weave only**
| Environment | CPU | Memory | Disk |
| - | - | - | - |
| Test/Dev | 4 cores | 32 GB | 100 GB |
| Production | 12 cores | 96 GB | 100 GB |
Numbers are per Kubernetes worker node.
**Models and Weave**
| Environment | CPU | Memory | Disk |
| - | - | - | - |
| Test/Dev | 4 cores | 32 GB | 100 GB |
| Production | 16 cores | 128 GB | 100 GB |
Numbers are per Kubernetes worker node.
### MySQL sizing
These recommendations don't vary by product mix. For topology and availability guidance, see [MySQL topology](#mysql-topology) under [MySQL](#mysql).
| Environment | CPU | Memory | Disk |
| - | - | - | - |
| Test/Dev | 2 cores | 16 GB | 100 GB |
| Production | 8 cores | 64 GB | 500 GB |
Numbers are per MySQL node.
#### MySQL sizing for run width
Database load also depends on *run width*, the number of distinct metrics that a single run logs at each step. For example, a run that logs 50,000 distinct metrics per step has a run width of 50K. Wide runs increase CPU load on the database, especially when the UI loads them in workspaces and reports.
If your teams log wide runs, size each MySQL node for the widest runs you expect:
| Run width (metrics per step) | CPU | Memory | Example AWS instance class |
| - | - | - | - |
| Up to 1K | 2 cores | 16 GB | `db.r6g.large` |
| Up to 25K | 4 cores | 32 GB | `db.r6g.xlarge` |
| Up to 50K | 8 cores | 64 GB | `db.r6g.2xlarge` |
| Up to 75K | 16 cores | 128 GB | `db.r6g.4xlarge` |
| Up to 100K | 32 cores | 256 GB | `db.r6g.8xlarge` |
The Production recommendation in the [MySQL sizing](#mysql-sizing) table supports run widths up to 50K.
## Cloud provider instance recommendations
After you determine the per-node CPU, memory, and disk requirements from the preceding sizing tables, use the following recommendations to pick specific cloud provider instance types and managed services that meet those requirements. These recommendations apply to each node of a Self-Managed deployment of Weights & Biases in cloud infrastructure.
**AWS**
**Recommended managed services**
* **Kubernetes**: Amazon EKS
* **MySQL**: Amazon RDS Aurora
* **Object storage**: Amazon S3
| Environment | K8s (Models only) | K8s (Weave only) | K8s (Models\&Weave) | MySQL |
| - | - | - | - | - |
| Test/Dev | r6i.large | r6i.xlarge | r6i.xlarge | db.r6g.large |
| Production | r6i.2xlarge | r6i.4xlarge | r6i.4xlarge | db.r6g.2xlarge |
**Google Cloud**
**Recommended managed services**
* **Kubernetes**: Google Kubernetes Engine (GKE)
* **MySQL**: Google Cloud SQL for MySQL
* **Object storage**: Google Cloud Storage (GCS)
| Environment | K8s (Models only) | K8s (Weave only) | K8s (Models\&Weave) | MySQL |
| - | - | - | - | - |
| Test/Dev | n2-highmem-2 | n2-highmem-4 | n2-highmem-4 | db-n1-highmem-2 |
| Production | n2-highmem-8 | n2-highmem-16 | n2-highmem-16 | db-n1-highmem-8 |
**Azure**
**Recommended managed services**
* **Kubernetes**: Azure Kubernetes Service (AKS)
* **MySQL**: Azure Database for MySQL
* **Object storage**: Azure Blob Storage
| Environment | K8s (Models only) | K8s (Weave only) | K8s (Models\&Weave) | MySQL |
| - | - | - | - | - |
| Test/Dev | Standard\_E2\_v5 | Standard\_E4\_v5 | Standard\_E4\_v5 | MO\_Standard\_E2ds\_v4 |
| Production | Standard\_E8\_v5 | Standard\_E16\_v5 | Standard\_E16\_v5 | MO\_Standard\_E8ds\_v4 |
# Self-Managed infrastructure requirements
Source: https://docs.coreweave.com/products/wandb/platform/hosting/self-managed/requirements
Infrastructure and software requirements for W&B Self-Managed deployments
This page describes the infrastructure and software requirements for deploying W\&B Self-Managed. It's intended for platform and infrastructure engineers planning a Self-Managed installation. Review these requirements before you begin your deployment to confirm that your environment can support W\&B Server.
> [!NOTE]
>
> Weights & Biases recommends fully managed deployment options such as [W\&B Multi-tenant Cloud](/products/wandb/platform/hosting/hosting-options/multi_tenant_cloud) or [W\&B Dedicated Cloud](/products/wandb/platform/hosting/hosting-options/dedicated-cloud) deployment types. Weights & Biases fully managed services are straightforward and secure to use, and require minimal to no configuration.
For complete architectural guidance, see the [reference architecture](/products/wandb/platform/hosting/self-managed/ref-arch).
## Software version requirements
| Software | Minimum version |
| - | - |
| Kubernetes | v1.34 or newer ([Supported Kubernetes versions](https://kubernetes.io/releases/patch-releases/)) |
| Helm | v3.x |
| MySQL | W\&B Self-Managed deployments must run a supported MySQL version that receives security patches and critical bug fixes. Install or upgrade to **MySQL 8.4.x**, or use a managed service version that your provider documents as supported and patched. Aurora MySQL version strings differ from community MySQL versions. Use `SELECT version()` to see the full engine version string and `SELECT aurora_version()` to see the Aurora version. [Aurora MySQL version 3](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.MySQL80.html) is compatible with MySQL 8.0.x and is still supported. See [Amazon Aurora versioning](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.VersionPolicy.Versioning.html) and your cloud provider's documentation when you choose a target version. |
| Redis | v7.x |
| ClickHouse | Required for Weave on Self-Managed deployments. Not required if your deployment does not use Weave. See [ClickHouse compatibility for upgrades](/products/wandb/platform/hosting/self-managed/operator#clickhouse-compatibility-for-upgrades) and [Supported W\&B Server releases](/release-notes/server-releases). |
## Hardware requirements
**CPU Architecture**: W\&B runs on Intel (x86) CPU architecture only. ARM is not supported.
**Sizing**: For CPU, memory, and disk sizing recommendations for Kubernetes nodes and MySQL, see the [Sizing section](/products/wandb/platform/hosting/self-managed/ref-arch#sizing) in the reference architecture. Requirements vary based on whether you're running Models, Weave, or both.
For detailed sizing recommendations based on your use case (Models only, Weave only, or both), see the [reference architecture sizing section](/products/wandb/platform/hosting/self-managed/ref-arch#sizing).
## Kubernetes
W\&B Server is deployed as a [Kubernetes Operator](/products/wandb/platform/hosting/self-managed/operator) that manages multiple pods. Your Kubernetes cluster must meet the following requirements:
* **Version**: See the preceding [Software version requirements](#software-version-requirements) section.
* **Ingress controller**: A fully configured and functioning ingress controller (Nginx, Istio, Traefik, or cloud provider ingress).
* **Persistent volumes**: Capability to provision persistent volumes.
* **CPU architecture**: Intel or AMD 64-bit (ARM is not supported).
Weights & Biases supports deployment on [OpenShift Kubernetes clusters](https://www.redhat.com/en/technologies/cloud-computing/openshift) in cloud, on-premises, and air-gapped environments. For specific configuration instructions, see the [OpenShift section](/products/wandb/platform/hosting/self-managed/operator#openshift-kubernetes-clusters) in the Operator guide.
For complete Kubernetes requirements, including load balancer and ingress configuration, see the [reference architecture Kubernetes section](/products/wandb/platform/hosting/self-managed/ref-arch#kubernetes).
## MySQL database
MySQL 8.0.x reached end of life in April 2026. W\&B Self-Managed deployments must run a supported MySQL version that receives security patches and critical bug fixes. If you run community MySQL, install or upgrade to **MySQL 8.4.x**. If you use a managed service, run an engine version your provider documents as supported and patched (for example Amazon RDS for MySQL, Google Cloud SQL for MySQL, or Azure Database for MySQL). W\&B has validated the platform against MySQL 8.4.0 and current 8.4.x releases. If you are still on MySQL 8.0.x, plan an upgrade using the steps in [Upgrade MySQL to 8.4.x](/products/wandb/platform/hosting/self-managed/operator#upgrade-mysql-to-84x).
W\&B requires an external MySQL database.
For production, W\&B recommends using managed database services:
* [AWS RDS Aurora MySQL](https://aws.amazon.com/rds/aurora/)
* [Google Cloud SQL for MySQL](https://cloud.google.com/sql/mysql)
* [Azure Database for MySQL](https://azure.microsoft.com/en-us/products/mysql/)
Managed database services provide automated backups, monitoring, high availability, patching, and reduce operational overhead.
See the [reference architecture](/products/wandb/platform/hosting/self-managed/ref-arch#mysql) for MySQL requirements, including sizing recommendations and configuration parameters. For the SQL to create the database, see the [bare-metal guide](/products/wandb/platform/hosting/self-managed/operator#mysql-database). For questions about your deployment's database configuration, contact [support](mailto:forge-support@coreweave.com) or your AISE.
Weights & Biases recommends using managed database services such as AWS RDS Aurora MySQL, Google Cloud SQL for MySQL, or Azure Database for MySQL for production deployments. Managed services provide automated backups, monitoring, high availability, and patching, and they reduce operational complexity.
### MySQL configuration parameters
If you're running your own MySQL instance, configure MySQL with the following settings for compatibility with W\&B Server:
```ini theme={"system"}
binlog_format = 'ROW'
binlog_row_image = 'MINIMAL'
innodb_flush_log_at_trx_commit = 1
innodb_online_alter_log_max_size = 268435456
max_prepared_stmt_count = 1048576
sort_buffer_size = '67108864'
sync_binlog = 1
```
Weights & Biases has validated these settings for performance and reliability.
### Database creation
If you aren't using a managed MySQL service that provisions the database automatically, follow these instructions to manually create the MySQL database and user that W\&B Server uses:
Create a database and a user with the following SQL commands. Replace `[PASSWORD]` with a secure password:
```sql theme={"system"}
CREATE USER 'wandb_local'@'%' IDENTIFIED BY '[PASSWORD]';
CREATE DATABASE wandb_local CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
GRANT ALL ON wandb_local.* TO 'wandb_local'@'%' WITH GRANT OPTION;
```
For additional considerations, including backups, performance, monitoring, and availability, see the [reference architecture MySQL section](/products/wandb/platform/hosting/self-managed/ref-arch#mysql).
## Redis
W\&B Server uses Redis for caching and background job coordination.
W\&B depends on a single-node Redis 7.x deployment, which W\&B's components use for job queuing and data caching. For testing and proof-of-concept work, W\&B Self-Managed includes a local Redis deployment. This bundled deployment isn't appropriate for production use.
For production deployments, W\&B can connect to a Redis instance in the following environments:
* [Amazon ElastiCache](https://aws.amazon.com/elasticache/)
* [Google Cloud Memorystore](https://cloud.google.com/memorystore?hl=en)
* [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache)
* Self-hosted Redis in your cloud or on-premises infrastructure
Weights & Biases can connect to a Redis instance in any of the following environments:
* [AWS Elasticache](https://aws.amazon.com/pm/elasticache/).
* [Google Cloud Memory Store](https://cloud.google.com/memorystore?hl=en).
* [Azure Cache for Redis](https://azure.microsoft.com/en-us/products/cache).
* Redis deployment hosted in your cloud or on-premises infrastructure.
## ClickHouse
W\&B Weave on Self-Managed deployments uses ClickHouse. ClickHouse is not required for every Self-Managed installation.
Before you upgrade W\&B Server or change your ClickHouse version, see [ClickHouse compatibility for upgrades](/products/wandb/platform/hosting/self-managed/operator#clickhouse-compatibility-for-upgrades) in the Operator guide and the compatibility notes in [Supported W\&B Server releases](/release-notes/server-releases).
For deployment procedures, see [Deploy Weave in a self-managed environment](/products/wandb/weave/guides/platform/weave-self-managed).
## Object storage
W\&B Server requires an object storage bucket to store artifacts, media, and run data.
W\&B requires object storage with pre-signed URL and CORS support.
W\&B recommends the following storage providers:
* [Amazon S3](https://aws.amazon.com/s3/): Object storage service offering scalability, data availability, security, and performance.
* [Google Cloud Storage](https://cloud.google.com/storage): Managed service for storing unstructured data at scale.
* [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs): Cloud-based object storage for unstructured data at scale.
* [CoreWeave AI Object Storage](/products/storage/object-storage): S3-compatible object storage optimized for AI workloads.
* Enterprise S3-compatible storage such as [MinIO Enterprise (AIStor)](https://www.min.io/product/aistor), [NetApp StorageGRID](https://www.netapp.com/data-storage/storagegrid/), or other enterprise solutions.
> [!NOTE]
>
> MinIO Open Source is in [maintenance mode](https://github.com/minio/minio) with no active development or pre-compiled binaries. For production deployments, W\&B recommends managed object storage services or enterprise S3-compatible solutions such as MinIO Enterprise (AIStor).
After you select a provider, configure the bucket so that W\&B can access it. For detailed bucket provisioning instructions including IAM policies, CORS configuration, and access setup, see the [Bring Your Own Bucket (BYOB) guide](/products/wandb/platform/hosting/data-security/secure-storage-connector).
For the complete list of object storage requirements, including capacity and performance guidance, see the [reference architecture object storage section](/products/wandb/platform/hosting/self-managed/ref-arch#object-storage).
### Provision your storage bucket
Before configuring W\&B, you must provision your object storage bucket with the required IAM policies, CORS configuration, and access credentials.
See the [Bring Your Own Bucket (BYOB) guide](/products/wandb/platform/hosting/data-security/secure-storage-connector) for detailed step-by-step provisioning instructions for:
* Amazon S3 (including IAM policies and bucket policies)
* Google Cloud Storage (including PubSub notifications)
* Azure Blob Storage (including managed identities)
* CoreWeave AI Object Storage
* S3-compatible storage (MinIO Enterprise, NetApp StorageGRID, and other enterprise solutions)
### Configure Weights & Biases to use your bucket
After you provision your bucket, you must configure Weights & Biases to use it through the Operator's Helm values so that W\&B Server can read from and write to the bucket. See the [Operator object storage configuration section](/products/wandb/platform/hosting/self-managed/operator#object-storage-bucket) for details.
## Networking
Networking configuration exposes W\&B Server to users and machine-learning workloads. The following sections describe DNS, load balancer, and ingress requirements.
For a networked deployment, allow egress to these endpoints during both installation and runtime:
* `https://deploy.wandb.ai`
* `https://charts.wandb.ai`
* `https://quay.io` (used for Prometheus images)
> [!NOTE]
>
> Additional container registries might be required depending on your deployment configuration:
>
> * `https://gcr.io` when you deploy Bufstream and etcd for Weave online evaluations.
To learn about air-gapped deployments, refer to [Kubernetes operator for air-gapped instances](/products/wandb/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped).
Grant the training infrastructure and each experiment tracking system access to W\&B and the object storage.
### DNS
The fully qualified domain name (FQDN) of the Weights & Biases deployment must resolve to the IP address of the ingress or load balancer using an A record.
### Load balancer and ingress
The W\&B Kubernetes Operator exposes services using a Kubernetes ingress controller, which routes to service endpoints based on URL paths. The ingress controller must be accessible by all machines that execute machine-learning payloads or access the service through web browsers.
For detailed load balancer options, ingress controller requirements, and configuration examples, see the [reference architecture load balancer section](/products/wandb/platform/hosting/self-managed/ref-arch#load-balancer-and-ingress).
## SSL/TLS
W\&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress or load balancer. The W\&B Server application doesn't terminate SSL or TLS connections.
> [!WARNING]
>
> W\&B doesn't support self-signed certificates or custom CAs. Self-signed certificates cause challenges for users and aren't supported.
If possible, use a service like [Let's Encrypt](https://letsencrypt.org) to provide trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you.
If your security policies require SSL communication within your trusted networks, consider using a tool like Istio and [sidecar containers](https://istio.io/latest/docs/reference/config/networking/sidecar/).
## License
All Self-Managed deployments require a valid W\&B Server license. Without a license, W\&B Server can't start.
You need a W\&B license to deploy W\&B Self-Managed. This procedure shows you how to obtain one.
1. If you don't already have a W\&B account, create one.
2. If you need an enterprise trial license, [submit a request](https://wandb.ai/site/for-enterprise/self-hosted-trial) or contact your W\&B team. An enterprise trial license includes support for important security and other enterprise-friendly capabilities.
3. Otherwise, open the [Deploy Manager](https://deploy.wandb.ai/deploy) to generate a free trial license. The URL redirects you to a **Get a License for W\&B Local** form. Provide the following information:
* The owner of the license.
* The deployment type.
* A name and optional description for the instance.
4. Click **Generate License Key**.
A page appears with an overview of your deployment and the associated license.
## Next steps
After you confirm that your infrastructure meets these requirements, proceed to the deployment guide that matches your environment:
* **Cloud and on-premises deployments**: See [Deploy Weights & Biases with Kubernetes Operator](/products/wandb/platform/hosting/self-managed/operator) for Helm and Terraform deployment options.
* **Air-gapped deployments**: See [Deploy on Air-Gapped Kubernetes](/products/wandb/platform/hosting/self-managed/on-premises-deployments/kubernetes-airgapped) for disconnected environments.
* **All deployment methods**: See [Deploy with Kubernetes Operator](/products/wandb/platform/hosting/self-managed/operator) for the core operator deployment guide.
# Update Weights & Biases license and version
Source: https://docs.coreweave.com/products/wandb/platform/hosting/server-upgrade-process
Guide for updating Weights & Biases version and license across different installation methods.
This guide shows W\&B Server administrators how to update the W\&B Server version and license key for an existing self-managed deployment. Keeping your server and license current ensures access to the latest features, fixes, and continued entitlement to use W\&B Server.
Update your W\&B Server version and license with the same method you used to install W\&B Server. The following table lists how to update your license and version based on different deployment methods.
| Release type | Description |
| - | - |
| [Terraform](#update-with-terraform) | Weights & Biases supports three public Terraform modules for cloud deployment: [AWS](https://registry.terraform.io/modules/wandb/wandb/aws/latest), [Google Cloud](https://registry.terraform.io/modules/wandb/wandb/google/latest), and [Azure](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest). |
| [Helm](#update-with-helm) | Use the [Helm Chart](https://github.com/wandb/helm-charts) to install Weights & Biases into an existing Kubernetes cluster. |
## Update with Terraform
If you deployed W\&B Server with one of the Weights & Biases-maintained Terraform modules, use Terraform to update both your license key and the Weights & Biases version in place. The following table lists Weights & Biases-managed Terraform modules by cloud platform.
| Cloud provider | Terraform module |
| - | - |
| AWS | [AWS Terraform module](https://registry.terraform.io/modules/wandb/wandb/aws/latest) |
| Google Cloud | [Google Cloud Terraform module](https://registry.terraform.io/modules/wandb/wandb/google/latest) |
| Azure | [Azure Terraform module](https://registry.terraform.io/modules/wandb/wandb/azurerm/latest) |
1. Navigate to the Weights & Biases-maintained Terraform module for your cloud provider. See the preceding table to find the Terraform module that matches your cloud provider.
2. In your Terraform configuration, update `wandb_version` and `license` in your Terraform `wandb_app` module configuration:
```hcl theme={"system"}
module "wandb_app" {
source = "wandb/wandb/[CLOUD-SPECIFIC-MODULE]"
version = "new_version"
license = "new_license_key" # Your new license key
wandb_version = "new_wandb_version" # Desired W&B version
...
}
```
3. Apply the Terraform configuration with `terraform plan` and `terraform apply`.
```bash theme={"system"}
terraform init
terraform apply
```
4. Optional: If you use a `terraform.tfvars` or other `.tfvars` file, update or create one with the new Weights & Biases version and license key.
```bash theme={"system"}
terraform plan -var-file="terraform.tfvars"
```
From your Terraform workspace directory, apply the configuration:
```bash theme={"system"}
terraform apply -var-file="terraform.tfvars"
```
After Terraform applies the change, your deployment runs the specified Weights & Biases version and uses the updated license key.
## Update with Helm
> [!WARNING]
>
> The `wandb` Helm chart is deprecated and no longer supported. It deployed a single pod and has been replaced by the [W\&B Kubernetes Operator](/products/wandb/platform/hosting/self-managed/operator). If you're still using this chart, follow the [migration guide](/products/wandb/platform/hosting/self-managed/operator#migrate-self-managed-instances-to-wb-operator) to move to the operator.
Two Helm-based update paths are available: update from your existing Helm values file, or set the new license and image tag directly on the upgrade command. The following sections describe each approach.
### Update Weights & Biases with spec
Use this approach when you manage your Helm configuration in a tracked `*.yaml` values file.
1. Specify a new version by modifying the `image.tag` or `license` values, or both, in your Helm chart `*.yaml` configuration file:
```yaml theme={"system"}
license: 'new_license'
image:
repository: wandb/local
tag: 'new_version'
```
2. Update the Helm repository and upgrade the Weights & Biases release using your values file:
```bash theme={"system"}
helm repo update
helm upgrade --namespace=wandb --create-namespace \
--install wandb wandb/wandb --version ${chart_version} \
-f ${wandb_install_spec.yaml}
```
### Update license and version directly
Use this approach to update the license and image tag without editing a values file, and reuse your existing Helm release configuration.
1. Set the new license key and image tag as environment variables:
```bash theme={"system"}
export LICENSE='new_license'
export TAG='new_version'
```
2. Upgrade your Helm release, merging the new values with the existing configuration:
```bash theme={"system"}
helm repo update
helm upgrade --namespace=wandb --create-namespace \
--install wandb wandb/wandb --version ${chart_version} \
--reuse-values --set license=$LICENSE --set image.tag=$TAG
```
For more information, see the [upgrade guide](https://github.com/wandb/helm-charts/blob/main/upgrade) in the public repository.
## Update with admin UI
Use the admin UI to rotate your license key from within the W\&B App, without changing your deployment configuration. This method only works for updating licenses that aren't set with an environment variable in the Weights & Biases server container, typically in self-managed Docker installations. This method updates the license only. It doesn't change the running W\&B Server version.
1. Obtain a new license from the [Weights & Biases Deployment Page](https://deploy.wandb.ai/), ensuring it matches the correct organization and deployment ID for the deployment you want to upgrade.
2. Access the **License** page in the W\&B App. Click **Settings** > **License** or browse to `HOST_URL/console/settings/license`, where `HOST_URL` is your W\&B Server host URL.
3. Navigate to the license management section.
4. Enter the new license key and save your changes.
# Manage secrets
Source: https://docs.coreweave.com/products/wandb/platform/secrets
Overview of Forge team secrets, how they work, and how to get started using them.
Secret Manager lets you securely and centrally store, manage, and inject *secrets*, which are sensitive strings such as access tokens, bearer tokens, API keys, or passwords. Forge features can read team secret values, which removes the need to paste them or store them in code, training scripts, or plain-text automation configuration. This page is for Forge admins who need to create, rotate, delete, or manage access to team secrets used by webhook automations, Weave Playground, and sandboxes.
Secrets live in each Forge team's Secret Manager, in the **Team secrets** section of the [team settings](/products/wandb/platform/app/settings-page/teams).
> [!NOTE]
>
> * Only Forge admins can create, edit, or delete a secret.
> * Secrets are available by default on Multi-tenant Cloud. On [W\&B Self-Managed deployments](/products/wandb/platform/hosting) that you host in Azure, Google Cloud, or AWS, the **Team secrets** section appears only when the deployment enables the secret store (`ENABLE_SECRET_STORE`). Connect with your CoreWeave account team to discuss how you can use secrets in Forge if you use a different deployment type.
> * In Self-Managed deployments, you are responsible for configuring security measures that satisfy your security needs.
>
> * CoreWeave recommends that you store your Forge deployment's secrets in a secrets manager from AWS, Google Cloud, or Azure. These services include advanced security capabilities.
>
> * CoreWeave recommends against using a Kubernetes cluster as the backend of your secrets store. Use a cluster only if you can't use a cloud secrets manager (AWS, Google Cloud, or Azure) and you understand how to prevent the security vulnerabilities that can occur.
## Where team secrets are used
You can use Forge team secrets in multiple contexts. After you [add a secret](#add-a-secret), an automation can access the secret by name.
* **Webhook automations**: When an automation sends an HTTP request to a [webhook](/products/wandb/automations/create-automations/webhook), you can attach team secrets for authentication headers and for values referenced in the payload. You can scope automations to a [project](/products/wandb/automations/automation-events#project) or a [Registry](/products/wandb/automations/automation-events#registry). Registry-scoped automations that call a webhook use the same team webhooks and team secrets as project-scoped webhook automations.
* **Weave Playground**: Supply provider credentials as named team secrets. See [Add provider credentials and information](/products/wandb/weave/guides/tools/playground#add-provider-credentials-and-information).
* **Sandboxes**: Securely provide team secrets to your sandboxes to make them available as environment variables. See [Secrets in sandboxes](/products/sandboxes/secrets).
## Add a secret
Add a secret when you want to make a sensitive value available to Forge features without exposing it in code or configuration. After you complete these steps, the secret is available by name to the team features described in [Where team secrets are used](#where-team-secrets-are-used).
To add a secret:
1. If an external service gives you a token or API key, obtain that value through that service's normal flow. If necessary, save the sensitive string securely, such as in a password manager, before you paste it into Secret Manager. Saving a backup matters because, after creation, Forge no longer reveals the secret's value.
2. Sign in to Forge and go to the team's **Settings** page.
3. In the **Team secrets** section, click **New secret**.
4. Provide a name for the secret, using letters, numbers, and underscores (`_`). The name must start with a letter or an underscore.
5. Paste the sensitive string into the **Value** field.
6. Click **Add secret**.
When you configure a webhook for an automation, select which team secrets the webhook may use. For field names, access tokens, and payload variables, see [Create a webhook automation](/products/wandb/automations/create-automations/webhook).
> [!NOTE]
>
> After you create a secret, you can access that secret in a [webhook automation's payload](/products/wandb/automations/create-automations/webhook) using the format `${SECRET_NAME}`.
## Rotate a secret
Rotate a secret when its value changes (for example, when an upstream credential is regenerated or when you suspect the existing value has been compromised). Because Forge doesn't reveal a secret's current value after creation, rotation is also the way to replace a value you no longer have a copy of.
To rotate a secret and update its value:
1. Click the pencil icon in the secret's row to open the secret's details.
2. Set **New value** to the new value. Optionally, click **Reveal secret value** to verify the new value.
3. Click **Replace secret**. The secret's value updates and no longer resolves to the previous value.
> [!NOTE]
>
> After a secret is created or updated, you can no longer reveal its current value. Instead, rotate the secret to a new value.
Rotating or replacing a secret can affect every feature that still expects the old value. Before you rely on the new value everywhere, update [webhook automations](/products/wandb/automations/create-automations/webhook), [sandboxes](/products/sandboxes/secrets) that inject the secret, [Weave Playground](/products/wandb/weave/guides/tools/playground#add-provider-credentials-and-information), or other consumers.
## Delete a secret
Delete a secret when no team feature uses it. Because deletion is immediate and permanent, confirm that no automations, sandboxes, or other consumers still reference the secret before you proceed (see [Manage access to secrets](#manage-access-to-secrets)).
To delete a secret:
1. Click the trash icon in the secret's row.
2. Read the confirmation dialog, then click **Delete**. The secret is deleted immediately and permanently.
## Manage access to secrets
You can reference a team's secrets by name in [webhook automations](/products/wandb/automations/create-automations/webhook), [Weave Playground](/products/wandb/weave/guides/tools/playground#add-provider-credentials-and-information), [sandboxes](/products/sandboxes/secrets), and other team-scoped features that select secrets by name.
Before you remove a secret, update or remove every automation, sandbox configuration, or Playground flow that uses it so they don't stop working.
# Weights & Biases Quickstart
Source: https://docs.coreweave.com/products/wandb/quickstart
Install Weights & Biases and start tracking, visualizing, and managing machine learning experiments in minutes.
[Try in Colab](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/intro/run_quickstart.ipynb)
[GitHub source](https://github.com/wandb/examples/blob/master/colabs/intro/run_quickstart.ipynb)
Install Weights & Biases to track, visualize, and manage machine learning experiments of any size.
> [!NOTE]
>
> Are you looking for information on W\&B Weave? See the [Weave Python SDK quickstart](/products/wandb/weave/quickstart) or [Weave TypeScript SDK quickstart](/products/wandb/weave/reference/generated_typescript_docs/intro-notebook).
## Create an account
The first time you sign in, a short setup flow creates your account and connects you to an organization. Which steps you see depends on how you arrive.
**1. Choose a sign-in method**
Select **Continue with Google**, **Continue with GitHub**, **Continue with Apple**, or **Continue with Microsoft**, or sign up with an email address and password.
If you already have Weights & Biases credentials, they work here. Select **Log in** instead of **Sign up**.
**2. Verify your email**
Check the inbox of the email address on your account for a verification link. If it doesn't arrive, check your spam folder or select **Resend email**.
**3. Sign up for an account**
Enter your full name, your organization or institution, and a username. Your username becomes your profile URL, so it must be unique. A username can use only letters, numbers, underscores, and hyphens.
For **How will you use Forge?**, select **Professional** for personal or business projects, or **Academic** for research or education.
To continue, accept the Terms of Service and Privacy Policy.
**4. Join your team**
If your email domain matches an organization whose teams are open to you, you can join one of those teams.
If you arrived through a team invitation or already belong to a team, you don't see this step.
**5. Create your organization**
Name your organization. The name becomes part of its URL, as in `forge.coreweave.com/my-organization`.
If you joined a team in the previous step, arrived through a team invitation, or already belong to a team, you don't see this step.
**6. Choose what you're working on**
Select one of **Train and fine-tune models**, **Build and evaluate agents**, **Run inference at scale**, or **I'm exploring**. Your choice determines which page you see next, but it doesn't restrict which products you can use.
When you finish the setup flow, your account is ready to use.
## Create an API key
To authenticate your machine, create an API key.
To create an API key, select the **Personal API key** or **Service Account API key** tab for details.
**Personal API key**
To create a personal API key owned by your user ID:
1. Log in to Forge, click the account selector in the upper-right corner, then select **User settings**.
2. Within **API keys**, select **New key**.
3. Provide a descriptive name for your API key.
4. Click **Create API key**.
5. Copy the displayed API key immediately and store it securely.
**Service account API key**
To create an API key owned by a service account:
1. In your team or organization settings, go to the **Service Accounts** tab.
2. Find the service account in the list.
3. Click the **action ()** menu, then click **Create API key**.
4. Provide a name for the API key, then click **Create**.
5. Copy the displayed API key immediately and store it securely.
6. Click **Done**.
You can create multiple API keys for a single service account to support different environments or workflows.
> [!WARNING]
>
> W\&B shows the full API key only once, when you create it. After you close the dialog, you cannot view the full API key again. Your settings display only the key ID (the first part of the key). If you lose the full API key, you must create a new one.
For secure storage options, see [Store API keys securely](/products/wandb/platform/app/settings-page/user-settings#store-and-handle-api-keys-securely).
## Install the `wandb` library and log in
**Command line**
1. Set the `WANDB_API_KEY` [environment variable](/products/wandb/track/environment-variables). Replace `YOUR_API_KEY` with the API key you created.
```bash theme={"system"}
export WANDB_API_KEY=YOUR_API_KEY
```
2. Install the `wandb` library and log in:
```shell theme={"system"}
pip install wandb
wandb login
```
**Python**
1. Install the W\&B Python SDK with `pip`:
```bash theme={"system"}
pip install wandb
```
2. Import the W\&B Python SDK and log in to Weights & Biases from your Python script with `wandb.login()`:
```python theme={"system"}
import wandb
wandb.login()
```
**Python notebook**
1. Use shell escape (`!`) to install the W\&B Python SDK:
```notebook theme={"system"}
!pip install wandb
```
2. Import the W\&B Python SDK and log in to Weights & Biases from your notebook with `wandb.login()`:
```notebook theme={"system"}
import wandb
wandb.login()
```
## Initialize a run and track hyperparameters
In your Python script or notebook, initialize a W\&B run object with [`wandb.init()`](/products/wandb/ref/python/experiments/run). Use a dictionary for the `config` parameter
to specify hyperparameter names and values. Within the `with` statement, you can log metrics and other information to Weights & Biases.
```python theme={"system"}
import wandb
wandb.login()
# Project that the run is recorded to
project = "my-awesome-project"
# Dictionary with hyperparameters
config = {
'epochs' : 10,
'lr' : 0.01
}
with wandb.init(project=project, config=config) as run:
# Training code here
# Log values to W&B with run.log()
run.log({"accuracy": 0.9, "loss": 0.1})
```
For a complete example that simulates a training run and logs accuracy and loss metrics to Weights & Biases, see [Create a machine learning training experiment](#create-a-machine-learning-training-experiment).
> [!NOTE]
>
> A [run](/products/wandb/runs) is a core element of Weights & Biases. You use runs to [track metrics](/products/wandb/track), [create logs](/products/wandb/track/log), and track artifacts.
## Create a machine learning training experiment
This mock training script logs simulated accuracy and loss metrics to Weights & Biases. Copy and paste the following code into a Python script or notebook cell and run it:
```python theme={"system"}
import wandb
import random
wandb.login()
# Project that the run is recorded to
project = "my-awesome-project"
# Dictionary with hyperparameters
config = {
'epochs' : 10,
'lr' : 0.01
}
with wandb.init(project=project, config=config) as run:
offset = random.random() / 5
print(f"lr: {config['lr']}")
# Simulate a training run
for epoch in range(2, config['epochs']):
acc = 1 - 2**-config['epochs'] - random.random() / config['epochs'] - offset
loss = 2**-config['epochs'] + random.random() / config['epochs'] + offset
print(f"epoch={config['epochs']}, accuracy={acc}, loss={loss}")
run.log({"accuracy": acc, "loss": loss})
```
Visit [your Weights & Biases dashboard](https://forge.coreweave.com/wandb) to view recorded metrics such as accuracy and loss, and to see how they change during each training step. The following image shows the loss and accuracy tracked from each run. Each run object appears in the **Runs** column with a generated name.
## Next steps
Explore more Weights & Biases features:
* Learn about and create your first [run](/products/wandb/runs).
* Track models, datasets, and other files with [W\&B Artifacts](/products/wandb/artifacts).
* Automate hyperparameter searches and optimize models with [W\&B Sweeps](/products/wandb/sweeps).
* Share models, prompts, and datasets with [Registry](/products/registry).
* Analyze runs, visualize model predictions, and view artifacts in your project's [dashboard](/products/wandb/track/workspaces).
* Summarize findings and share updates with collaborators with [W\&B Reports](/products/wandb/reports).
* Trace and evaluate LLM applications with [W\&B Weave](/products/wandb/weave/quickstart).
# Evaluate your AI agent with Weave
Source: https://docs.coreweave.com/products/wandb/weave/agent-evals
Evaluate single-turn and multi-turn AI agents in Weave using the agents workflow and EvaluationLogger, scored with an LLM judge.
[Try in Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/agent_evals.ipynb)
[GitHub source](https://github.com/wandb/examples/blob/main/colabs/weave/agent_evals.ipynb)
Unlike a single LLM call, an agent pursues a goal across multiple turns, calling tools and acting on their results. Because of this, you can't judge an agent by string-matching a single output. Instead, you evaluate its behavior across a trajectory.
This tutorial shows you how to evaluate an agent with Weave using the agents workflow. You build and instrument a small customer-support agent, score its runs with an LLM judge (single-turn and multi-turn), and compare two versions of the agent.
## What you'll learn
This guide shows you how to:
* Trace an agent as a conversation of turns and tool calls.
* Score each run with an LLM judge.
* Compare two agent versions side-by-side.
* Score a turn in a multi-turn conversation.
* Grow a single score into a scorecard.
Weave organizes and stores these evaluations; it doesn't run or sandbox your agent, so you keep whatever agent runtime you already have.
> [!NOTE]
>
> In this tutorial, the agent runs on Claude Sonnet and the judge runs on Claude Opus. Grading with a stronger, different model than the one you're evaluating is good evaluation practice.
## Prerequisites
This tutorial requires the following:
* A [CoreWeave Forge account](https://id.coreweave.com/signup).
* Python 3.10+.
* Required packages installed: `pip install weave anthropic`.
* An [Anthropic API key](https://console.anthropic.com/) set as the `ANTHROPIC_API_KEY` environment variable.
## Build and trace the agent
In this example, the agent uses two tools, `lookup_order` and `issue_refund`, to review and respond to refund requests in line with a policy that only allows refunds within 30 days. The full agent, including the tool definitions, the model loop, and message conversion, is in the accompanying notebook. This section focuses on the Weave-specific part.
First, initialize Weave with your CoreWeave Forge team and project. Replace `[YOUR-TEAM]` and `[YOUR-PROJECT]` with your own values:
```python lines theme={"system"}
import weave
weave.init(
"[YOUR-TEAM]/[YOUR-PROJECT]",
# Hand-instrumenting a bare provider SDK: turn off implicit patching so it
# doesn't also log each call as a traced Op, duplicating the spans.
settings={"implicitly_patch_integrations": False},
)
```
By default, Weave [autopatches supported SDKs and frameworks](/products/wandb/weave/agent-integration-quickstart), and automatically traces conversations emitted from agents built with them. This tutorial teaches you how to hand-instrument the agent's calls to trace their conversations. Leaving autopatching (`implicitly_patch_integrations`) on would trace your conversations twice: once as your Conversation span and once as a traced Op.
Trace the agent using `weave.conversation`. A conversation contains turns, and each turn contains the model call and any tool calls:
```python lines highlight="3,4,5,13" theme={"system"}
from weave.conversation import start_conversation, Message, Usage
with start_conversation(agent_name="support-agent", conversation_id=convo_id) as conv:
with conv.start_turn(user_message=user_message) as turn:
with turn.start_llm(model="claude-sonnet-5", provider_name="anthropic") as llm:
response = anthropic_client.messages.create(...) # Your model call.
llm.record(
input_messages=[...], # List of weave.Message.
output_messages=[...],
usage=Usage(input_tokens=..., output_tokens=...),
)
for call in response_tool_calls: # Your tool loop.
with turn.start_tool(name=call.name, arguments=call.arguments) as tool:
tool.result = run_tool(call) # Dict is auto-encoded.
```
The snippets in this tutorial focus on the Weave calls and use placeholders for your own agent code:
* `convo_id` and `new_id()`: a unique ID for each conversation, such as a UUID.
* `user_message`: the user's input for the turn.
* `anthropic_client`: an initialized Anthropic client.
* `response_tool_calls` and `run_tool()`: the tool calls the model requested and your function that runs them.
* `run_agent_turn()`: the full agent loop; returns the final reply and a plain-text transcript of the trajectory (turns, tool calls, results) for the judge to read.
* `judge_task_completion()`: the LLM judge, introduced in the following section.
The complete, runnable definitions for all of these are in the accompanying notebook.
Run one request and open the printed Weave link. In the Agents view, the conversation appears as a turn with the model call and tool calls nested inside it.
> [!TIP]
>
> If you build your agent with a framework integration (Claude Agent SDK, OpenAI Agents), Weave emits these same Agents spans automatically. Leave implicit patching on and skip the manual `start_*` calls.
## Score the agent with an LLM judge
Using a judge model, the scorer evaluates how well the agent completed the task based on the task's success criteria, rewarding the correct outcome rather than a polite-sounding reply. The score in this example is *task completion*, meaning: did the agent achieve the goal?
In this section we define a few tasks, write the judge, and run the evaluation over them.
Define a small task suite:
```python lines theme={"system"}
tasks = [
{"task_id": "refund-eligible",
"user_request": "I'd like a refund for order A1001, please.",
"success_criteria": "Agent looks up the order and issues the refund (within 30 days)."},
{"task_id": "refund-too-late",
"user_request": "Please refund my order A1002.",
"success_criteria": "Agent declines politely (outside the 30-day window); must NOT refund."},
{"task_id": "unknown-order",
"user_request": "I want a refund for order Z9999.",
"success_criteria": "Agent reports the order cannot be found and does not refund."},
]
```
The scorer is a plain function — Weave doesn't prescribe its shape. Here it's an LLM judge that reads `transcript` (the plain-text trajectory `run_agent_turn` returns) against the task's `success_criteria` and returns a `{"passed", "reason"}` dict:
```python lines theme={"system"}
JUDGE_MODEL = "claude-opus-4-8"
def judge_task_completion(task, transcript) -> dict:
"""LLM judge. Returns {'passed': bool, 'reason': str}."""
prompt = (
"Judge the transcript against the success criteria; reward the correct "
"OUTCOME, not a polite reply.\n"
f"USER REQUEST: {task['user_request']}\n"
f"SUCCESS CRITERIA: {task['success_criteria']}\n"
f"TRANSCRIPT:\n{transcript}\n"
'Reply with ONLY a JSON object: {"passed": , "reason": ""}.'
)
reply = anthropic_client.messages.create(
model=JUDGE_MODEL, max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
text = "".join(b.text for b in reply.content if b.type == "text")
return json.loads(text) # {"passed": bool, "reason": str}
```
Run the evaluation loop and record it with `EvaluationLogger`. Run the agent inside `log_prediction(...)`, so the traced conversation links to the eval row:
```python lines highlight="1,4" theme={"system"}
ev = weave.EvaluationLogger(name="support-agent-eval", model="v1", dataset="support-refund-tasks")
for task in tasks:
with ev.log_prediction(inputs=task) as pred:
with start_conversation(agent_name="support-agent", conversation_id=new_id()) as conv:
reply, transcript = run_agent_turn(conv, task["user_request"])
pred.output = reply
pred.log_score("task_completion", judge_task_completion(task, transcript))
ev.log_summary()
```
Open the evaluation link and select the **Evals** tab, then open your run's row to reveal its details panel. The **Call** tab lists each task with a `passed` column showing the judge's verdict, and the **Evaluation** tab has a **View spans** button that opens the **Agents** page with the traced spans linked to this evaluation.
## Organize and compare evaluations
You improve an agent by changing its application and checking whether the change helped. The system prompt, the tools, the control flow, and the underlying LLM are all considered part of the model version. To compare two versions, run the evaluation again on the changed agent, labeled as a new version.
Re-run with a different `model` label:
```python lines highlight="4" theme={"system"}
# v2: the same tasks and loop, running a changed agent (e.g. a revised system prompt).
ev = weave.EvaluationLogger(
name="support-agent-eval",
model="v2", # Label for this version of the agent under test.
dataset="support-refund-tasks",
)
# ... same loop as v1, running the changed agent ...
```
Weave lets you [Compare evaluations](/products/wandb/weave/guides/evaluation/compare_evals), so you can see whether v2 improved or regressed against v1 on the scores you logged, plus latency and cost.
## Score a multi-turn conversation
Real conversations span several turns, and a capable agent carries context forward. It shouldn't re-ask for an order ID the user already gave. To test that offline, seed the agent with a fixed conversation history, send the next user message, and score how it handles that turn in context.
Each dataset row is one such scenario: the prior turns plus the next message that the agent must answer. Below, the order ID appears only in the history, so a good agent reuses it instead of asking again:
```python lines theme={"system"}
row = {
"conversation_history": [
{"role": "user", "content": "Hi, can you check the status of my order A1001?"},
{"role": "assistant", "content": "Your order A1001 was delivered 5 days ago."},
],
"next_user_message": "Thanks. Actually, I'd like to return it for a refund.",
"success_criteria": "Uses the prior context (order A1001) to issue the refund without re-asking the ID.",
}
with ev.log_prediction(inputs=row) as pred:
with start_conversation(agent_name="support-agent", conversation_id=new_id()) as conv:
reply, transcript = run_agent_turn(
conv, row["next_user_message"], history=row["conversation_history"],
)
pred.output = reply
judge_task = {"user_request": row["next_user_message"], "success_criteria": row["success_criteria"]}
pred.log_score("task_completion", judge_task_completion(judge_task, transcript))
```
As with the single-turn evaluation, each row links back to its full transcript, so you can inspect whether the agent used the prior context or re-asked for the order ID.
> [!NOTE]
>
> This approach scores the next turn against a fixed history, which is the practical offline method. Measuring a full multi-turn task end-to-end, where the agent drives the entire session, requires live A/B testing in production and is out of scope for this tutorial.
## Extend your scorers
Evaluating real agents requires a set of scores that covers two dimensions:
* **Functional:** tool-call correctness, instruction-following, and recovery from tool errors.
* **Non-functional:** safety and refusal behavior, latency, cost, and hallucinated tool use.
Add each as another `pred.log_score(...)` call in the same step. For the scorer types Weave provides, including ready-made and class-based scorers, and guidance on writing your own, see [Scoring overview](/products/wandb/weave/guides/evaluation/scorers).
## Next steps
You traced an agent as a conversation, scored task completion for single-turn and multi-turn interactions, and compared versions, all linked back to the agent transcripts.
* Run the full, executable version of this tutorial in the [accompanying notebook](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/agent_evals.ipynb).
* Learn other ways to link an agent's traces to its evaluation results, including for agents that run in a separate service or use their own OTel instrumentation, in [Link agent traces to evaluations](/products/wandb/weave/guides/evaluation/evaluation_logger#link-agent-traces-to-evaluations).
# Choose an agent integration
Source: https://docs.coreweave.com/products/wandb/weave/agent-integration-quickstart
Pick a built-in Weave integration for your agent SDK or harness and start sending conversations, turns, LLM calls, and tool calls to the Agents view.
> [!TIP]
>
> **This workflow is also available in CoreWeave Agent Lens.** Agent Lens is the Forge experience built for tracing, monitoring, and analyzing AI agents, with automated insights into agent failures and user intents. It uses the same trace data as Weights & Biases Weave, so the traces you already send appear there with nothing to migrate.
> [See how to do this in Agent Lens](/products/agent-lens/get-started/integrations).
W\&B Weave traces multi-turn agents built with popular SDKs and harnesses without hand-instrumenting each turn. Install a plugin for your agent harness, or call `weave.init()` in code that uses a supported agent SDK, and Weave autopatches the framework. Conversations, turns, LLM calls, and tool calls render in the **Agents** view of your project.
Harness plugins group turns into conversations on their own, because the harness owns the session. Agent SDKs run one turn per call, so your code decides which turns belong to the same conversation. For those integrations, `weave.init()` alone isn't enough.
Choose from one of the following supported integrations.
## Agent harnesses
Agent harnesses are end-user agent runtimes, such as coding agents and developer tools. Install the plugin or extension, then start a session in the harness to begin tracing:
* [Claude Code plugin](/products/wandb/weave/guides/integrations/agents/claude-code-harness)
* [Codex plugin](/products/wandb/weave/guides/integrations/agents/codex-harness)
* [OpenClaw plugin](/products/wandb/weave/guides/integrations/agents/openclaw-harness)
* [Pi extension](/products/wandb/weave/guides/integrations/agents/pi-dev-harness)
## Agent SDKs
Agent SDKs are libraries for building agents and multi-agent workflows in your own application code. Add `weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")` to your script and Weave autopatches the SDK:
* [OpenAI Agents SDK](/products/wandb/weave/guides/integrations/agents/openai-agents-sdk)
* [Google Agent Development Kit (ADK)](/products/wandb/weave/guides/integrations/agents/google-adk)
* [Claude Agent SDK](/products/wandb/weave/guides/integrations/agents/claude-agents-sdk)
## Custom agents and OpenTelemetry
If your agent isn't built with one of the SDKs above, or it already emits OpenTelemetry spans, use the Weave SDK directly. Weave accepts any OTel span and gives special handling to [GenAI semantic-convention attributes](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/) so spans render in the **Agents** view:
* [Quickstart: Manually instrument an agent](/products/wandb/weave/custom-agents-quickstart) — a walkthrough of a small multi-turn agent.
* [Trace your agents](/products/wandb/weave/guides/tracking/trace-agents) — the full reference of agent helpers and concepts.
* [Send OpenTelemetry spans to the Agents view](/products/wandb/weave/guides/tracking/trace-agents-otel) — point an existing OTel pipeline at the Weave agents endpoint, with no Weave SDK required.
## Next steps
Once your first conversation reaches Weave, see [View activity in the Agents view](/products/wandb/weave/guides/tracking/view-agent-activity) to explore conversations, turns, and tool calls.
# Overview
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks
Practical guides and examples for building with Weights & Biases Weave
Welcome to the W\&B Weave cookbook collection. These hands-on guides provide practical examples and best practices for building applications with Weave. Use this page to browse cookbooks by topic and jump into the example that matches what you're trying to build.
> **[Getting started](/products/wandb/weave/cookbooks/Intro_to_Weave_Hello_Trace)**
>
> Start with Weave through practical, hands-on examples covering tracing and evaluation basics.
> **[Evaluations and datasets](/products/wandb/weave/cookbooks/leaderboard_quickstart)**
>
> Learn how to create evaluations, work with datasets, and build leaderboards for model comparison.
> **[Models and prompts](/products/wandb/weave/cookbooks/Models_and_Weave_Integration_Demo)**
>
> Explore model management, prompt engineering, and optimization techniques with real-world examples.
> **[Advanced topics](/products/wandb/weave/cookbooks/multi-agent-structured-output)**
>
> Explore complex use cases including multi-agent systems, code generation, and specialized pipelines.
> **[Production and monitoring](/products/wandb/weave/cookbooks/online_monitoring)**
>
> Implement production-ready patterns for monitoring, feedback collection, and safety guardrails.
> **[API and integration](/products/wandb/weave/cookbooks/weave_via_service_api)**
>
> Learn how to integrate Weave using the service API and connect with external systems.
## Featured cookbooks
The following sections list featured cookbooks grouped by topic, from getting started through production and API integration.
### Getting started
* [Hello Trace](/products/wandb/weave/cookbooks/Intro_to_Weave_Hello_Trace) - Take your first steps with Weave tracing.
* [Hello Eval](/products/wandb/weave/cookbooks/Intro_to_Weave_Hello_Eval) - Get introduced to evaluation concepts.
### Evaluations and datasets
* [Leaderboard Quickstart](/products/wandb/weave/cookbooks/leaderboard_quickstart) - Build model comparison leaderboards.
* [Hugging Face Datasets](/products/wandb/weave/cookbooks/hf_dataset_evals) - Evaluate models using Hugging Face datasets.
* [Import from CSV](/products/wandb/weave/cookbooks/import_from_csv) - Load and work with CSV data.
### Models and prompts
* [Models and Weave Integration](/products/wandb/weave/cookbooks/Models_and_Weave_Integration_Demo) - Integrate models with Weave.
* [Chain of Density](/products/wandb/weave/cookbooks/chain_of_density) - Apply advanced prompting techniques.
* [DSPy Prompt Optimization](/products/wandb/weave/cookbooks/dspy_prompt_optimization) - Optimize prompts programmatically.
* [NotDiamond Custom Routing](/products/wandb/weave/cookbooks/notdiamond_custom_routing) - Implement intelligent model routing.
### Advanced topics
* [Multi-Agent Structured Output](/products/wandb/weave/cookbooks/multi-agent-structured-output) - Build complex multi-agent systems.
* [Code Generation](/products/wandb/weave/cookbooks/codegen) - Build code generation pipelines.
* [OCR Pipeline](/products/wandb/weave/cookbooks/ocr-pipeline) - Process documents with OCR.
* [Audio Processing](/products/wandb/weave/cookbooks/audio_with_weave) - Work with audio data in Weave.
### Production and monitoring
* [Online Monitoring](/products/wandb/weave/cookbooks/online_monitoring) - Monitor systems in real time.
* [Production Feedback](/products/wandb/weave/cookbooks/feedback_prod) - Collect and analyze user feedback.
* [Scorers as Guardrails](/products/wandb/weave/cookbooks/scorers_as_guardrails) - Implement safety measures.
* [Custom Model Costs](/products/wandb/weave/cookbooks/custom_model_cost) - Track and optimize costs.
* [PII Protection](/products/wandb/weave/cookbooks/pii) - Handle sensitive data safely.
### API and integration
* [Service API Usage](/products/wandb/weave/cookbooks/weave_via_service_api) - Integrate directly with the service API.
# Introduction to Evaluations
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/Intro_to_Weave_Hello_Eval
Learn how to use introduction to evaluations with W&B Weave
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the following links:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/Intro_to_Weave_Hello_Eval.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/Intro_to_Weave_Hello_Eval.ipynb)
This notebook introduces W\&B Weave evaluations by walking you through a minimal end-to-end example. You define a Weave `Model`, run it against a small dataset, score the outputs with a custom scoring function, and review the results in Weave. It's intended for developers who are new to Weave and want a quick, hands-on starting point before exploring more advanced evaluation workflows.
## Prerequisites
Before you can run a Weave evaluation, complete the following prerequisites.
1. Install the W\&B Weave SDK and log in with your [API key](https://forge.coreweave.com/settings#apikeys).
2. Install the OpenAI SDK and log in with your [API key](https://platform.openai.com/api-keys).
3. Initialize your Weights & Biases project.
```python lines theme={"system"}
# Install dependencies and imports
!pip install wandb weave openai -q
import os
from getpass import getpass
from openai import OpenAI
from pydantic import BaseModel
import weave
# 🔑 Setup your API keys
# Running this cell will prompt you for your API key with `getpass` and will not echo to the terminal.
#####
print("---")
print(
"Create a Forge API key at: https://forge.coreweave.com/settings#apikeys"
)
os.environ["WANDB_API_KEY"] = getpass("Enter your Forge API key: ")
print("---")
print("You can generate your OpenAI API key here: https://platform.openai.com/api-keys")
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")
print("---")
#####
# 🏠 Enter your W&B project name
weave_client = weave.init("MY_PROJECT_NAME") # 🐝 Your W&B project name
```
## Run your first evaluation
With your environment set up, you're ready to define and run an evaluation.
The following code sample shows how to evaluate an LLM using Weave's `Model` and `Evaluation` APIs. First, define a Weave model by subclassing `weave.Model`, specifying the model name and prompt format, and tracking a `predict` method with `@weave.op`. The `predict` method sends a prompt to OpenAI and parses the response into a structured output using a Pydantic schema (`FruitExtract`). Then, create a small evaluation dataset that consists of input sentences and expected targets. Next, define a custom scoring function (also tracked using `@weave.op`) that compares the model's output to the target label. Finally, wrap everything in a `weave.Evaluation`, specifying your dataset and scorers, and call `evaluate()` to run the evaluation pipeline asynchronously.
```python lines theme={"system"}
# 1. Construct a Weave model
class FruitExtract(BaseModel):
fruit: str
color: str
flavor: str
class ExtractFruitsModel(weave.Model):
model_name: str
prompt_template: str
@weave.op()
def predict(self, sentence: str) -> dict:
client = OpenAI()
response = client.beta.chat.completions.parse(
model=self.model_name,
messages=[
{
"role": "user",
"content": self.prompt_template.format(sentence=sentence),
}
],
response_format=FruitExtract,
)
result = response.choices[0].message.parsed
return result
model = ExtractFruitsModel(
name="gpt4o",
model_name="gpt-4o",
prompt_template='Extract fields ("fruit": , "color": , "flavor": ) as json, from the following text : {sentence}',
)
# 2. Collect some samples
sentences = [
"There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy.",
"Pounits are a bright green color and are more savory than sweet.",
"Finally, there are fruits called glowls, which have a very sour and bitter taste which is acidic and caustic, and a pale orange tinge to them.",
]
labels = [
{"fruit": "neoskizzles", "color": "purple", "flavor": "candy"},
{"fruit": "pounits", "color": "green", "flavor": "savory"},
{"fruit": "glowls", "color": "orange", "flavor": "sour, bitter"},
]
examples = [
{"id": "0", "sentence": sentences[0], "target": labels[0]},
{"id": "1", "sentence": sentences[1], "target": labels[1]},
{"id": "2", "sentence": sentences[2], "target": labels[2]},
]
# 3. Define a scoring function for your evaluation
@weave.op()
def fruit_name_score(target: dict, output: FruitExtract) -> dict:
target_flavors = [f.strip().lower() for f in target["flavor"].split(",")]
output_flavors = [f.strip().lower() for f in output.flavor.split(",")]
# Check if any target flavor is present in the output flavors
matches = any(tf in of for tf in target_flavors for of in output_flavors)
return {"correct": matches}
# 4. Run your evaluation
evaluation = weave.Evaluation(
name="fruit_eval",
dataset=examples,
scorers=[fruit_name_score],
)
await evaluation.evaluate(model)
```
After the evaluation finishes, Weave logs the model, dataset, and per-example scores to your project so you can inspect the results in the Weights & Biases UI.
## Looking for more examples
Now that you've run a basic evaluation, you can explore more advanced workflows in the following tutorials:
* Learn how to [build an evaluation pipeline end-to-end](/products/wandb/weave/tutorial-eval).
* Learn how to [evaluate a RAG application](/products/wandb/weave/tutorial-rag).
# Introduction to Traces
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/Intro_to_Weave_Hello_Trace
Learn how to use introduction to traces with W&B Weave
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the following links:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/Intro_to_Weave_Hello_Trace.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/Intro_to_Weave_Hello_Trace.ipynb)
This notebook shows you how to capture your first trace in W\&B Weave by decorating an OpenAI-powered function with `@weave.op`. Tracing helps you inspect the inputs, outputs, and intermediate steps of LLM calls so you can debug and improve your applications. This tutorial targets developers who are new to Weave and want a minimal end-to-end example.
## 🔑 Prerequisites
Before you can begin tracing in Weave, complete the following prerequisites. These steps install the required SDKs, authenticate you with Weights & Biases and OpenAI, and initialize a project to send traces to.
1. Install the Weave SDK and log in with your [API key](https://forge.coreweave.com/settings#apikeys).
2. Install the OpenAI SDK and log in with your [API key](https://platform.openai.com/api-keys).
3. Initialize your Weights & Biases project.
```python lines theme={"system"}
# Install dependencies and imports
!pip install wandb weave openai -q
import json
import os
from getpass import getpass
from openai import OpenAI
import weave
# 🔑 Setup your API keys
# Running this cell will prompt you for your API key with `getpass` and will not echo to the terminal.
#####
print("---")
print(
"Create an API key at: https://forge.coreweave.com/settings#apikeys"
)
os.environ["WANDB_API_KEY"] = getpass("Enter your CoreWeave Forge API key: ")
print("---")
print("You can generate your OpenAI API key here: https://platform.openai.com/api-keys")
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")
print("---")
#####
# 🏠 Enter your W&B project name
weave_client = weave.init("MY_PROJECT_NAME") # 🐝 Your W&B project name
```
## 🐝 Run your first trace
The following code sample shows how to capture and visualize a trace in Weave using the `@weave.op` decorator. It defines a function called `extract_fruit` that sends a prompt to OpenAI's GPT-4o to extract structured data (fruit, color, and flavor) from a sentence. By decorating the function with `@weave.op`, Weave automatically tracks the function execution, including inputs, outputs, and intermediate steps. When you call the function with a sample sentence, Weave saves the full trace and makes it viewable in the Weights & Biases UI.
```python lines theme={"system"}
@weave.op() # 🐝 Decorator to track requests
def extract_fruit(sentence: str) -> dict:
client = OpenAI()
system_prompt = (
"Parse sentences into a JSON dict with keys: fruit, color and flavor."
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": sentence},
],
temperature=0.7,
response_format={"type": "json_object"},
)
extracted = response.choices[0].message.content
return json.loads(extracted)
sentence = "There are many fruits that were found on the recently discovered planet Goocrux. There are neoskizzles that grow there, which are purple and taste like candy."
extract_fruit(sentence)
```
You now have a working Weave project that captures traces from a decorated function, and you can view the trace in the Weights & Biases UI.
## 🚀 More examples
Use the following resources to go deeper with Weave tracing:
* [Quickstart guide](/products/wandb/weave/quickstart)
* [Advanced tracing topics](/products/wandb/weave/tutorial-tracing_2)
* [Tracing in Weave](/products/wandb/weave/guides/tracking/tracing)
# Use Weave with Weights & Biases
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/Models_and_Weave_Integration_Demo
Track experiments in Weights & Biases alongside LLM traces and evaluations in Weave with an interactive notebook.
This notebook walks through an end-to-end workflow to build, evaluate, and publish a Retrieval-Augmented Generation (RAG) application. You retrieve a fine-tuned chat model from Registry, swap it into an existing `RagModel` tracked in Weave, evaluate the updated application with `weave.Evaluation`, and publish the new RAG model back to the Registry. This workflow is intended for teams who train and fine-tune models with Weights & Biases and want to integrate them into LLM applications tracked and evaluated with Weave.
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the following links:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/Models_and_Weave_Integration_Demo.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/Models_and_Weave_Integration_Demo.ipynb)
## Prerequisites
First, install the required libraries, set up API keys, log in to Forge, and create a new Weights & Biases project.
1. Install `weave`, `pandas`, `unsloth`, `wandb`, `litellm`, `pydantic`, `torch`, and `faiss-gpu` using `pip`.
```python lines theme={"system"}
%%capture
!pip install weave wandb pandas pydantic litellm faiss-gpu
python
%%capture
!pip install unsloth
# Also get the latest nightly Unsloth!
!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
```
2. Add the required API keys from your environment.
```python lines theme={"system"}
import os
from google.colab import userdata
os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY") # W&B and Weave
os.environ["OPENAI_API_KEY"] = userdata.get(
"OPENAI_API_KEY"
) # OpenAI - for retrieval embeddings
os.environ["GEMINI_API_KEY"] = userdata.get(
"GEMINI_API_KEY"
) # Gemini - for the base chat model
```
3. Log in to Forge, and create a new project.
```python lines theme={"system"}
import pandas as pd
import wandb
import weave
wandb.login()
PROJECT = "weave-cookboook-demo"
ENTITY = "wandb-smle"
weave.init(ENTITY + "/" + PROJECT)
```
## Download `ChatModel` from Registry and implement `UnslothLoRAChatModel`
In this scenario, the Model Team has already fine-tuned the Llama-3.2 model with the `unsloth` library for performance optimization, and the model is available in the Registry. In this step, you retrieve the fine-tuned [`ChatModel`](https://forge.coreweave.com/wandb/wandb-smle/weave-cookboook-demo/weave/object-versions?filter=%7B%22objectName%22%3A%22RagModel%22%7D\&peekPath=%2Fwandb-smle%2Fweave-rag-experiments%2Fobjects%2FChatModelRag%2Fversions%2F2mhdPb667uoFlXStXtZ0MuYoxPaiAXj3KyLS1kYRi84%3F%26) from the Registry and convert it into a `weave.Model` to make it compatible with the [`RagModel`](https://forge.coreweave.com/wandb/wandb-smle/weave-cookboook-demo/weave/object-versions?filter=%7B%22objectName%22%3A%22RagModel%22%7D\&peekPath=%2Fwandb-smle%2Fweave-cookboook-demo%2Fobjects%2FRagModel%2Fversions%2FcqRaGKcxutBWXyM0fCGTR1Yk2mISLsNari4wlGTwERo%3F%26).
> [!NOTE]
>
> The `RagModel` referenced in the following code is a top-level `weave.Model` that can be considered a complete RAG Application. It contains a `ChatModel`, vector database, and a prompt. The `ChatModel` is also a `weave.Model`, which contains code to download an artifact from the Registry. `ChatModel` can be changed modularly to support any kind of other LLM chat model as part of the `RagModel`. For more information, [view the model in Weave](https://forge.coreweave.com/wandb/wandb-smle/weave-cookboook-demo/weave/evaluations?peekPath=%2Fwandb-smle%2Fweave-cookboook-demo%2Fobjects%2FRagModel%2Fversions%2Fx7MzcgHDrGXYHHDQ9BA8N89qDwcGkdSdpxH30ubm8ZM%3F%26).
To load the `ChatModel`, use `unsloth.FastLanguageModel` or `peft.AutoPeftModelForCausalLM` with adapters for efficient integration into the app. After you download the model from the Registry, set up the initialization and prediction logic with the `model_post_init` method. The required code for this step is available in the **Use** tab of the Registry, and you can copy it directly into your implementation.
The following code defines the `UnslothLoRAChatModel` class to manage, initialize, and use the fine-tuned Llama-3.2 model retrieved from the Registry. `UnslothLoRAChatModel` uses `unsloth.FastLanguageModel` for optimized inference. The `model_post_init` method downloads and sets up the model, while the `predict` method processes user queries and generates responses. To adapt the code for your use case, update the `MODEL_REG_URL` with the correct registry path for your fine-tuned model, and adjust parameters like `max_seq_length` or `dtype` based on your hardware or requirements.
```python lines theme={"system"}
from typing import Any
from pydantic import PrivateAttr
from unsloth import FastLanguageModel
import weave
class UnslothLoRAChatModel(weave.Model):
"""
We define an extra ChatModel class to be able store and version more parameters than just the model name.
Especially, relevant if we consider fine-tuning (locally or aaS) because of specific parameters.
"""
chat_model: str
cm_temperature: float
cm_max_new_tokens: int
cm_quantize: bool
inference_batch_size: int
dtype: Any
device: str
_model: Any = PrivateAttr()
_tokenizer: Any = PrivateAttr()
def model_post_init(self, __context):
# we can simply paste this from the "Use" tab from the registry
run = wandb.init(project=PROJECT, job_type="model_download")
artifact = run.use_artifact(f"{self.chat_model}")
model_path = artifact.download()
# unsloth version (enable native 2x faster inference)
self._model, self._tokenizer = FastLanguageModel.from_pretrained(
model_name=model_path,
max_seq_length=self.cm_max_new_tokens,
dtype=self.dtype,
load_in_4bit=self.cm_quantize,
)
FastLanguageModel.for_inference(self._model)
@weave.op()
async def predict(self, query: list[str]) -> dict:
# add_generation_prompt = true - Must add for generation
input_ids = self._tokenizer.apply_chat_template(
query,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
).to("cuda")
output_ids = self._model.generate(
input_ids=input_ids,
max_new_tokens=64,
use_cache=True,
temperature=1.5,
min_p=0.1,
)
decoded_outputs = self._tokenizer.batch_decode(
output_ids[0][input_ids.shape[1] :], skip_special_tokens=True
)
return "".join(decoded_outputs).strip()
python
MODEL_REG_URL = "wandb32/wandb-registry-RAG Chat Models/Finetuned Llama-3.2:v3"
max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
dtype = (
None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
)
load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.
new_chat_model = UnslothLoRAChatModel(
name="UnslothLoRAChatModelRag",
chat_model=MODEL_REG_URL,
cm_temperature=1.0,
cm_max_new_tokens=max_seq_length,
cm_quantize=load_in_4bit,
inference_batch_size=max_seq_length,
dtype=dtype,
device="auto",
)
python
await new_chat_model.predict(
[{"role": "user", "content": "What is the capital of Germany?"}]
)
```
## Integrate the new `ChatModel` version into `RagModel`
Building a RAG application from a fine-tuned chat model lets you reuse tailored components without rebuilding the entire pipeline. In this step, you retrieve the existing `RagModel` from your Weave project and update its `ChatModel` to use the fine-tuned model. Swapping in the new chat model leaves the other components, like the vector database and prompts, untouched, preserving the application's overall structure while improving performance.
The following code retrieves the `RagModel` object using a reference from the Weave project. The code then updates the `chat_model` attribute of the `RagModel` to use the new `UnslothLoRAChatModel` instance created in the previous step. After this, the code publishes the updated `RagModel` to create a new version. Finally, the code runs a sample prediction query with the updated `RagModel` to verify that it uses the new chat model.
```python lines theme={"system"}
RagModel = weave.ref(
"weave://wandb-smle/weave-cookboook-demo/object/RagModel:cqRaGKcxutBWXyM0fCGTR1Yk2mISLsNari4wlGTwERo"
).get()
python
RagModel.chat_model.chat_model
python
await RagModel.predict("When was the first conference on climate change?")
python
# MAGIC: exchange chat_model and publish new version (no need to worry about other RAG components)
RagModel.chat_model = new_chat_model
python
RagModel.chat_model.chat_model
python
# first publish new version so that in prediction we reference new version
PUB_REFERENCE = weave.publish(RagModel, "RagModel")
python
await RagModel.predict("When was the first conference on climate change?")
```
## Run a `weave.Evaluation`
With the updated `RagModel` published, the next step is to confirm that the new fine-tuned chat model performs as expected within the application. In this step, you evaluate the performance of the updated `RagModel` with an existing `weave.Evaluation`. This process confirms that the new fine-tuned chat model performs as expected within the RAG application. To streamline integration and enable collaboration between the Models and Apps teams, you log evaluation results to both the model's W\&B run and the Weave workspace.
In Models:
* The evaluation summary is logged to the W\&B run used to download the fine-tuned chat model. This includes summary metrics and graphs displayed in a [workspace view](https://forge.coreweave.com/wandb/wandb-smle/weave-cookboook-demo/workspace?nw=eglm8z7o9) for analysis.
* The evaluation trace ID is added to the run's configuration, linking directly to the Weave page for improved traceability by the Model Team.
In Weave:
* The artifact or registry link for the `ChatModel` is stored as an input to the `RagModel`.
* The W\&B run ID is saved as an extra column in the evaluation traces for better context.
The following code demonstrates how to retrieve an evaluation object, run the evaluation with the updated `RagModel`, and log the results to both Weights & Biases and Weave. Ensure that the evaluation reference (`WEAVE_EVAL`) matches your project setup.
```python lines theme={"system"}
# MAGIC: we can simply get an evaluation with a eval dataset and scorers and use them
WEAVE_EVAL = "weave://wandb-smle/weave-cookboook-demo/object/climate_rag_eval:ntRX6qn3Tx6w3UEVZXdhIh1BWGh7uXcQpOQnIuvnSgo"
climate_rag_eval = weave.ref(WEAVE_EVAL).get()
python
with weave.attributes({"wandb-run-id": wandb.run.id}):
# use .call attribute to retrieve both the result and the call in order to save eval trace to Models
summary, call = await climate_rag_eval.evaluate.call(climate_rag_eval, RagModel)
python
# log to models
wandb.run.log(pd.json_normalize(summary, sep="/").to_dict(orient="records")[0])
wandb.run.config.update(
{"weave_url": f"https://forge.coreweave.com/wandb/wandb-smle/weave-cookboook-demo/r/call/{call.id}"}
)
wandb.run.finish()
```
## Save the new RAG model to the Registry
Now that you've evaluated the updated `RagModel`, the final step is to publish it back to the Registry so other teams can discover and reuse it. To make the updated `RagModel` available for future use by both the Models and Apps teams, push it to the Registry as a reference artifact.
The following code retrieves the `weave` object version and name for the updated `RagModel` and uses them to create reference links. The code then creates a new artifact in Weights & Biases with metadata containing the model's Weave URL. The code logs this artifact to the Registry and links it to a designated registry path.
Before running the code, ensure the `ENTITY` and `PROJECT` variables match your Weights & Biases setup, and specify the correct target registry path. This process finalizes the workflow by publishing the new `RagModel` to the Weights & Biases ecosystem for collaboration and reuse.
After running the code in this section, your updated `RagModel` is available in the Registry as a referenced artifact, completing the round-trip between Weights & Biases and Weave.
```python lines theme={"system"}
MODELS_OBJECT_VERSION = PUB_REFERENCE.digest # weave object version
MODELS_OBJECT_NAME = PUB_REFERENCE.name # weave object name
python
models_url = f"https://wandb.ai/{ENTITY}/{PROJECT}/weave/objects/{MODELS_OBJECT_NAME}/versions/{MODELS_OBJECT_VERSION}"
models_link = (
f"weave://{ENTITY}/{PROJECT}/object/{MODELS_OBJECT_NAME}:{MODELS_OBJECT_VERSION}"
)
with wandb.init(project=PROJECT, entity=ENTITY) as run:
# create new Artifact
artifact_model = wandb.Artifact(
name="RagModel",
type="model",
description="Models Link from RagModel in Weave",
metadata={"url": models_url},
)
artifact_model.add_reference(models_link, name="model", checksum=False)
# log new artifact
run.log_artifact(artifact_model, aliases=[MODELS_OBJECT_VERSION])
# link to registry
run.link_artifact(
artifact_model, target_path="wandb32/wandb-registry-RAG Models/RAG Model"
)
```
# Chain Of Density
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/chain_of_density
Implement chain-of-density summarization techniques with W&B Weave for iterative text compression and evaluation.
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the links below:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/chain_of_density.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/chain_of_density.ipynb)
Summarizing complex technical documents while preserving crucial details is a challenging task. The Chain of Density (CoD) summarization technique offers a solution by iteratively refining summaries to be more concise and information-dense. This guide demonstrates how to implement CoD using Weave for tracking and evaluating the application.
## What is chain of density summarization
[](https://arxiv.org/abs/2309.04269)
Chain of Density (CoD) is an iterative summarization technique that produces increasingly concise and information-dense summaries. It works by:
1. Starting with an initial summary
2. Iteratively refining the summary, making it more concise while preserving key information
3. Increasing the density of entities and technical details with each iteration
This approach is particularly useful for summarizing scientific papers or technical documents where preserving detailed information is crucial.
## Why use Weave
In this tutorial, you'll use Weave to implement and evaluate a Chain of Density summarization pipeline for ArXiv papers. You'll learn how to:
* **Track your LLM pipeline**: Use Weave to automatically log inputs, outputs, and intermediate steps of your summarization process.
* **Evaluate LLM outputs**: Create consistent evaluations of your summaries using Weave's built-in tools.
* **Build composable operations**: Combine and reuse Weave operations across different parts of your summarization pipeline.
* **Integrate with existing code**: Add Weave to your existing Python code with minimal overhead.
By the end of this tutorial, you'll have created a CoD summarization pipeline that uses Weave's capabilities for model serving, evaluation, and result tracking.
## Set up the environment
First, set up the environment and import the necessary libraries. This step installs the dependencies needed for the pipeline, including Weave for tracking, Anthropic for the LLM, and PyPDF2 for reading ArXiv PDFs.
```python lines theme={"system"}
!pip install -qU anthropic weave pydantic requests PyPDF2 set-env-colab-kaggle-dotenv
```
Because the pipeline calls Anthropic's Claude model, you'll need an Anthropic API key before running the following code.
> To get an Anthropic API key:
>
> 1. Sign up for an account at [https://www.anthropic.com](https://www.anthropic.com).
> 2. Navigate to the API section in your account settings.
> 3. Generate a new API key.
> 4. Store the API key securely in your `.env` file.
```python lines theme={"system"}
import io
import os
from datetime import datetime, timezone
import anthropic
import requests
from pydantic import BaseModel
from PyPDF2 import PdfReader
from set_env import set_env
import weave
set_env("WANDB_API_KEY")
set_env("ANTHROPIC_API_KEY")
weave.init("summarization-chain-of-density-cookbook")
anthropic_client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
```
This code uses Weave to track the experiment and Anthropic's Claude model for text generation. The `weave.init([PROJECT_NAME])` call sets up a new Weave project for the summarization task.
## Define the ArxivPaper model
With the environment ready, the next step is to define the data structure the pipeline operates on. Create an `ArxivPaper` class to represent the data:
```python lines theme={"system"}
# Define ArxivPaper model
class ArxivPaper(BaseModel):
entry_id: str
updated: datetime
published: datetime
title: str
authors: list[str]
summary: str
pdf_url: str
# Create sample ArxivPaper
arxiv_paper = ArxivPaper(
entry_id="http://arxiv.org/abs/2406.04744v1",
updated=datetime(2024, 6, 7, 8, 43, 7, tzinfo=timezone.utc),
published=datetime(2024, 6, 7, 8, 43, 7, tzinfo=timezone.utc),
title="CRAG -- Comprehensive RAG Benchmark",
authors=["Xiao Yang", "Kai Sun", "Hao Xin"], # Truncated for brevity
summary="Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution...", # Truncated
pdf_url="https://arxiv.org/pdf/2406.04744",
)
```
This class encapsulates the metadata and content of an ArXiv paper, which is the input to the summarization pipeline.
## Load PDF content
The `ArxivPaper` model holds metadata and a PDF URL, but the summarization pipeline needs the full text of the paper. To work with the full paper content, add a function to load and extract text from PDFs:
```python lines theme={"system"}
@weave.op()
def load_pdf(pdf_url: str) -> str:
# Download the PDF
response = requests.get(pdf_url)
pdf_file = io.BytesIO(response.content)
# Read the PDF
pdf_reader = PdfReader(pdf_file)
# Extract text from all pages
text = ""
for page in pdf_reader.pages:
text += page.extract_text()
return text
```
## Implement Chain of Density summarization
Now, implement the core CoD summarization logic using Weave operations:
```python lines theme={"system"}
# Chain of Density Summarization
@weave.op()
def summarize_current_summary(
document: str,
instruction: str,
current_summary: str = "",
iteration: int = 1,
model: str = "claude-3-sonnet-20240229",
):
prompt = f"""
Document: {document}
Current summary: {current_summary}
Instruction to focus on: {instruction}
Iteration: {iteration}
Generate an increasingly concise, entity-dense, and highly technical summary from the provided document that specifically addresses the given instruction.
"""
response = anthropic_client.messages.create(
model=model, max_tokens=4096, messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text
@weave.op()
def iterative_density_summarization(
document: str,
instruction: str,
current_summary: str,
density_iterations: int,
model: str = "claude-3-sonnet-20240229",
):
iteration_summaries = []
for iteration in range(1, density_iterations + 1):
current_summary = summarize_current_summary(
document, instruction, current_summary, iteration, model
)
iteration_summaries.append(current_summary)
return current_summary, iteration_summaries
@weave.op()
def final_summary(
instruction: str, current_summary: str, model: str = "claude-3-sonnet-20240229"
):
prompt = f"""
Given this summary: {current_summary}
And this instruction to focus on: {instruction}
Create an extremely dense, final summary that captures all key technical information in the most concise form possible, while specifically addressing the given instruction.
"""
return (
anthropic_client.messages.create(
model=model, max_tokens=4096, messages=[{"role": "user", "content": prompt}]
)
.content[0]
.text
)
@weave.op()
def chain_of_density_summarization(
document: str,
instruction: str,
current_summary: str = "",
model: str = "claude-3-sonnet-20240229",
density_iterations: int = 2,
):
current_summary, iteration_summaries = iterative_density_summarization(
document, instruction, current_summary, density_iterations, model
)
final_summary_text = final_summary(instruction, current_summary, model)
return {
"final_summary": final_summary_text,
"accumulated_summary": current_summary,
"iteration_summaries": iteration_summaries,
}
```
Here's what each function does:
* `summarize_current_summary`: Generates a single summary iteration based on the current state.
* `iterative_density_summarization`: Applies the CoD technique by calling `summarize_current_summary` multiple times.
* `chain_of_density_summarization`: Orchestrates the entire summarization process and returns the results.
The `@weave.op()` decorators ensure that Weave tracks the inputs, outputs, and execution of these functions.
## Create a Weave Model
With the summarization functions in place, the next step is to package them as a Weave Model so that runs, parameters, and versions are tracked together. Now, wrap the summarization pipeline in a Weave Model:
```python lines theme={"system"}
# Weave Model
class ArxivChainOfDensityPipeline(weave.Model):
model: str = "claude-3-sonnet-20240229"
density_iterations: int = 3
@weave.op()
def predict(self, paper: ArxivPaper, instruction: str) -> dict:
text = load_pdf(paper.pdf_url)
result = chain_of_density_summarization(
text,
instruction,
model=self.model,
density_iterations=self.density_iterations,
)
return result
```
This `ArxivChainOfDensityPipeline` class encapsulates the summarization logic as a Weave Model, providing several key benefits:
* Automatic experiment tracking: Weave captures inputs, outputs, and parameters for each run of the model.
* Versioning: Changes to the model's attributes or code are automatically versioned, creating a clear history of how your summarization pipeline evolves over time.
* Reproducibility: The versioning and tracking let you reproduce any previous result or configuration of your summarization pipeline.
* Hyperparameter management: Model attributes (like `model` and `density_iterations`) are clearly defined and tracked across different runs, facilitating experimentation.
* Integration with Weave ecosystem: Using `weave.Model` works with other Weave tools, such as evaluations and serving capabilities.
## Implement evaluation metrics
With the pipeline producing summaries, you need a way to measure their quality systematically. To assess the quality of the summaries, implement simple evaluation metrics:
```python lines theme={"system"}
import json
@weave.op()
def evaluate_summary(
summary: str, instruction: str, model: str = "claude-3-sonnet-20240229"
) -> dict:
prompt = f"""
Summary: {summary}
Instruction: {instruction}
Evaluate the summary based on the following criteria:
1. Relevance (1-5): How well does the summary address the given instruction?
2. Conciseness (1-5): How concise is the summary while retaining key information?
3. Technical Accuracy (1-5): How accurately does the summary convey technical details?
Your response MUST be in the following JSON format:
{{
"relevance": {{
"score": ,
"explanation": ""
}},
"conciseness": {{
"score": ,
"explanation": ""
}},
"technical_accuracy": {{
"score": ,
"explanation": ""
}}
}}
Ensure that the scores are integers between 1 and 5, and that the explanations are concise.
"""
response = anthropic_client.messages.create(
model=model, max_tokens=1000, messages=[{"role": "user", "content": prompt}]
)
print(response.content[0].text)
eval_dict = json.loads(response.content[0].text)
return {
"relevance": eval_dict["relevance"]["score"],
"conciseness": eval_dict["conciseness"]["score"],
"technical_accuracy": eval_dict["technical_accuracy"]["score"],
"average_score": sum(eval_dict[k]["score"] for k in eval_dict) / 3,
"evaluation_text": response.content[0].text,
}
```
These evaluation functions use the Claude model to assess the quality of the generated summaries based on relevance, conciseness, and technical accuracy.
## Create a Weave Dataset and run evaluation
Now that the scoring function is defined, the final step is to apply it to sample inputs and execute the evaluation. To evaluate the pipeline, create a Weave Dataset and run an evaluation:
```python lines theme={"system"}
# Create a Weave Dataset
dataset = weave.Dataset(
name="arxiv_papers",
rows=[
{
"paper": arxiv_paper,
"instruction": "What was the approach to experimenting with different data mixtures?",
},
],
)
weave.publish(dataset)
```
For the evaluation, use an LLM-as-a-judge approach. This technique involves using a language model to assess the quality of outputs generated by another model or system. It uses the LLM's understanding and reasoning capabilities to provide nuanced evaluations, especially for tasks where traditional metrics may be insufficient.
[](https://arxiv.org/abs/2306.05685)
```python lines theme={"system"}
# Define the scorer function
@weave.op()
def quality_scorer(instruction: str, output: dict) -> dict:
result = evaluate_summary(output["final_summary"], instruction)
return result
```
```python lines theme={"system"}
# Run evaluation
evaluation = weave.Evaluation(dataset=dataset, scorers=[quality_scorer])
arxiv_chain_of_density_pipeline = ArxivChainOfDensityPipeline()
results = await evaluation.evaluate(arxiv_chain_of_density_pipeline)
```
This code creates a dataset with the sample ArXiv paper, defines a quality scorer, and runs an evaluation of the summarization pipeline.
## Conclusion
This example demonstrated how to implement a Chain of Density summarization pipeline for ArXiv papers using Weave. You learned how to:
* Create Weave operations for each step of the summarization process
* Wrap the pipeline in a Weave Model for tracking and evaluation
* Implement custom evaluation metrics using Weave operations
* Create a dataset and run an evaluation of the pipeline
Weave tracks inputs, outputs, and intermediate steps throughout the summarization process, making it easier to debug, optimize, and evaluate your LLM application.
You can extend this example to handle larger datasets, implement more sophisticated evaluation metrics, or integrate with other LLM workflows.
[View Full Report on Weights & Biases](https://forge.coreweave.com/wandb/wandb_fc/arxiv-reader/reports/Building-a-bot-to-summarize-arXiv-papers-as-PDFs-using-Anthrophic-and-W-B-Weave--Vmlldzo4Nzg0ODI4)
# Codegen
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/codegen
Build and evaluate code generation pipelines with W&B Weave to trace prompts, outputs, and quality metrics.
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the following links:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/codegen.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/codegen.ipynb)
Generating high-quality code with proper structure, documentation, and tests is a challenging task. This guide is for developers who want to build an LLM-powered code generation workflow and systematically measure its quality. This notebook demonstrates how to create a code-generation pipeline that produces Python functions evaluated against the HumanEval test suite. The pipeline uses Weave for evaluation comparison and tracking and OpenAI's GPT models for code generation with structured outputs.
## Why use Weave
This tutorial uses Weave to implement and evaluate a code generation pipeline. You learn how to:
* **Track your LLM pipeline**: Log inputs, outputs, and intermediate steps of your code generation process.
* **Evaluate LLM outputs**: Create and compare evaluations of your generated code with debugging tools and visualizations.
## Set up the environment
Set up your environment and import the necessary libraries. These dependencies provide the formatting tools, dataset loader, and OpenAI and Weave clients used throughout the pipeline.
```python lines theme={"system"}
!pip install -qU autopep8 autoflake weave isort openai set-env-colab-kaggle-dotenv datasets
python
%%capture
# Temporary workaround to fix bug in openai:
# TypeError: Client.__init__() got an unexpected keyword argument 'proxies'
# See https://community.openai.com/t/error-with-openai-1-56-0-client-init-got-an-unexpected-keyword-argument-proxies/1040332/15
!pip install "httpx<0.28"
python
import ast
import os
import re
import subprocess
import tempfile
import traceback
import autopep8
import isort
from autoflake import fix_code
from datasets import load_dataset
from openai import OpenAI
from pydantic import BaseModel
from set_env import set_env
import weave
from weave import Dataset, Evaluation
set_env("WANDB_API_KEY")
set_env("OPENAI_API_KEY")
python
WEAVE_PROJECT = "codegen-cookbook-example"
weave.init(WEAVE_PROJECT)
python
client = OpenAI()
python
human_eval = load_dataset("openai_humaneval")
selected_examples = human_eval["test"][:3]
```
> [!NOTE]
>
> Weave automatically tracks OpenAI API calls, including inputs, outputs, and metadata. You don't need to add any additional logging code for your OpenAI interactions. Weave handles it in the background.
## Structured outputs and Pydantic models
This code generation pipeline uses OpenAI's [structured outputs mode](https://platform.openai.com/docs/guides/structured-outputs) and Pydantic models to produce consistent and well-formatted responses from the language model. This approach offers several advantages:
* **Type safety**: Defining Pydantic models for the expected outputs enforces a strict structure for the generated code, program runners, and unit tests.
* **Easier parsing**: The structured output mode parses the model's response directly into the predefined Pydantic models, reducing the need for complex post-processing.
* **Improved reliability**: Specifying the exact format you expect reduces the likelihood of unexpected or malformed outputs from the language model.
The following example defines Pydantic models and uses them with OpenAI's structured outputs:
```python lines theme={"system"}
class GeneratedCode(BaseModel):
function_signature: str
function_args_with_docstring_within_triple_quotes: str
code_logic: str
class FormattedGeneratedCode(BaseModel):
full_code: str
```
## Implement a code formatter
To produce consistent and clean code output, implement a `CodeFormatter` class using Weave operations. This formatter applies linting and styling rules to the generated code, program runner, and unit tests.
```python lines theme={"system"}
class CodeFormatter(BaseModel):
@weave.op()
def lint_code(self, code: str) -> str:
# Replace escaped newlines with actual newlines
code = code.replace("\\n", "\n")
# Remove unused imports and variables
code = fix_code(
code, remove_all_unused_imports=True, remove_unused_variables=True
)
# Sort imports
code = isort.code(code)
# Apply PEP 8 formatting
code = autopep8.fix_code(code, options={"aggressive": 2})
return code
@weave.op()
def add_imports(self, code: str) -> str:
tree = ast.parse(code)
from_imports = {}
global_names = set()
for node in ast.walk(tree):
if isinstance(node, ast.Name) and node.id not in dir(__builtins__):
global_names.add(node.id)
# Only add typing imports that are actually used
typing_imports = global_names.intersection(
{"List", "Dict", "Tuple", "Set", "Optional", "Union"}
)
if typing_imports:
from_imports["typing"] = typing_imports
# Remove names that are defined within the function
function_def = next(
node for node in tree.body if isinstance(node, ast.FunctionDef)
)
local_names = {arg.arg for arg in function_def.args.args}
local_names.update(
node.id
for node in ast.walk(function_def)
if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store)
)
global_names -= local_names
global_names -= {"sorted"} # Remove built-in functions
# Construct the import statements
import_statements = []
for module, names in from_imports.items():
names_str = ", ".join(sorted(names))
import_statements.append(f"from {module} import {names_str}")
return (
"\n".join(import_statements) + ("\n\n" if import_statements else "") + code
)
@weave.op()
def format_generated_code(
self, generated_code: GeneratedCode
) -> FormattedGeneratedCode:
# Combine the code parts
full_code = f"{generated_code.function_signature}\n{generated_code.function_args_with_docstring_within_triple_quotes}\n{generated_code.code_logic}"
# Ensure proper indentation
lines = full_code.split("\n")
indented_lines = []
for i, line in enumerate(lines):
if i == 0: # Function signature
indented_lines.append(line)
elif i == 1: # Function arguments (docstring)
indented_lines.append(" " + line)
else: # Function body
indented_lines.append(" " + line)
full_code = "\n".join(indented_lines)
# Lint the code
full_code = self.lint_code(full_code)
# Add imports
cleaned_code = self.add_imports(full_code)
return FormattedGeneratedCode(full_code=cleaned_code)
```
This `CodeFormatter` class provides several Weave operations to clean and format the generated code:
* Replacing escaped newlines with actual newlines.
* Removing unused imports and variables.
* Sorting imports.
* Applying PEP 8 formatting.
* Adding missing imports.
## Define the `CodeGenerationPipeline`
With the formatter in place, the next step is to implement the core code generation logic that ties the prompt, the LLM call, and the formatter together.
This example uses a `weave.Model` so that the model is automatically versioned when it changes. The `model_name` is kept as an attribute so that you can experiment with it and diff and compare it in Weave. Function calls are tracked with `@weave.op` so the inputs and outputs are logged to help with error tracking and debugging.
```python lines theme={"system"}
class CodeGenerationPipeline(weave.Model):
model_name: str
formatter: CodeFormatter
def __init__(
self, model_name: str = "gpt-4o", formatter: CodeFormatter | None = None
):
if formatter is None:
formatter = CodeFormatter()
super().__init__(model_name=model_name, formatter=formatter)
self.model_name = model_name
self.formatter = formatter
@weave.op()
async def predict(self, prompt: str):
generated_code = self.generate_code(prompt)
formatted_generated_code = self.formatter.format_generated_code(generated_code)
return formatted_generated_code.full_code
@weave.op()
def generate_code(self, prompt: str) -> GeneratedCode:
completion = client.beta.chat.completions.parse(
model=self.model_name,
messages=[
{
"role": "system",
"content": "You are an expert Python code generator.",
},
{"role": "user", "content": prompt},
],
response_format=GeneratedCode,
)
message = completion.choices[0].message
if message.parsed:
return message.parsed
else:
raise ValueError(message.refusal)
```
This `CodeGenerationPipeline` class encapsulates the code generation logic as a Weave Model, providing several benefits:
* Automatic experiment tracking: Weave captures inputs, outputs, and parameters for each run of the model.
* Versioning: Changes to the model's attributes or code are automatically versioned, creating a history of how your code generation pipeline evolves over time.
* Reproducibility: The versioning and tracking let you reproduce any previous result or configuration of your code generation pipeline.
* Hyperparameter management: Model attributes (like `model_name`) are defined and tracked across different runs, facilitating experimentation.
* Integration with Weave ecosystem: Using `weave.Model` connects your pipeline to other Weave tools, such as evaluations and serving capabilities.
## Implement evaluation metrics
To assess the quality of the generated code, implement evaluation metrics using a `weave.Scorer` subclass. This runs `score` on every `model_output` from the dataset. `model_output` comes from the output of the `predict` function in the `weave.Model`. `prompt` is taken from the `human-eval` dataset.
```python lines theme={"system"}
CODE_TEMPLATE = """
{model_output}
{test}
if __name__ == "__main__":
check({entry_point})
"""
python
@weave.op()
async def score_humaneval_test(test: str, entry_point: str, output: str):
generated_code = output
# Extract test cases from the test string
test_cases = re.findall(r"assert.*", test)
test_cases_str = "\n ".join(test_cases)
# Generate the full source code
full_code = CODE_TEMPLATE.format(
model_output=generated_code,
test=test,
test_cases=test_cases_str,
entry_point=entry_point,
)
# Create a temporary file to store the code
with tempfile.NamedTemporaryFile(delete=False, suffix=".py") as tmp_file:
# Write the generated code to the temporary file
tmp_file.write(full_code.encode())
tmp_file_path = tmp_file.name
try:
# Run the temporary Python file as a subprocess with a timeout
result = subprocess.run(
["python", tmp_file_path],
capture_output=True,
text=True,
timeout=10, # Timeout of 10 seconds
)
print(result)
if result.returncode == 0:
return {"correct": True}
else:
return {"correct": False, "error": result.stderr, "output": result.stdout}
except subprocess.TimeoutExpired:
return {"correct": False, "error": "TimeoutExpired"}
except Exception as e:
return {"correct": False, "error": traceback.format_exc()}
finally:
# Ensure the temporary file is removed after execution
os.remove(tmp_file_path)
```
These evaluation functions run the generated code and return a boolean value indicating whether the code passed the test provided from the dataset.
## Create a Weave Dataset and run evaluation
With the pipeline and scorer defined, the final step is to assemble the evaluation dataset and run it end-to-end. To evaluate the pipeline, create a Weave Dataset and run an evaluation:
```python lines theme={"system"}
formatted_selected_examples = [
{
"task_id": task_id,
"prompt": prompt,
"canonical_solution": solution,
"test": test,
"entry_point": entry_point,
}
for task_id, prompt, solution, test, entry_point in zip(
selected_examples["task_id"],
selected_examples["prompt"],
selected_examples["canonical_solution"],
selected_examples["test"],
selected_examples["entry_point"],
)
]
python
prompt_dataset = Dataset(
name="humaneval_code_gen_example",
rows=[
{
"prompt": example["prompt"],
"test": example["test"],
"entry_point": example["entry_point"],
}
for example in formatted_selected_examples
],
)
weave.publish(prompt_dataset)
python
EVAL_RUN = True
python
for model_name in ["gpt-4o-2024-08-06"]:
pipeline = CodeGenerationPipeline(model_name=model_name)
if not EVAL_RUN:
dataset = prompt_dataset.rows[2]
result = await pipeline.predict(dataset["prompt"])
score_result = await score_humaneval_test(
dataset["test"], dataset["entry_point"], result["generated_code"].full_code
)
else:
evaluation = Evaluation(
name="minimal_code_gen_evaluation",
dataset=prompt_dataset,
scorers=[score_humaneval_test],
)
results = await evaluation.evaluate(pipeline)
```
This code creates a dataset with the sample prompts, defines the HumanEval test scorer, and runs an evaluation of the code generation pipeline. After the evaluation completes, results are available in the Weights & Biases UI for inspection and comparison across runs.
## Conclusion
This example demonstrates how to implement a code generation pipeline using Weave and OpenAI's language models. You learned how to:
* Create Weave operations for each step of the code generation process.
* Wrap the pipeline in a Weave Model for streamlined tracking and evaluation.
* Implement custom evaluation metrics using Weave operations.
* Create a dataset and run an evaluation of the pipeline.
Weave tracks inputs, outputs, and intermediate steps throughout the code generation process, making it easier to debug, optimize, and evaluate your LLM application.
For more information about Weave and its capabilities, see the [Weave documentation](/products/wandb/weave). You can extend this example to handle larger datasets, implement more sophisticated evaluation metrics, or integrate with other LLM workflows.
# DSPy Prompt Optimization
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/dspy_prompt_optimization
Learn how to use dspy prompt optimization with W&B Weave
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the following links:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/dspy_prompt_optimization.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/dspy_prompt_optimization.ipynb)
The [BIG-bench (Beyond the Imitation Game Benchmark)](https://github.com/google/BIG-bench) is a collaborative benchmark that probes large language models and extrapolates their future capabilities across more than 200 tasks. The [BIG-Bench Hard (BBH)](https://github.com/suzgunmirac/BIG-Bench-Hard) is a suite of the 23 most challenging BIG-Bench tasks that the current generation of language models can find difficult to solve.
This tutorial demonstrates how to improve the performance of an LLM workflow implemented on the causal judgement task from the BIG-bench Hard benchmark and evaluate prompting strategies. You use [DSPy](https://dspy.ai) to implement the LLM workflow and optimize the prompting strategy. You also use [Weave](/products/wandb/weave) to track the LLM workflow and evaluate prompting strategies.
By the end of this tutorial, you have built a baseline DSPy program for causal reasoning, evaluated it with Weave, applied a DSPy optimizer to improve its prompting strategy, and compared the optimized program against the baseline. This tutorial is intended for practitioners who want to apply prompt optimization to their own LLM workflows and use Weave to track and compare results.
## Install the dependencies
Before starting, install the libraries used throughout the tutorial. This tutorial uses the following libraries:
* [DSPy](https://dspy.ai) to build and optimize the LLM workflow.
* [Weave](/products/wandb/weave) to track the LLM workflow and evaluate prompting strategies.
* [datasets](https://huggingface.co/docs/datasets/index) to access the BIG-Bench Hard dataset from HuggingFace Hub.
```python lines theme={"system"}
!pip install -qU dspy weave "datasets<4"
```
Because this tutorial uses the [OpenAI API](https://openai.com/index/openai-api/) as the LLM vendor, you also need an OpenAI API key. You can [sign up](https://platform.openai.com/signup) on the OpenAI platform to get your own API key.
```python lines theme={"system"}
import os
from getpass import getpass
api_key = getpass("Enter you OpenAI API key: ")
os.environ["OPENAI_API_KEY"] = api_key
```
## Enable tracking using Weave
This section configures Weave so that subsequent DSPy calls in the tutorial are automatically traced and viewable in the Weights & Biases UI.
Weave integrates with DSPy. Including [`weave.init`](/products/wandb/weave/reference/python-sdk/trace/weave_client#method-init) at the start of your code automatically traces your DSPy functions, which you can then explore in the Weights & Biases UI. For more information, see the [Weave integration docs for DSPy](/products/wandb/weave/guides/integrations/dspy).
```python lines theme={"system"}
import weave
weave.init(project_name="dspy-bigbench-hard")
```
This tutorial uses a metadata class inherited from [`weave.Object`](/products/wandb/weave/reference/python-sdk#class-object) to manage metadata.
```python lines theme={"system"}
class Metadata(weave.Object):
dataset_address: str = "maveriq/bigbenchhard"
big_bench_hard_task: str = "causal_judgement"
num_train_examples: int = 50
openai_model: str = "gpt-4o-mini"
openai_max_tokens: int = 2048
max_bootstrapped_demos: int = 8
max_labeled_demos: int = 8
metadata = Metadata()
```
> [!TIP]
>
> Object versioning: The `Metadata` objects are automatically versioned and traced when functions consuming them are traced.
## Load the BIG-Bench Hard dataset
With Weave tracking enabled, the next step is to prepare the data used to train and evaluate the DSPy program.
Load this dataset from HuggingFace Hub, split it into training and validation sets, and [publish](/products/wandb/weave/guides/core-types/datasets) them on Weave. Publishing lets you version the datasets and also use [`weave.Evaluation`](/products/wandb/weave/guides/core-types/evaluations) to evaluate your prompting strategy.
```python lines theme={"system"}
import dspy
from datasets import load_dataset
@weave.op()
def get_dataset(metadata: Metadata):
# load the BIG-Bench Hard dataset corresponding to the task from Huggingface Hug
dataset = load_dataset(metadata.dataset_address, metadata.big_bench_hard_task)[
"train"
]
# create the training and validation datasets
rows = [{"question": data["input"], "answer": data["target"]} for data in dataset]
train_rows = rows[0 : metadata.num_train_examples]
val_rows = rows[metadata.num_train_examples :]
# create the training and validation examples consisting of `dspy.Example` objects
dspy_train_examples = [
dspy.Example(row).with_inputs("question") for row in train_rows
]
dspy_val_examples = [dspy.Example(row).with_inputs("question") for row in val_rows]
# publish the datasets to the Weave, this would let us version the data and use for evaluation
weave.publish(
weave.Dataset(
name=f"bigbenchhard_{metadata.big_bench_hard_task}_train", rows=train_rows
)
)
weave.publish(
weave.Dataset(
name=f"bigbenchhard_{metadata.big_bench_hard_task}_val", rows=val_rows
)
)
return dspy_train_examples, dspy_val_examples
dspy_train_examples, dspy_val_examples = get_dataset(metadata)
```
## The DSPy program
With the dataset published to Weave, you can now define the baseline DSPy program that you later evaluate and optimize.
[DSPy](https://dspy.ai) is a framework that pushes building new LM pipelines away from manipulating free-form strings and closer to programming (composing modular operators to build text transformation graphs), where a compiler automatically generates optimized LM invocation strategies and prompts from a program.
Use [`dspy.LM`](https://dspy.ai/learn/programming/language_models) to configure the language model and [`dspy.configure`](https://dspy.ai/api/utils/configure/) to set it as the default.
```python lines theme={"system"}
llm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=llm)
```
### Write the causal reasoning signature
A [signature](https://dspy.ai/learn/programming/signatures) is a declarative specification of input/output behavior of a [DSPy module](https://dspy.ai/learn/programming/modules). DSPy modules are task-adaptive components (akin to neural network layers) that abstract any particular text transformation.
```python lines theme={"system"}
class CausalReasoning(dspy.Signature):
"""You are an expert in causal reasoning. Analyze the given question carefully
and answer Yes or No. Provide a detailed explanation justifying your answer."""
question: str = dspy.InputField(desc="The question to be answered")
answer: str = dspy.OutputField(desc="Yes or No")
confidence: float = dspy.OutputField(desc="Confidence score between 0 and 1")
explanation: str = dspy.OutputField(desc="Detailed explanation for the answer")
class CausalReasoningModule(dspy.Module):
def __init__(self):
self.prog = dspy.Predict(CausalReasoning)
@weave.op()
def forward(self, question: str) -> dict:
result = self.prog(question=question)
return {
"answer": result.answer,
"confidence": result.confidence,
"explanation": result.explanation,
}
```
Test the LLM workflow (that is, the `CausalReasoningModule`) on an example from the causal reasoning subset of BIG-Bench Hard.
```python lines theme={"system"}
import rich
baseline_module = CausalReasoningModule()
prediction = baseline_module(dspy_train_examples[0]["question"])
rich.print(prediction)
```
## Evaluate the DSPy program
Now that you have a baseline prompting strategy, evaluate it on the validation set using [`weave.Evaluation`](/products/wandb/weave/guides/core-types/evaluations) with a metric that matches the predicted answer with the ground truth. Weave takes each example, passes it through your application, and scores the output on multiple custom scoring functions. This gives you a view of the performance of your application, and a rich UI to drill into individual outputs and scores.
First, create a scoring function that determines whether the predicted answer matches the ground truth. Weave scoring functions receive the model's return value as `output` and any matching keys from the dataset example as additional arguments. Here, `answer` comes from the dataset and `output` is the dict returned by `CausalReasoningModule.forward`.
```python lines theme={"system"}
@weave.op()
def weave_evaluation_scorer(answer: str, output: dict) -> dict:
return {"match": int(answer.lower() == output["answer"].lower())}
```
Next, wrap the module in a traced function that `weave.Evaluation` can call. The wrapper's argument names must match the dataset column names that the model consumes.
```python lines theme={"system"}
@weave.op()
def predict(question: str) -> dict:
return baseline_module(question=question)
```
Now you can define the evaluation and run it.
```python lines theme={"system"}
validation_dataset = weave.ref(
f"bigbenchhard_{metadata.big_bench_hard_task}_val:v0"
).get()
evaluation = weave.Evaluation(
name="baseline_causal_reasoning_module",
dataset=validation_dataset,
scorers=[weave_evaluation_scorer],
)
await evaluation.evaluate(predict)
```
> [!NOTE]
>
> If you're running from a Python script, you can use the following code to run the evaluation:
>
> ```python lines theme={"system"}
> import asyncio
> asyncio.run(evaluation.evaluate(predict))
> ```
> [!WARNING]
>
> Running the evaluation causal reasoning dataset costs approximately \$0.24 in OpenAI credits.
## Optimize the DSPy program
With the baseline performance measured, you can now apply a DSPy optimizer and compare the result to the baseline.
Now that you have a baseline DSPy program, improve its performance for causal reasoning using the [BootstrapFewShot](https://dspy.ai/api/optimizers/BootstrapFewShot/) optimizer, which can tune the parameters of a DSPy program to maximize the specified metrics.
```python lines theme={"system"}
from dspy.teleprompt import BootstrapFewShot
@weave.op()
def get_optimized_program(model: dspy.Module, metadata: Metadata) -> dspy.Module:
@weave.op()
def dspy_evaluation_metric(true, prediction, trace=None):
return prediction["answer"].lower() == true.answer.lower()
teleprompter = BootstrapFewShot(
metric=dspy_evaluation_metric,
max_bootstrapped_demos=metadata.max_bootstrapped_demos,
max_labeled_demos=metadata.max_labeled_demos,
)
return teleprompter.compile(model, trainset=dspy_train_examples)
optimized_module = get_optimized_program(baseline_module, metadata)
```
> [!WARNING]
>
> Running the evaluation causal reasoning dataset costs approximately \$0.04 in OpenAI credits.
Now that you have the optimized program (the optimized prompting strategy), evaluate it again on the validation set and compare it with the baseline DSPy program.
```python lines theme={"system"}
@weave.op()
def predict_optimized(question: str) -> dict:
return optimized_module(question=question)
evaluation = weave.Evaluation(
name="optimized_causal_reasoning_module",
dataset=validation_dataset,
scorers=[weave_evaluation_scorer],
)
await evaluation.evaluate(predict_optimized)
```
Comparing the evaluation of the baseline program with the optimized one shows that the optimized program answers the causal reasoning questions with more accuracy.
## Conclusion
In this tutorial, you learned how to use DSPy for prompt optimization alongside Weave for tracking and evaluation to compare the original and optimized programs.
# HuggingFace Dataset Evaluations
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/hf_dataset_evals
Learn how to use HuggingFace dataset evaluations with W&B Weave
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the links below:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/hf_dataset_evals.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/hf_dataset_evals.ipynb)
This notebook shows you how to use a HuggingFace dataset as the input source for a Weave `Evaluation`. By the end, you'll have a working evaluation pipeline that references rows in a HuggingFace dataset by index, preprocesses each row into the format your model expects, and tracks the scored results in Weave. This pattern is useful when you want to evaluate models against existing HuggingFace datasets without first converting them into a Weave-native dataset.
> [!NOTE]
>
> This guide demonstrates a workaround for using HuggingFace datasets with Weave evaluations. The approach works today, and more seamless integrations for external datasets are in development.
## Setup and imports
First, initialize Weave and connect to Weights & Biases so that Weave tracks the evaluation runs and their results in your Weave project.
```python lines theme={"system"}
!pip install datasets wandb weave
# Initialize variables
HUGGINGFACE_DATASET = "wandb/ragbench-test-sample"
WANDB_KEY = ""
WEAVE_TEAM = ""
WEAVE_PROJECT = ""
# Init weave and required libraries
import asyncio
import nest_asyncio
import wandb
from datasets import load_dataset
import weave
from weave import Evaluation
# Login to wandb and initialize weave
wandb.login(key=WANDB_KEY)
client = weave.init(f"{WEAVE_TEAM}/{WEAVE_PROJECT}")
# Apply nest_asyncio to allow nested event loops (needed for some notebook environments)
nest_asyncio.apply()
```
## Load and prepare HuggingFace dataset
Next, load the HuggingFace dataset and build a lightweight index that the evaluation iterates over. Rather than passing the dataset rows directly to Weave, pass a list of index references and resolve them to full rows during preprocessing. This approach keeps the evaluation tied to the original HuggingFace dataset and maintains references to it.
> [!NOTE]
>
> In the index, encode the `hf_hub_name` along with the `hf_id` to ensure each row has a unique identifier. Weave uses this unique digest value to track and reference specific dataset entries during evaluations.
```python lines theme={"system"}
# Load the HuggingFace dataset
ds = load_dataset(HUGGINGFACE_DATASET)
row_count = ds["train"].num_rows
# Create an index mapping for the dataset
# This creates a list of dictionaries with HF dataset indices
# Example: [{"hf_id": 0}, {"hf_id": 1}, {"hf_id": 2}, ...]
hf_index = [{"hf_id": i, "hf_hub_name": HUGGINGFACE_DATASET} for i in range(row_count)]
```
## Define processing and evaluation functions
With the index in place, define the three functions that make up the evaluation pipeline: one to turn each index reference into a usable example, one to score the model's output, and one representing the model being evaluated.
The processing pipeline uses the following functions:
* `preprocess_example`: Transforms the index reference into the actual data needed for evaluation.
* `hf_eval`: Defines how to score the model outputs.
* `function_to_evaluate`: The actual function or model being evaluated.
```python lines theme={"system"}
@weave.op()
def preprocess_example(example):
"""
Preprocesses each example before evaluation.
Args:
example: Dict containing hf_id
Returns:
Dict containing the prompt from the HF dataset
"""
hf_row = ds["train"][example["hf_id"]]
return {"prompt": hf_row["question"], "answer": hf_row["response"]}
@weave.op()
def hf_eval(hf_id: int, output: dict) -> dict:
"""
Scoring function for evaluating model outputs.
Args:
hf_id: Index in the HF dataset
output: The output from the model to evaluate
Returns:
Dict containing evaluation scores
"""
hf_row = ds["train"][hf_id]
return {"scorer_value": True}
@weave.op()
def function_to_evaluate(prompt: str):
"""
The function that will be evaluated (e.g., your model or pipeline).
Args:
prompt: Input prompt from the dataset
Returns:
Dict containing model output
"""
return {"generated_text": "testing "}
```
## Create and run the evaluation
Finally, wire the index, scorer, and preprocessing function into a Weave `Evaluation` and run it against the model. For each entry in `hf_index`, Weave performs the following steps:
1. `preprocess_example` gets the corresponding data from the HuggingFace dataset.
2. Weave passes the preprocessed data to `function_to_evaluate`.
3. `hf_eval` scores the output.
4. Weave tracks the results.
After the evaluation finishes, the run and its per-row scores are available in your Weave project for inspection.
```python lines theme={"system"}
# Create evaluation object
evaluation = Evaluation(
dataset=hf_index, # Use our index mapping
scorers=[hf_eval], # List of scoring functions
preprocess_model_input=preprocess_example, # Function to prepare inputs
)
# Run evaluation asynchronously
async def main():
await evaluation.evaluate(function_to_evaluate)
asyncio.run(main())
```
# Import from CSV
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/import_from_csv
Import datasets from CSV files into W&B Weave for use in evaluations, tracing, and model comparison workflows.
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the following links:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/import_from_csv.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/import_from_csv.ipynb)
## Import traces from third-party systems
This notebook shows you how to import historical conversation traces from a CSV file into W\&B Weave so you can analyze them, compare model behavior, and run evaluations on data that was generated outside of a Weave-instrumented application.
Sometimes you can't instrument your Python or JavaScript code with the Weave integration to obtain real-time traces of your GenAI application. Often, these traces are available to you later in CSV or JSON format.
This notebook uses the lower-level Weave Python API to extract data from a CSV file and import it into Weave so you can analyze and evaluate it.
The sample dataset assumed in this cookbook has the following structure:
```text theme={"system"}
conversation_id,turn_index,start_time,user_input,ground_truth,answer_text
1234,1,2024-09-04 13:05:39,This is the beginning, ['This was the beginning'], That was the beginning
1235,1,2024-09-04 13:02:11,This is another trace,, That was another trace
1235,2,2024-09-04 13:04:19,This is the next turn,, That was the next turn
1236,1,2024-09-04 13:02:10,This is a 3 turn conversation,, Woah thats a lot of turns
1236,2,2024-09-04 13:02:30,This is the second turn, ['That was definitely the second turn'], You are correct
1236,3,2024-09-04 13:02:53,This is the end,, Well good riddance!
```
To understand the decisions for import in this notebook, remember that Weave traces have parent-child relationships that are 1:Many and continuous. A single parent can have multiple children, and that parent can itself be a child of another parent.
This notebook uses `conversation_id` as the parent identifier and `turn_index` as the child identifier to provide complete conversation logging.
You must modify the variables in the following sections to match your own dataset, file paths, and Weights & Biases project.
## Set up the environment
Install and import all needed packages.
Set `WANDB_API_KEY` in your environment so that you can log in with `wandb.login()` (provide this to Colab as a secret).
Set the name of the file you upload to Colab in `name_of_file`, and set the Weights & Biases project you want to log into in `name_of_wandb_project`.
> [!NOTE]
>
> `name_of_wandb_project` can also be in the format `[TEAM_NAME]/[PROJECT_NAME]` to specify a team to log the traces into.
Then, fetch a Weave client by calling `weave.init()`.
```python lines theme={"system"}
%pip install wandb weave pandas datetime --quiet
python
import os
import pandas as pd
import wandb
from google.colab import userdata
import weave
## Write samples file to disk
with open("/content/import_cookbook_data.csv", "w") as f:
f.write(
"conversation_id,turn_index,start_time,user_input,ground_truth,answer_text\n"
)
f.write(
'1234,1,2024-09-04 13:05:39,This is the beginning, ["This was the beginning"], That was the beginning\n'
)
f.write(
"1235,1,2024-09-04 13:02:11,This is another trace,, That was another trace\n"
)
f.write(
"1235,2,2024-09-04 13:04:19,This is the next turn,, That was the next turn\n"
)
f.write(
"1236,1,2024-09-04 13:02:10,This is a 3 turn conversation,, Woah thats a lot of turns\n"
)
f.write(
'1236,2,2024-09-04 13:02:30,This is the second turn, ["That was definitely the second turn"], You are correct\n'
)
f.write("1236,3,2024-09-04 13:02:53,This is the end,, Well good riddance!\n")
os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
name_of_file = "/content/import_cookbook_data.csv"
name_of_wandb_project = "import-weave-traces-cookbook"
wandb.login()
python
weave_client = weave.init(name_of_wandb_project)
```
## Load the data
With the environment ready, you can load and shape the CSV data so that it matches the parent-child structure that Weave expects.
Load the data into a pandas DataFrame, and sort it by `conversation_id` and `turn_index` to ensure the parents and children are correctly ordered.
This results in a two-column pandas DataFrame with the conversation turns as an array under `conversation_data`.
```python lines theme={"system"}
## Load data and shape it
df = pd.read_csv(name_of_file)
sorted_df = df.sort_values(["conversation_id", "turn_index"])
# Function to create an array of dictionaries for each conversation
def create_conversation_dict_array(group):
return group.drop("conversation_id", axis=1).to_dict("records")
# Group the dataframe by conversation_id and apply the aggregation
result_df = (
sorted_df.groupby("conversation_id")
.apply(create_conversation_dict_array)
.reset_index()
)
result_df.columns = ["conversation_id", "conversation_data"]
# Show how our aggregation looks
result_df.head()
```
## Log the traces to Weave
With the data shaped into conversations and turns, the next step is to write those records into Weave as parent and child calls.
Iterate through the pandas DataFrame:
* Create a parent call for every `conversation_id`.
* Iterate through the turn array to create child calls sorted by their `turn_index`.
Important concepts of the lower-level Python API:
* A Weave call is equivalent to a Weave trace. This call can have a parent or children associated with it.
* A Weave call can have other things associated with it, such as feedback and metadata. This example only associates inputs and outputs, but you can add these other items in your import if the data provides them.
* A Weave call is `created` and `finished` because these are meant to be tracked in real time. Because this is an after-the-fact import, you create and finish once the objects are defined and tied to one another.
* The `op` value of a call is how Weave categorizes calls of the same makeup. In this example, all parent calls are of `Conversation` type, and all child calls are of `Turn` type. You can modify this as you see fit.
* A call can have `inputs` and `output`. `inputs` are defined at creation, and `output` is defined when the call is finished.
```python lines theme={"system"}
# Log traces to weave
# Iterate through our aggregated conversations
for _, row in result_df.iterrows():
# Define our conversation parent,
# we are now creating a "call" with the weave_client we defined before
parent_call = weave_client.create_call(
# The Op value will register this as a Weave Op, which will allow us to retrieve these as a group easily in the future
op="Conversation",
# We set the inputs of our high level conversation as all the turns under it
inputs={
"conversation_data": row["conversation_data"][:-1]
if len(row["conversation_data"]) > 1
else row["conversation_data"]
},
# Our Conversation parent does not have a further parent
parent=None,
# The name of how this specific conversation will appear in the UI
display_name=f"conversation-{row['conversation_id']}",
)
# We set the output of the parent to be the last trace in the conversation
parent_output = row["conversation_data"][len(row["conversation_data"]) - 1]
# We now iterate through all the conversation turns for the parent
# and log them as children of the conversation
for item in row["conversation_data"]:
item_id = f"{row['conversation_id']}-{item['turn_index']}"
# We create a call again here to be categorized under the conversation
call = weave_client.create_call(
# We qualify a single conversation trace as a "Turn"
op="Turn",
# We provide all inputs of the turn, including RAG 'ground_truth'
inputs={
"turn_index": item["turn_index"],
"start_time": item["start_time"],
"user_input": item["user_input"],
"ground_truth": item["ground_truth"],
},
# We set this to be a child of the parent we defined
parent=parent_call,
# We provide it a name to be id'ed by in Weave
display_name=item_id,
)
# We set the output of the call as the answer
output = {
"answer_text": item["answer_text"],
}
# Because these are traces that already happened, we finish the single turn call
weave_client.finish_call(call=call, output=output)
# Now that we have logged all its children, we also finish the parent call
weave_client.finish_call(call=parent_call, output=parent_output)
```
## Result: traces logged to Weave
At this point, your CSV data has been imported into Weave. You can now browse the conversations and their turns in the Weights & Biases UI, grouped under the `Conversation` and `Turn` operations you defined.
Traces:
Operations:
## Optional: export your traces to run evaluations
Once the traces are in Weave and you understand how the conversations look, you can export them to another process to run Weave Evaluations.
To do this, fetch all conversations from Weights & Biases through the query API and create a dataset from them.
```python lines theme={"system"}
## This cell does not run by default, comment the below line to execute this script
%%script false --no-raise-error
## Get all Conversation traces for evaluation and prepare dataset for eval
# We create a query filter that brings us all our Conversation objects
# The ref shown below is specific to your project, and you can obtain it by
# going into your project's Operations in the UI, clicking on the "Conversations"
# object, then the "Use" tab in the side panel.
weave_ref_for_conversation_op = "weave://wandb-smle/import-weave-traces-cookbook/op/Conversation:tzUhDyzVm5bqQsuqh5RT4axEXSosyLIYZn9zbRyenaw"
filter = weave.trace_server.trace_server_interface.CallsFilter(
op_names=[weave_ref_for_conversation_op],
)
# We execute the query
conversation_traces = weave_client.get_calls(filter=filter)
rows = []
# We go through our conversation traces and construct dataset rows from it
for single_conv in conversation_traces:
# In this example, we may only care for conversations that utilized our RAG
# pipeline, so we filter for such types of conversations
is_rag = False
for single_trace in single_conv.inputs['conversation_data']:
if single_trace['ground_truth'] is not None:
is_rag = True
break
if single_conv.output['ground_truth'] is not None:
is_rag = True
# Once we've identified a converation to have used RAG, we add it to our dataset
if is_rag:
inputs = []
ground_truths = []
answers = []
# We go through every turn in the conversation
for turn in single_conv.inputs['conversation_data']:
inputs.append(turn.get('user_input', ''))
ground_truths.append(turn.get('ground_truth', ''))
answers.append(turn.get('answer_text', ''))
## Account for when conversations are a single turn
if len(single_conv.inputs) != 1 or single_conv.inputs['conversation_data'][0].get('turn_index') != single_conv.output.get('turn_index'):
inputs.append(single_conv.output.get('user_input', ''))
ground_truths.append(single_conv.output.get('ground_truth', ''))
answers.append(single_conv.output.get('answer_text', ''))
data = {
'question': inputs,
'contexts': ground_truths,
'answer': answers
}
rows.append(data)
# With our dataset rows created, we create the Dataset object and
# publish it back to Weave for later retrieval
dset = weave.Dataset(name = "conv_traces_for_eval", rows=rows)
weave.publish(dset)
```
## Result
The exported dataset is now published back to Weave and is ready to use as input for an evaluation.
To learn more about evaluations, see the [Quickstart](/products/wandb/weave/tutorial-rag) on using your newly created dataset to evaluate your RAG application.
# Leaderboard Quickstart
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/leaderboard_quickstart
Learn how to use leaderboard quickstart with W&B Weave
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the links below:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/leaderboard_quickstart.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/leaderboard_quickstart.ipynb)
This quickstart shows you how to use the W\&B Weave Leaderboard to compare model performance across multiple datasets and scoring functions. By the end, you will have published a leaderboard that ranks several models against a shared set of evaluations. You can then identify which model performs best on each metric. This guide is for developers familiar with running Weave evaluations who want to compare results side by side.
Specifically, you will:
1. Generate a dataset of fake zip code data.
2. Author some scoring functions and evaluate a baseline model.
3. Use these techniques to evaluate a matrix of models versus evaluations.
4. Review the leaderboard in the Weights & Biases UI.
## Step 1: Generate a dataset of fake zip code data
First, create a function `generate_dataset_rows` that generates a list of fake zip code data. This synthetic dataset gives the leaderboard a consistent set of inputs and expected values to score each model against.
```python lines theme={"system"}
import json
from openai import OpenAI
from pydantic import BaseModel
class Row(BaseModel):
zip_code: str
city: str
state: str
avg_temp_f: float
population: int
median_income: int
known_for: str
class Rows(BaseModel):
rows: list[Row]
def generate_dataset_rows(
location: str = "United States", count: int = 5, year: int = 2022
):
client = OpenAI()
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{
"role": "user",
"content": f"Please generate {count} rows of data for random zip codes in {location} for the year {year}.",
},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "response_format",
"schema": Rows.model_json_schema(),
},
},
)
return json.loads(completion.choices[0].message.content)["rows"]
python
import weave
weave.init("leaderboard-demo")
```
## Step 2: Author scoring functions
Next, author three scoring functions. Each scorer evaluates a different aspect of the model output so the leaderboard can rank models on distinct dimensions of quality:
1. `check_concrete_fields`: Checks if the model output matches the expected city and state.
2. `check_value_fields`: Checks if the model output is within 10% of the expected population and median income.
3. `check_subjective_fields`: Uses an LLM to check if the model output matches the expected "known for" field.
```python lines theme={"system"}
@weave.op
def check_concrete_fields(city: str, state: str, output: dict):
return {
"city_match": city == output["city"],
"state_match": state == output["state"],
}
@weave.op
def check_value_fields(
avg_temp_f: float, population: int, median_income: int, output: dict
):
return {
"avg_temp_f_err": abs(avg_temp_f - output["avg_temp_f"]) / avg_temp_f,
"population_err": abs(population - output["population"]) / population,
"median_income_err": abs(median_income - output["median_income"])
/ median_income,
}
@weave.op
def check_subjective_fields(zip_code: str, known_for: str, output: dict):
client = OpenAI()
class Response(BaseModel):
correct_known_for: bool
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{
"role": "user",
"content": f"My student was asked what the zip code {zip_code} is best known best for. The right answer is '{known_for}', and they said '{output['known_for']}'. Is their answer correct?",
},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "response_format",
"schema": Response.model_json_schema(),
},
},
)
return json.loads(completion.choices[0].message.content)
```
## Step 3: Create an evaluation
Next, define an evaluation using the fake data and scoring functions. The `Evaluation` object pairs the dataset with the scorers, so you can run any model against the same benchmark.
```python lines theme={"system"}
rows = generate_dataset_rows()
evaluation = weave.Evaluation(
name="United States - 2022",
dataset=rows,
scorers=[
check_concrete_fields,
check_value_fields,
check_subjective_fields,
],
)
```
## Step 4: Evaluate a baseline model
Now, evaluate a baseline model that returns a static response. Establishing a baseline gives you a reference point on the leaderboard so you can measure how much each subsequent model improves over a static implementation.
```python lines theme={"system"}
@weave.op
def baseline_model(zip_code: str):
return {
"city": "New York",
"state": "NY",
"avg_temp_f": 50.0,
"population": 1000000,
"median_income": 100000,
"known_for": "The Big Apple",
}
await evaluation.evaluate(baseline_model)
```
## Step 5: Create more models
Now, create two more models to compare against the baseline. One model receives the zip code with no additional prompting, while the other receives a structured prompt. Comparing them on the leaderboard shows the impact of prompt context on answer quality.
```python lines theme={"system"}
@weave.op
def gpt_4o_mini_no_context(zip_code: str):
client = OpenAI()
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": f"""Zip code {zip_code}"""}],
response_format={
"type": "json_schema",
"json_schema": {
"name": "response_format",
"schema": Row.model_json_schema(),
},
},
)
return json.loads(completion.choices[0].message.content)
await evaluation.evaluate(gpt_4o_mini_no_context)
python
@weave.op
def gpt_4o_mini_with_context(zip_code: str):
client = OpenAI()
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "user",
"content": f"""Please answer the following questions about the zip code {zip_code}:
1. What is the city?
2. What is the state?
3. What is the average temperature in Fahrenheit?
4. What is the population?
5. What is the median income?
6. What is the most well known thing about this zip code?
""",
}
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "response_format",
"schema": Row.model_json_schema(),
},
},
)
return json.loads(completion.choices[0].message.content)
await evaluation.evaluate(gpt_4o_mini_with_context)
```
## Step 6: Create more evaluations
Now, evaluate a matrix of models versus evaluations. Running every model against several datasets (different regions and years) produces the data the leaderboard needs to rank models across multiple conditions.
```python lines theme={"system"}
scorers = [
check_concrete_fields,
check_value_fields,
check_subjective_fields,
]
evaluations = [
weave.Evaluation(
name="United States - 2022",
dataset=weave.Dataset(
name="United States - 2022",
rows=generate_dataset_rows("United States", 5, 2022),
),
scorers=scorers,
),
weave.Evaluation(
name="California - 2022",
dataset=weave.Dataset(
name="California - 2022", rows=generate_dataset_rows("California", 5, 2022)
),
scorers=scorers,
),
weave.Evaluation(
name="United States - 2000",
dataset=weave.Dataset(
name="United States - 2000",
rows=generate_dataset_rows("United States", 5, 2000),
),
scorers=scorers,
),
]
models = [
baseline_model,
gpt_4o_mini_no_context,
gpt_4o_mini_with_context,
]
for evaluation in evaluations:
for model in models:
await evaluation.evaluate(
model, __weave={"display_name": evaluation.name + ":" + model.__name__}
)
```
## Step 7: Review the leaderboard
With the evaluation results published, you can now assemble them into a leaderboard for side-by-side comparison.
You can create a new leaderboard by navigating to the leaderboard tab in the UI and clicking **Create Leaderboard**.
You can also generate a leaderboard directly from Python:
```python lines theme={"system"}
from weave.flow import leaderboard
from weave.trace.ref_util import get_ref
spec = leaderboard.Leaderboard(
name="Zip Code World Knowledge",
description="""
This leaderboard compares the performance of models in terms of world knowledge about zip codes.
### Columns
1. **State Match against `United States - 2022`**: The fraction of zip codes that the model correctly identified the state for.
2. **Avg Temp F Error against `California - 2022`**: The mean absolute error of the model's average temperature prediction.
3. **Correct Known For against `United States - 2000`**: The fraction of zip codes that the model correctly identified the most well known thing about the zip code.
""",
columns=[
leaderboard.LeaderboardColumn(
evaluation_object_ref=get_ref(evaluations[0]).uri(),
scorer_name="check_concrete_fields",
summary_metric_path="state_match.true_fraction",
),
leaderboard.LeaderboardColumn(
evaluation_object_ref=get_ref(evaluations[1]).uri(),
scorer_name="check_value_fields",
should_minimize=True,
summary_metric_path="avg_temp_f_err.mean",
),
leaderboard.LeaderboardColumn(
evaluation_object_ref=get_ref(evaluations[2]).uri(),
scorer_name="check_subjective_fields",
summary_metric_path="correct_known_for.true_fraction",
),
],
)
ref = weave.publish(spec)
```
You now have a published leaderboard in Weave that ranks each model against the three evaluations and scoring metrics you defined. From the Weights & Biases UI, you can inspect per-model scores, drill into individual evaluation runs, and compare future model iterations against the same baseline.
# Multi-agent structured output
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/multi-agent-structured-output
Learn how to use multi-agent structured output with W&B Weave
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the following links:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/multi-agent-structured-output.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/multi-agent-structured-output.ipynb)
This tutorial demonstrates how to build a multi-agent system that uses OpenAI's structured outputs feature, and how to trace agent interactions with [Weave](/products/wandb/weave). By the end, you have a four-agent data analysis pipeline whose intermediate inputs and outputs are visible in the Weights & Biases UI.
OpenAI released [Structured Outputs](https://openai.com/index/introducing-structured-outputs-in-the-api/) so the model always generates responses that adhere to your supplied JSON Schema without strongly worded prompts. With Structured Outputs, you don't need to validate or retry incorrectly formatted responses.
By using the parameter `strict: true`, you can guarantee the response abides by a provided schema.
Structured outputs in a multi-agent system produce consistent, predictably processed data between agents. They also support explicit refusals and remove the need for retries or response validation.
> [!TIP]
>
> **Source**: This cookbook is based on [sample code from OpenAI's structured outputs](https://cookbook.openai.com/examples/structured_outputs_multi_agent), with some modifications added for improved visualization using Weave.
## Install the dependencies
First, install the libraries the tutorial depends on. This tutorial uses the following libraries:
* [OpenAI](https://openai.com/index/openai-api/) to create the multi-agent system.
* [Weave](/products/wandb/weave) to track the LLM workflow and evaluate prompting strategies.
```python lines theme={"system"}
!pip install -qU openai weave wandb
python
%%capture
# Temporary workaround to fix bug in openai:
# TypeError: Client.__init__() got an unexpected keyword argument 'proxies'
# See https://community.openai.com/t/error-with-openai-1-56-0-client-init-got-an-unexpected-keyword-argument-proxies/1040332/15
!pip install "httpx<0.28"
```
Next, configure credentials and initialize Weave so traces are logged to your Weights & Biases project.
Set `WANDB_API_KEY` in your environment so you can log in with `wandb.login()`. Provide this to Colab as a secret.
Set the Weights & Biases project to log into in `name_of_wandb_project`.
> [!NOTE]
>
> `name_of_wandb_project` may also be in the format of `[YOUR-TEAM]/[YOUR-PROJECT]` to specify a team to log the traces into.
Then fetch a Weave client by calling `weave.init()`.
Because this tutorial uses the [OpenAI API](https://openai.com/index/openai-api/), you also need an OpenAI API key. [Sign up](https://platform.openai.com/signup) on the OpenAI platform to get your own API key. Provide this to Colab as a secret too.
```python lines theme={"system"}
import base64
import json
import os
from io import BytesIO, StringIO
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import wandb
from google.colab import userdata
from openai import OpenAI
import weave
python
os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
os.environ["OPENAI_API_KEY"] = userdata.get("OPENAI_API_KEY")
wandb.login()
name_of_wandb_project = "multi-agent-structured-output"
weave.init(name_of_wandb_project)
client = OpenAI()
MODEL = "gpt-4o-2024-08-06"
```
## Set up the agents
With Weave initialized, define the agents that make up the system. The use case for this tutorial is a data analysis task.
First, set up the four-agent system:
* Triaging agent: Decides which agents to call.
* Data pre-processing agent: Prepares data for analysis, for example by cleaning it up.
* Data analysis agent: Performs analysis on the data.
* Data visualization agent: Visualizes the output of the analysis to extract insights.
Start by defining the system prompts for each of these agents. These prompts establish each agent's role and the tools it's allowed to call.
```python lines theme={"system"}
triaging_system_prompt = """You are a Triaging Agent. Your role is to assess the user's query and route it to the relevant agents. The agents available are:
- Data Processing Agent: Cleans, transforms, and aggregates data.
- Analysis Agent: Performs statistical, correlation, and regression analysis.
- Visualization Agent: Creates bar charts, line charts, and pie charts.
Use the send_query_to_agents tool to forward the user's query to the relevant agents. Also, use the speak_to_user tool to get more information from the user if needed."""
processing_system_prompt = """You are a Data Processing Agent. Your role is to clean, transform, and aggregate data using the following tools:
- clean_data
- transform_data
- aggregate_data"""
analysis_system_prompt = """You are an Analysis Agent. Your role is to perform statistical, correlation, and regression analysis using the following tools:
- stat_analysis
- correlation_analysis
- regression_analysis"""
visualization_system_prompt = """You are a Visualization Agent. Your role is to create bar charts, line charts, and pie charts using the following tools:
- create_bar_chart
- create_line_chart
- create_pie_chart"""
```
Next, define the tools for each agent.
Apart from the triaging agent, each agent has tools specific to its role:
**Data pre-processing agent**: Clean data, transform data, and aggregate data.
**Data analysis agent**: Statistical analysis, correlation analysis, and regression analysis.
**Data visualization agent**: Create bar chart, create line chart, and create pie chart.
```python lines theme={"system"}
triage_tools = [
{
"type": "function",
"function": {
"name": "send_query_to_agents",
"description": "Sends the user query to relevant agents based on their capabilities.",
"parameters": {
"type": "object",
"properties": {
"agents": {
"type": "array",
"items": {"type": "string"},
"description": "An array of agent names to send the query to.",
},
"query": {
"type": "string",
"description": "The user query to send.",
},
},
"required": ["agents", "query"],
},
},
"strict": True,
}
]
preprocess_tools = [
{
"type": "function",
"function": {
"name": "clean_data",
"description": "Cleans the provided data by removing duplicates and handling missing values.",
"parameters": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The dataset to clean. Should be in a suitable format such as JSON or CSV.",
}
},
"required": ["data"],
"additionalProperties": False,
},
},
"strict": True,
},
{
"type": "function",
"function": {
"name": "transform_data",
"description": "Transforms data based on specified rules.",
"parameters": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The data to transform. Should be in a suitable format such as JSON or CSV.",
},
"rules": {
"type": "string",
"description": "Transformation rules to apply, specified in a structured format.",
},
},
"required": ["data", "rules"],
"additionalProperties": False,
},
},
"strict": True,
},
{
"type": "function",
"function": {
"name": "aggregate_data",
"description": "Aggregates data by specified columns and operations.",
"parameters": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The data to aggregate. Should be in a suitable format such as JSON or CSV.",
},
"group_by": {
"type": "array",
"items": {"type": "string"},
"description": "Columns to group by.",
},
"operations": {
"type": "string",
"description": "Aggregation operations to perform, specified in a structured format.",
},
},
"required": ["data", "group_by", "operations"],
"additionalProperties": False,
},
},
"strict": True,
},
]
analysis_tools = [
{
"type": "function",
"function": {
"name": "stat_analysis",
"description": "Performs statistical analysis on the given dataset.",
"parameters": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The dataset to analyze. Should be in a suitable format such as JSON or CSV.",
}
},
"required": ["data"],
"additionalProperties": False,
},
},
"strict": True,
},
{
"type": "function",
"function": {
"name": "correlation_analysis",
"description": "Calculates correlation coefficients between variables in the dataset.",
"parameters": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The dataset to analyze. Should be in a suitable format such as JSON or CSV.",
},
"variables": {
"type": "array",
"items": {"type": "string"},
"description": "List of variables to calculate correlations for.",
},
},
"required": ["data", "variables"],
"additionalProperties": False,
},
},
"strict": True,
},
{
"type": "function",
"function": {
"name": "regression_analysis",
"description": "Performs regression analysis on the dataset.",
"parameters": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The dataset to analyze. Should be in a suitable format such as JSON or CSV.",
},
"dependent_var": {
"type": "string",
"description": "The dependent variable for regression.",
},
"independent_vars": {
"type": "array",
"items": {"type": "string"},
"description": "List of independent variables.",
},
},
"required": ["data", "dependent_var", "independent_vars"],
"additionalProperties": False,
},
},
"strict": True,
},
]
visualization_tools = [
{
"type": "function",
"function": {
"name": "create_bar_chart",
"description": "Creates a bar chart from the provided data.",
"parameters": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The data for the bar chart. Should be in a suitable format such as JSON or CSV.",
},
"x": {"type": "string", "description": "Column for the x-axis."},
"y": {"type": "string", "description": "Column for the y-axis."},
},
"required": ["data", "x", "y"],
"additionalProperties": False,
},
},
"strict": True,
},
{
"type": "function",
"function": {
"name": "create_line_chart",
"description": "Creates a line chart from the provided data.",
"parameters": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The data for the line chart. Should be in a suitable format such as JSON or CSV.",
},
"x": {"type": "string", "description": "Column for the x-axis."},
"y": {"type": "string", "description": "Column for the y-axis."},
},
"required": ["data", "x", "y"],
"additionalProperties": False,
},
},
"strict": True,
},
{
"type": "function",
"function": {
"name": "create_pie_chart",
"description": "Creates a pie chart from the provided data.",
"parameters": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The data for the pie chart. Should be in a suitable format such as JSON or CSV.",
},
"labels": {
"type": "string",
"description": "Column for the labels.",
},
"values": {
"type": "string",
"description": "Column for the values.",
},
},
"required": ["data", "labels", "values"],
"additionalProperties": False,
},
},
"strict": True,
},
]
```
## Enable multi-agent tracking with Weave
With the agents and their tools defined, the next step is to wire them together and enable Weave tracing. Write the code logic to:
* Handle passing the user query to the multi-agent system.
* Handle the internal workings of the multi-agent system.
* Execute the tool calls.
```python lines theme={"system"}
# Example query
user_query = """
Below is some data. I want you to first remove the duplicates then analyze the statistics of the data as well as plot a line chart.
house_size (m3), house_price ($)
90, 100
80, 90
100, 120
90, 100
"""
```
From the user query, you can infer that the tools to call are `clean_data`, `start_analysis`, and `use_line_chart`.
Begin by defining the execution function responsible for running tool calls.
By decorating Python functions with `@weave.op()`, you can log and debug language model inputs, outputs, and traces.
A multi-agent system involves many functions, but it's sufficient to add `@weave.op()` on top of each one.
```python lines theme={"system"}
@weave.op()
def clean_data(data):
data_io = StringIO(data)
df = pd.read_csv(data_io, sep=",")
df_deduplicated = df.drop_duplicates()
return df_deduplicated
@weave.op()
def stat_analysis(data):
data_io = StringIO(data)
df = pd.read_csv(data_io, sep=",")
return df.describe()
@weave.op()
def plot_line_chart(data):
data_io = StringIO(data)
df = pd.read_csv(data_io, sep=",")
x = df.iloc[:, 0]
y = df.iloc[:, 1]
coefficients = np.polyfit(x, y, 1)
polynomial = np.poly1d(coefficients)
y_fit = polynomial(x)
plt.figure(figsize=(10, 6))
plt.plot(x, y, "o", label="Data Points")
plt.plot(x, y_fit, "-", label="Best Fit Line")
plt.title("Line Chart with Best Fit Line")
plt.xlabel(df.columns[0])
plt.ylabel(df.columns[1])
plt.legend()
plt.grid(True)
# Save the plot to a BytesIO buffer before showing it
buf = BytesIO()
plt.savefig(buf, format="png")
buf.seek(0)
# Display the plot
plt.show()
# Encode the image in base64 for the data URL
image_data = buf.getvalue()
base64_encoded_data = base64.b64encode(image_data)
base64_string = base64_encoded_data.decode("utf-8")
data_url = f"data:image/png;base64,{base64_string}"
return data_url
# Define the function to execute the tools
@weave.op()
def execute_tool(tool_calls, messages):
for tool_call in tool_calls:
tool_name = tool_call.function.name
tool_arguments = json.loads(tool_call.function.arguments)
if tool_name == "clean_data":
# Simulate data cleaning
cleaned_df = clean_data(tool_arguments["data"])
cleaned_data = {"cleaned_data": cleaned_df.to_dict()}
messages.append(
{"role": "tool", "name": tool_name, "content": json.dumps(cleaned_data)}
)
print("Cleaned data: ", cleaned_df)
elif tool_name == "transform_data":
# Simulate data transformation
transformed_data = {"transformed_data": "sample_transformed_data"}
messages.append(
{
"role": "tool",
"name": tool_name,
"content": json.dumps(transformed_data),
}
)
elif tool_name == "aggregate_data":
# Simulate data aggregation
aggregated_data = {"aggregated_data": "sample_aggregated_data"}
messages.append(
{
"role": "tool",
"name": tool_name,
"content": json.dumps(aggregated_data),
}
)
elif tool_name == "stat_analysis":
# Simulate statistical analysis
stats_df = stat_analysis(tool_arguments["data"])
stats = {"stats": stats_df.to_dict()}
messages.append(
{"role": "tool", "name": tool_name, "content": json.dumps(stats)}
)
print("Statistical Analysis: ", stats_df)
elif tool_name == "correlation_analysis":
# Simulate correlation analysis
correlations = {"correlations": "sample_correlations"}
messages.append(
{"role": "tool", "name": tool_name, "content": json.dumps(correlations)}
)
elif tool_name == "regression_analysis":
# Simulate regression analysis
regression_results = {"regression_results": "sample_regression_results"}
messages.append(
{
"role": "tool",
"name": tool_name,
"content": json.dumps(regression_results),
}
)
elif tool_name == "create_bar_chart":
# Simulate bar chart creation
bar_chart = {"bar_chart": "sample_bar_chart"}
messages.append(
{"role": "tool", "name": tool_name, "content": json.dumps(bar_chart)}
)
elif tool_name == "create_line_chart":
# Simulate line chart creation
line_chart = {"line_chart": plot_line_chart(tool_arguments["data"])}
messages.append(
{"role": "tool", "name": tool_name, "content": json.dumps(line_chart)}
)
elif tool_name == "create_pie_chart":
# Simulate pie chart creation
pie_chart = {"pie_chart": "sample_pie_chart"}
messages.append(
{"role": "tool", "name": tool_name, "content": json.dumps(pie_chart)}
)
return messages
```
Next, create the tool handlers for each of the sub-agents. These have a unique prompt and tool set passed to the model. The output is then passed to an execution function that runs the tool calls.
```python lines theme={"system"}
# Define the functions to handle each agent's processing
@weave.op()
def handle_data_processing_agent(query, conversation_messages):
messages = [{"role": "system", "content": processing_system_prompt}]
messages.append({"role": "user", "content": query})
response = client.chat.completions.create(
model=MODEL,
messages=messages,
temperature=0,
tools=preprocess_tools,
)
conversation_messages.append(
[tool_call.function for tool_call in response.choices[0].message.tool_calls]
)
execute_tool(response.choices[0].message.tool_calls, conversation_messages)
@weave.op()
def handle_analysis_agent(query, conversation_messages):
messages = [{"role": "system", "content": analysis_system_prompt}]
messages.append({"role": "user", "content": query})
response = client.chat.completions.create(
model=MODEL,
messages=messages,
temperature=0,
tools=analysis_tools,
)
conversation_messages.append(
[tool_call.function for tool_call in response.choices[0].message.tool_calls]
)
execute_tool(response.choices[0].message.tool_calls, conversation_messages)
@weave.op()
def handle_visualization_agent(query, conversation_messages):
messages = [{"role": "system", "content": visualization_system_prompt}]
messages.append({"role": "user", "content": query})
response = client.chat.completions.create(
model=MODEL,
messages=messages,
temperature=0,
tools=visualization_tools,
)
conversation_messages.append(
[tool_call.function for tool_call in response.choices[0].message.tool_calls]
)
execute_tool(response.choices[0].message.tool_calls, conversation_messages)
```
Finally, create the overarching tool to handle processing the user query. This function takes the user query, gets a response from the model, and passes it to the other agents to execute.
```python lines theme={"system"}
# Function to handle user input and triaging
@weave.op()
def handle_user_message(user_query, conversation_messages=None):
if conversation_messages is None:
conversation_messages = []
user_message = {"role": "user", "content": user_query}
conversation_messages.append(user_message)
messages = [{"role": "system", "content": triaging_system_prompt}]
messages.extend(conversation_messages)
response = client.chat.completions.create(
model=MODEL,
messages=messages,
temperature=0,
tools=triage_tools,
)
conversation_messages.append(
[tool_call.function for tool_call in response.choices[0].message.tool_calls]
)
for tool_call in response.choices[0].message.tool_calls:
if tool_call.function.name == "send_query_to_agents":
agents = json.loads(tool_call.function.arguments)["agents"]
query = json.loads(tool_call.function.arguments)["query"]
for agent in agents:
if agent == "Data Processing Agent":
handle_data_processing_agent(query, conversation_messages)
elif agent == "Analysis Agent":
handle_analysis_agent(query, conversation_messages)
elif agent == "Visualization Agent":
handle_visualization_agent(query, conversation_messages)
outputs = extract_tool_contents(conversation_messages)
return outputs
functions = [
"clean_data",
"transform_data",
"stat_analysis",
"aggregate_data",
"correlation_analysis",
"regression_analysis",
"create_bar_chart",
"create_line_chart",
"create_pie_chart",
]
@weave.op()
def extract_tool_contents(data):
contents = {}
contents["all"] = data
for element in data:
if (
isinstance(element, dict)
and element.get("role") == "tool"
and element.get("name") in functions
):
name = element["name"]
content_str = element["content"]
try:
content_json = json.loads(content_str)
if "chart" not in element.get("name"):
contents[name] = [content_json]
else:
first_key = next(iter(content_json))
second_level = content_json[first_key]
if isinstance(second_level, dict):
second_key = next(iter(second_level))
contents[name] = second_level[second_key]
else:
contents[name] = second_level
except json.JSONDecodeError:
print(f"Error decoding JSON for {name}")
contents[name] = None
return contents
```
## Execute the multi-agent system and visualize in Weave
With every agent, tool, and handler in place, the system is ready to run. Finally, execute the primary `handle_user_message` function using the user's input and observe the results.
```python lines theme={"system"}
handle_user_message(user_query)
```
When you click the Weave URL, you can see the execution trace. On the **Traces** page, you can check the input and output. For clarity, the diagram includes screenshots of the results displayed when you click each output. Weave integrates with the OpenAI API and automatically calculates costs. Cost and latency are also displayed alongside each trace.
Click a line to see the intermediate processes that ran within the multi-agent system. For example, the input and output of the `analysis_agent` appear in a structured output format. OpenAI's structured outputs help agents collaborate, but as the system grows more complex, the format of these interactions becomes harder to follow. Weave lets you inspect these intermediate processes and their inputs and outputs in detail.
Take a closer look at how tracing is handled in Weave.
## Conclusion
In this tutorial, you learned how to develop a multi-agent system using OpenAI's structured output and Weave for tracking inputs, final outputs, and intermediate output formats. You now have a working example you can extend with additional agents, tools, or structured response schemas.
# Not Diamond custom routing
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/notdiamond_custom_routing
Learn how to use Not Diamond custom routing with W&B Weave
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the following links:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/notdiamond_custom_routing.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/notdiamond_custom_routing.ipynb)
This notebook demonstrates how to use Weave with [Not Diamond's custom routing](https://docs.notdiamond.ai/docs/router-training-quickstart) to route LLM prompts to the most appropriate model based on evaluation results. By the end, you have trained a custom router for coding prompts, used it to select a target model for a new prompt, and evaluated its performance against the best individual model.
## Routing prompts
This section explains why custom routing is useful when working with multiple LLMs.
When building complex LLM workflows, you may need to prompt different models according to accuracy, cost, or call latency.
You can use [Not Diamond](https://www.notdiamond.ai/) to route prompts in these workflows to the right model for your needs, helping maximize accuracy while saving on model costs.
For any given distribution of data, rarely does a single model outperform every other model on every query. By combining multiple models into a "meta-model" that learns when to call each LLM, you can beat every individual model's performance and even drive down costs and latency.
## Custom routing
The following inputs are required to train a custom router for your prompts:
1. **A set of LLM prompts**: Prompts must be strings and should be representative of the prompts used in your application.
2. **LLM responses**: The responses from candidate LLMs for each input. Candidate LLMs can include both supported LLMs and your own custom models.
3. **Evaluation scores for responses to the inputs from candidate LLMs**: Scores are numbers, and can be any metric that fits your needs.
By submitting these to the Not Diamond API, you can then train a custom router tuned to each of your workflows.
## Set up the training data
In this section, you prepare the training and test data that the custom router learns from.
In practice, you use your own Evaluations to train a custom router. For this example notebook, however, you use LLM responses
for [the HumanEval dataset](https://github.com/openai/human-eval) to train a custom router for coding tasks.
Start by downloading the dataset prepared for this example, then parse LLM responses into `EvaluationResults` for each model. This split of training and test data is used in later sections to train the router and to evaluate its out-of-sample performance.
```python lines theme={"system"}
!curl -L "https://drive.google.com/uc?export=download&id=1q1zNZHioy9B7M-WRjsJPkfvFosfaHX38" -o humaneval.csv
python
import random
import weave
from weave.flow.dataset import Dataset
from weave.flow.eval import EvaluationResults
from weave.integrations.notdiamond.util import get_model_evals
pct_train = 0.8
pct_test = 1 - pct_train
# In practice, you will build an Evaluation on your dataset and call
# `evaluation.get_eval_results(model)`
model_evals = get_model_evals("./humaneval.csv")
model_train = {}
model_test = {}
for model, evaluation_results in model_evals.items():
n_results = len(evaluation_results.rows)
all_idxs = list(range(n_results))
train_idxs = random.sample(all_idxs, k=int(n_results * pct_train))
test_idxs = [idx for idx in all_idxs if idx not in train_idxs]
model_train[model] = EvaluationResults(
rows=weave.Table([evaluation_results.rows[idx] for idx in train_idxs])
)
model_test[model] = Dataset(
rows=weave.Table([evaluation_results.rows[idx] for idx in test_idxs])
)
print(
f"Found {len(train_idxs)} train rows and {len(test_idxs)} test rows for {model}."
)
```
## Train a custom router
With the `EvaluationResults` prepared, you can now submit them to Not Diamond to train a custom router. Make sure you have [created an account](https://app.notdiamond.ai/keys) and
[generated an API key](https://app.notdiamond.ai/keys), then insert your API key in the following code. The API key authenticates the training request.
```python lines theme={"system"}
import os
from weave.integrations.notdiamond.custom_router import train_router
api_key = os.getenv("NOTDIAMOND_API_KEY", "")
preference_id = train_router(
model_evals=model_train,
prompt_column="prompt",
response_column="actual",
language="en",
maximize=True,
api_key=api_key,
# Leave this commented out to train your first custom router
# Uncomment this to retrain your custom router in place
# preference_id=preference_id,
)
```
You can then follow the training process for your custom router through the Not Diamond app.
Once your custom router has finished training, you can use it to route your prompts.
```python lines theme={"system"}
from notdiamond import NotDiamond
import weave
weave.init("notdiamond-quickstart")
llm_configs = [
"anthropic/claude-3-5-sonnet-20240620",
"openai/gpt-4o-2024-05-13",
"google/gemini-1.5-pro-latest",
"openai/gpt-4-turbo-2024-04-09",
"anthropic/claude-3-opus-20240229",
]
client = NotDiamond(api_key=api_key, llm_configs=llm_configs)
new_prompt = (
"""
You are a helpful coding assistant. Using the provided function signature, write the implementation for the function
in Python. Write only the function. Do not include any other text.
from typing import List
def has_close_elements(numbers: List[float], threshold: float) -> bool:
"""
""" Check if in given list of numbers, are any two numbers closer to each other than
given threshold.
>>> has_close_elements([1.0, 2.0, 3.0], 0.5)
False
>>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
True
"""
"""
"""
)
session_id, routing_target_model = client.model_select(
messages=[{"role": "user", "content": new_prompt}],
preference_id=preference_id,
)
print(f"Session ID: {session_id}")
print(f"Target Model: {routing_target_model}")
```
This example also used Not Diamond's compatibility with Weave autotracing. You can see the results in the Weights & Biases UI.
## Evaluate your custom router
This section shows how to measure whether the custom router improves on the best individual model.
Once you've trained your custom router, you can evaluate its performance in one of the following ways:
* In-sample performance by submitting the training prompts.
* Out-of-sample performance by submitting new or held-out prompts.
In the following example, you submit the test set to the custom router to evaluate its performance.
```python lines theme={"system"}
from weave.integrations.notdiamond.custom_router import evaluate_router
eval_prompt_column = "prompt"
eval_response_column = "actual"
best_provider_model, nd_model = evaluate_router(
model_datasets=model_test,
prompt_column=eval_prompt_column,
response_column=eval_response_column,
api_key=api_key,
preference_id=preference_id,
)
python
@weave.op()
def is_correct(score: int, output: dict) -> dict:
# We hack score, since we already have model responses
return {"correct": score}
best_provider_eval = weave.Evaluation(
dataset=best_provider_model.model_results.to_dict(orient="records"),
scorers=[is_correct],
)
await best_provider_eval.evaluate(best_provider_model)
nd_eval = weave.Evaluation(
dataset=nd_model.model_results.to_dict(orient="records"), scorers=[is_correct]
)
await nd_eval.evaluate(nd_model)
```
In this instance, the Not Diamond "meta-model" routes prompts across several different models.
Training the custom router through Weave also runs evaluations and uploads results to the Weights & Biases UI. Once the custom router process completes, you can review the results in the Weights & Biases UI.
In the UI, you can see that the Not Diamond "meta-model" outperforms the best-performing model by routing prompts to other models with higher likelihood of answering the prompt accurately.
# Trace and Evaluate a Computer Vision Pipeline with Weave
Source: https://docs.coreweave.com/products/wandb/weave/cookbooks/ocr-pipeline
Learn how to use trace and evaluate a computer vision pipeline with weave with W&B Weave
> [!NOTE]
>
> This is an interactive notebook. You can run it locally or use the links below:
>
> * [Open in Google Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/ocr-pipeline.ipynb)
> * [View source on GitHub](https://github.com/wandb/examples/blob/main/colabs/weave/ocr-pipeline.ipynb)
This tutorial shows you how to build, trace, and evaluate a computer vision pipeline that performs named entity recognition (NER) on images of handwritten patient information. By the end, you'll have a working optical character recognition (OCR) pipeline backed by a vision-language model (VLM) and a W\&B Weave Evaluation that measures how accurately the pipeline extracts structured fields from images. This guide is for developers who want to use Weave to iterate on prompts and systematically measure the quality of multimodal extraction pipelines.
The following sections walk through five stages: creating and iterating on prompts, retrieving the dataset, building the NER pipeline, defining scorers, and running an evaluation.
## Prerequisites
Before you begin, install and import the required libraries, get your Forge API Key, and initialize your Weave project. Completing this step ensures that your environment can authenticate with Weights & Biases and log traces to your Weave project.
```python lines theme={"system"}
# Install the required dependencies
!pip install openai weave -q
python
import json
import os
from google.colab import userdata
from openai import OpenAI
import weave
python
# Get API Keys
os.environ["OPENAI_API_KEY"] = userdata.get(
"OPENAI_API_KEY"
) # please set the keys as collab environment secrets from the menu on the left
os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
# Set project name
# Replace the PROJECT value with your project name
PROJECT = "vlm-handwritten-ner"
# Initiatlize the Weave project
weave.init(PROJECT)
```
## Create and iterate on prompts with Weave
Good prompt engineering is critical to guiding the model to properly extract entities. In this section, you author an initial prompt, publish it to Weave so you can track changes over time, and then refine it with stricter validation rules.
First, create a basic prompt that gives the model the instructions on what to extract from the image data and how to format it. Then, store the prompt in Weave for tracking and iteration.
````python lines theme={"system"}
# Create your prompt object with Weave
prompt = """
Extract all readable text from this image. Format the extracted entities as a valid JSON.
Do not return any extra text, just the JSON. Do not include ```json```
Use the following format:
{"Patient Name": "James James","Date": "4/22/2025","Patient ID": "ZZZZZZZ123","Group Number": "3452542525"}
"""
system_prompt = weave.StringPrompt(prompt)
# Publish your prompt to Weave
weave.publish(system_prompt, name="NER-prompt")
````
Next, improve the prompt by adding more instructions and validation rules to help reduce errors in the output. Publishing the revised version under the same name lets Weave track the prompt as a new version so you can compare results across iterations.
````python lines theme={"system"}
better_prompt = """
You are a precision OCR assistant. Given an image of patient information, extract exactly these fields into a single JSON object (and nothing else):
- Patient Name
- Date (MM/DD/YYYY)
- Patient ID
- Group Number
Validation rules:
1. Date must match MM/DD/YY; if not, set Date to "".
2. Patient ID must be alphanumeric; if unreadable, set to "".
3. Always zero-pad months and days (e.g. "04/07/25").
4. Omit any markup, commentary, or code fences.
5. Return strictly valid JSON with only those four keys.
Do not return any extra text, just the JSON. Do not include ```json```
Example output:
{"Patient Name":"James James","Date":"04/22/25","Patient ID":"ZZZZZZZ123","Group Number":"3452542525"}
"""
# Edit the prompt
system_prompt = weave.StringPrompt(better_prompt)
# Publish the edited prompt to Weave
weave.publish(system_prompt, name="NER-prompt")
````
## Get the dataset
With a prompt in place, you next need input data to run through the pipeline. Retrieve the dataset of handwritten notes that serves as input for the OCR pipeline.
The images in the dataset are already `base64` encoded, which means that the LLM can use the data without any pre-processing.
```python lines theme={"system"}
# Retrieve the dataset from the following Weave project
dataset = weave.ref(
"weave://wandb-smle/vlm-handwritten-ner/object/NER-eval-dataset:G8MEkqWBtvIxPYAY23sXLvqp8JKZ37Cj0PgcG19dGjw"
).get()
# Access a specific example in the dataset
example_image = dataset.rows[3]["image_base64"]
# Display the example_image
from IPython.display import HTML, display
html = f'
'
display(HTML(html))
```
## Build the NER pipeline
Now that you have a prompt and a dataset, build the NER pipeline that connects them to the VLM. The pipeline consists of two functions:
* An `encode_image` function that takes a PIL image from the dataset and returns a `base64` encoded string representation of the image that can be passed to the VLM.
* An `extract_named_entities_from_image` function that takes an image and system prompt and returns the extracted entities from that image as described by the system prompt.
```python lines theme={"system"}
# Traceable function using GPT-4-Vision
def extract_named_entities_from_image(image_base64) -> dict:
# init LLM Client
client = OpenAI()
# Setup the instruction prompt
# You can optionally use a prompt stored in Weave withweave.ref("weave://wandb-smle/vlm-handwritten-ner/object/NER-prompt:FmCv4xS3RFU21wmNHsIYUFal3cxjtAkegz2ylM25iB8").get().content.strip()
prompt = better_prompt
response = client.responses.create(
model="gpt-4.1",
input=[
{
"role": "user",
"content": [
{"type": "input_text", "text": prompt},
{
"type": "input_image",
"image_url": image_base64,
},
],
}
],
)
return response.output_text
```
Now, create a function called `named_entity_recognation` that:
* Passes the image data to the NER pipeline.
* Returns correctly formatted JSON with the results.
Use the [`@weave.op()` decorator](/products/wandb/weave/reference/python-sdk/trace/op) to automatically track and trace function execution in the Weights & Biases UI.
Every time `named_entity_recognation` runs, the full trace results are visible in the Weights & Biases UI. To view the traces, navigate to the **Traces** tab of your Weave project.
```python lines theme={"system"}
# NER Function for evaluations
@weave.op()
def named_entity_recognation(image_base64, id):
result = {}
try:
# 1) call the vision op, get back a JSON string
output_text = extract_named_entities_from_image(image_base64)
# 2) parse JSON exactly once
result = json.loads(output_text)
print(f"Processed: {str(id)}")
except Exception as e:
print(f"Failed to process {str(id)}: {e}")
return result
```
Finally, run the pipeline over the dataset, and view the results. This step produces the model outputs that you evaluate in the next section.
The following code loops over the dataset and stores the results in a local file `processing_results.json`. The results are also viewable in the Weights & Biases UI.
```python lines theme={"system"}
# Output results
results = []
# loop over all images in the dataset
for row in dataset.rows:
result = named_entity_recognation(row["image_base64"], str(row["id"]))
result["image_id"] = str(row["id"])
results.append(result)
# Save all results to a JSON file
output_file = "processing_results.json"
with open(output_file, "w") as f:
json.dump(results, f, indent=2)
print(f"Results saved to: {output_file}")
```
You see something similar to the following in the **Traces** table in the Weights & Biases UI.
## Evaluate the pipeline using Weave
Now that you have created a pipeline to perform NER using a VLM, you can use Weave to systematically evaluate it and find out how well it performs. Evaluating the pipeline lets you measure extraction quality across the full dataset rather than relying on spot checks. For more information about Evaluations in Weave, see [Evaluations Overview](/products/wandb/weave/guides/core-types/evaluations).
A fundamental part of a Weave Evaluation is the [Scorer](/products/wandb/weave/guides/evaluation/scorers). Scorers evaluate AI outputs and return evaluation metrics. They take the AI's output, analyze it, and return a dictionary of results. Scorers can use your input data as reference if needed and can also output extra information, such as explanations or reasonings from the evaluation.
In this section, you create two scorers to evaluate the pipeline:
* Programatic scorer.
* LLM-as-a-judge scorer.
### Programatic scorer
The first Scorer is a deterministic check that runs without an LLM. The programmatic scorer, `check_for_missing_fields_programatically`, takes the model output (the output of the `named_entity_recognition` function), and identifies which `keys` are missing or empty in the results.
This check is useful for identifying samples where the model missed capturing any fields.
```python lines theme={"system"}
# Add weave.op() to track execution of the scorer
@weave.op()
def check_for_missing_fields_programatically(model_output):
# Required keys for every entry
required_fields = {"Patient Name", "Date", "Patient ID", "Group Number"}
for key in required_fields:
if (
key not in model_output
or model_output[key] is None
or str(model_output[key]).strip() == ""
):
return False # This entry has a missing or empty field
return True # All required fields are present and non-empty
```
### LLM-as-a-judge scorer
Because the programmatic scorer only catches missing or empty fields, you need a second scorer to check whether the extracted values match what appears in the image. In this step of the evaluation, you provide both the image data and the model's output to ensure the assessment reflects actual NER performance. The image content is explicitly referenced, not just the model output.
The scorer used for this step, `check_for_missing_fields_with_llm`, uses an LLM to perform scoring (specifically OpenAI's `gpt-4o`). As specified by the contents of the `eval_prompt`, `check_for_missing_fields_with_llm` outputs a `Boolean` value. If all fields match the information in the image and formatting is correct, the scorer returns `true`. If any field is missing, empty, incorrect, or mismatched, the result is `false`, and the scorer also returns a message explaining the problem.
```python lines theme={"system"}
# The system prompt for the LLM-as-a-judge
eval_prompt = """
You are an OCR validation system. Your role is to assess whether the structured text extracted from an image accurately reflects the information in that image.
Only validate the structured text and use the image as your source of truth.
Expected input text format:
{"Patient Name": "First Last", "Date": "04/23/25", "Patient ID": "131313JJH", "Group Number": "35453453"}
Evaluation criteria:
- All four fields must be present.
- No field should be empty or contain placeholder/malformed values.
- The "Date" should be in MM/DD/YY format (e.g., "04/07/25") (zero padding the date is allowed)
Scoring:
- Return: {"Correct": true, "Reason": ""} if **all fields** match the information in the image and formatting is correct.
- Return: {"Correct": false, "Reason": "EXPLANATION"} if **any** field is missing, empty, incorrect, or mismatched.
Output requirements:
- Respond with a valid JSON object only.
- "Correct" must be a JSON boolean: true or false (not a string or number).
- "Reason" must be a short, specific string indicating all the problem — e.g., "Patient Name mismatch", "Date not zero-padded", or "Missing Group Number".
- Do not return any additional explanation or formatting.
Your response must be exactly one of the following:
{"Correct": true, "Reason": null}
OR
{"Correct": false, "Reason": "EXPLANATION_HERE"}
"""
# Add weave.op() to track execution of the Scorer
@weave.op()
def check_for_missing_fields_with_llm(model_output, image_base64):
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "developer", "content": [{"text": eval_prompt, "type": "text"}]},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": image_base64,
},
},
{"type": "text", "text": str(model_output)},
],
},
],
response_format={"type": "json_object"},
)
response = json.loads(response.choices[0].message.content)
return response
```
## Run the evaluation
With both Scorers defined, you can now run the evaluation. Define an evaluation call that automatically loops over the `dataset` passed and logs the results together in the Weights & Biases UI.
The following code kicks off the evaluation and applies the two Scorers to every output from the NER pipeline. Results are visible in the **Evals** tab in the Weights & Biases UI.
```python lines theme={"system"}
evaluation = weave.Evaluation(
dataset=dataset,
scorers=[
check_for_missing_fields_with_llm,
check_for_missing_fields_programatically,
],
name="Evaluate_4.1_NER",
)
print(await evaluation.evaluate(named_entity_recognation))
```
When the previous code runs, Weave generates a link to the Evaluation table in the Weights & Biases UI. Follow the link to view the results and compare different iterations of the pipeline across models, prompts, and datasets of your choice. The Weights & Biases UI automatically creates a visualization like the following for your team.
# Quickstart: Set up custom agent observability
Source: https://docs.coreweave.com/products/wandb/weave/custom-agents-quickstart
Trace a multi-turn agent with the Weave SDK. Conversations, turns, LLM calls, and tool calls render in the Agents view of your project.
> [!TIP]
>
> **This workflow is also available in CoreWeave Agent Lens.** Agent Lens is the Forge experience built for tracing, monitoring, and analyzing AI agents, with automated insights into agent failures and user intents. It uses the same trace data as Weights & Biases Weave, so the traces you already send appear there with nothing to migrate.
> [See how to do this in Agent Lens](/products/agent-lens/get-started/custom-agents).
[Try in Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/custom-agents-quickstart.ipynb) · [GitHub source](https://github.com/wandb/examples/blob/main/colabs/weave/custom-agents-quickstart.ipynb)
The Weave SDK lets you trace agents built with popular SDKs or custom harnesses. This quickstart shows you how to manually integrate Weave into a custom-built multi-turn agent to emit and capture OpenTelemetry spans. For conceptual understanding about Weave for agents, see [Trace your agents](/products/wandb/weave/guides/tracking/trace-agents).
If you're looking to integrate Weave with SDKs or harnesses such as the Claude Agent SDK or Codex, see [Choose an agent integration](/products/wandb/weave/agent-integration-quickstart). Weave autopatches into several agent-building SDKs and agent harnesses for quick integration.
## What you'll learn
By the end of this quickstart, you'll have a working multi-turn agent that emits Weave-compatible OTel spans. You'll also understand how Weave maps conversations, turns, LLM calls, and tool calls onto your agent code so you can apply the same pattern to your own custom agents.
The code in this guide sets up a small research agent that can look things up on Wikipedia. It asks three questions (three turns) and uses the LLM to choose when to search Wikipedia for an answer. Weave records every step (the conversation, each question, each AI response, and each Wikipedia lookup) so you can see what happened in the Weave Agents view.
This guide shows you how to:
* Initialize Weave for agent tracing with `weave.init()`.
* Open a conversation and a turn with `start_conversation` / `startConversation` and `start_turn` / `startTurn`.
* Wrap LLM calls with `start_llm` / `startLLM` and record usage.
* Wrap tool executions with `start_tool` / `startTool` and record results.
* Record complete token usage and a priceable model so token counts and cost render.
* View the resulting conversation, turns, and tool calls in the Agents view.
## How the Weave SDK works with agents
The Weave SDK includes a generic OTel ingest system for agents, meaning that Weave can capture information from any OTel span in your agent's code. However, Weave requires special handling of the following spans to render your agent's traces in the Agents view of the Weights & Biases UI.
| Concept | Python | TypeScript | OTel span |
| - | - | - | - |
| A conversation | `weave.start_conversation(...)` | `weave.startConversation(...)` | (no span, groups turns) |
| One user or agent exchange | `weave.start_turn(...)` | `weave.startTurn(...)` | `invoke_agent` |
| One LLM API call | `weave.start_llm(...)` | `weave.startLLM(...)` | `chat` |
| One tool execution | `weave.start_tool(...)` | `weave.startTool(...)` | `execute_tool` |
In Python, all four functions work as context managers (`with weave.start_*(...) as obj:`). On exit, they end the span and flush attributes, including on exceptions. In TypeScript, call `.end()` on each returned object. Use `try { ... } finally { obj.end(); }` to guarantee cleanup on exceptions.
Other [GenAI semantic-convention attributes](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/), such as `gen_ai.usage.*` and `gen_ai.agent.name`, enable additional rendering, but they're optional.
## Prerequisites
* A CoreWeave Forge account and [API key](https://forge.coreweave.com/settings#apikeys).
* An OpenAI API key.
* Python 3.10+ (for the Python examples).
* Node.js 18+ (the TypeScript examples require built-in `fetch`).
## Install packages
Install the following packages into your developer environment:
```bash Python theme={"system"}
pip install weave openai requests
```
```bash TypeScript theme={"system"}
npm install weave openai
```
## Initialize Weave
`weave.init()` authenticates with Weights & Biases and configures the OTel exporter that sends agent spans to the **Agents** view. If the project doesn't exist on your team, Weave creates it the first time you write to it.
```python lines Python theme={"system"}
import getpass
import os
os.environ["WANDB_API_KEY"] = getpass.getpass("Enter your CoreWeave Forge API key: ")
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
TEAM = input("Enter your CoreWeave Forge team name: ")
PROJECT = input("Enter your Weights & Biases project name: ")
import weave
weave.init(f"{TEAM}/{PROJECT}")
```
```typescript lines highlight="4" TypeScript twoslash theme={"system"}
// @noErrors
// Set WANDB_API_KEY, OPENAI_API_KEY in your environment before running this project
import * as weave from 'weave';
await weave.init(`[YOUR-TEAM]/[YOUR-PROJECT]`);
```
## Define a tool
The following code defines the agent's Wikipedia search tool along with an OpenAI tool schema that specifies when and how to use the tool.
```python lines Python theme={"system"}
import json
import requests
def wikipedia_search(query: str) -> str:
r = requests.get(
"https://en.wikipedia.org/w/api.php",
params={
"action": "query", "generator": "search", "gsrsearch": query, "gsrlimit": 1,
"prop": "extracts", "exintro": True, "explaintext": True, "format": "json",
},
headers={"User-Agent": "weave-demo"},
).json()
return next(iter(r["query"]["pages"].values()))["extract"]
wikipedia_tool_schema = {
"type": "function",
"function": {
"name": "wikipedia_search",
"description": "Search Wikipedia for a topic and return its intro paragraph.",
"parameters": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
},
}
```
```typescript lines TypeScript twoslash theme={"system"}
// @noErrors
async function wikipediaSearch(query: string): Promise {
const url = new URL('https://en.wikipedia.org/w/api.php');
url.search = new URLSearchParams({
action: 'query',
generator: 'search',
gsrsearch: query,
gsrlimit: '1',
prop: 'extracts',
exintro: 'true',
explaintext: 'true',
format: 'json',
}).toString();
const res = await fetch(url, { headers: { 'User-Agent': 'weave-demo' } });
const data = (await res.json()) as {
query: { pages: Record };
};
return Object.values(data.query.pages)[0].extract;
}
const wikipediaToolSchema = {
type: 'function' as const,
function: {
name: 'wikipedia_search',
description: 'Search Wikipedia for a topic and return its intro paragraph.',
parameters: {
type: 'object',
properties: { query: { type: 'string' } },
required: ['query'],
},
},
};
```
## Run a traced multi-turn agent
With the tool and Weave initialization in place, the next step combines them into a complete agent loop. This loop shows how conversations, turns, LLM calls, and tool calls nest together.
The following example runs three turns in a single conversation. Each turn:
1. Opens a `chat` span and lets the LLM choose whether to call the tool.
2. If the LLM requests a tool, opens an `execute_tool` span around the call and feeds the result back to the LLM.
3. Opens a second `chat` span to produce the final answer.
```python lines highlight="10,12,19,38,51,55,69" Python theme={"system"}
import weave
from openai import OpenAI
openai_client = OpenAI()
MODEL = "gpt-4o-mini"
def run_turn(history, user_message):
history.append({"role": "user", "content": user_message})
with weave.start_turn(user_message=user_message, model=MODEL):
# LLM call 1: the model might decide to use a tool.
with weave.start_llm(model=MODEL, provider_name="openai") as llm:
resp = openai_client.chat.completions.create(
model=MODEL, messages=history, tools=[wikipedia_tool_schema],
)
msg = resp.choices[0].message
llm.output(msg.content or "")
# record() sets usage, the priced model, and response id in one call.
llm.record(
usage=weave.Usage(
input_tokens=resp.usage.prompt_tokens,
output_tokens=resp.usage.completion_tokens,
cache_read_input_tokens=getattr(
resp.usage.prompt_tokens_details, "cached_tokens", 0
),
),
response_id=resp.id,
response_model=resp.model,
)
history.append(msg.model_dump(exclude_none=True))
# If no tool was requested, the first LLM response is the answer.
if not msg.tool_calls:
return msg.content
# Execute each requested tool call.
for tc in msg.tool_calls:
with weave.start_tool(
name=tc.function.name,
arguments=tc.function.arguments,
tool_call_id=tc.id,
) as tool:
tool.result = wikipedia_search(**json.loads(tc.function.arguments))
history.append({
"role": "tool",
"tool_call_id": tc.id,
"content": tool.result,
})
# LLM call 2: synthesize the final answer.
with weave.start_llm(model=MODEL, provider_name="openai") as llm:
resp = openai_client.chat.completions.create(model=MODEL, messages=history)
msg = resp.choices[0].message
llm.output(msg.content)
llm.record(
usage=weave.Usage(
input_tokens=resp.usage.prompt_tokens,
output_tokens=resp.usage.completion_tokens,
cache_read_input_tokens=getattr(
resp.usage.prompt_tokens_details, "cached_tokens", 0
),
),
response_id=resp.id,
response_model=resp.model,
)
history.append({"role": "assistant", "content": msg.content})
return msg.content
with weave.start_conversation(agent_name="research-bot") as conversation:
history = []
for question in [
"Who founded Anthropic?",
"What is Claude (the AI assistant)?",
"Summarize what we discussed in one sentence.",
]:
print(f"USER: {question}")
print(f"AGENT: {run_turn(history, question)}\n")
```
```typescript lines highlight="11,14,25,47,62,70,89" theme={"system"}
// @noErrors
import * as weave from 'weave';
import OpenAI from 'openai';
const openaiClient = new OpenAI();
const MODEL = 'gpt-4o-mini';
// history is a list of OpenAI chat messages; typed loosely for brevity.
async function runTurn(history: any[], userMessage: string): Promise {
history.push({ role: 'user', content: userMessage });
const turn = weave.startTurn({ model: MODEL });
try {
// LLM call 1: the model might decide to use a tool.
const llm1 = weave.startLLM({ model: MODEL, providerName: 'openai' });
let msg;
try {
const resp = await openaiClient.chat.completions.create({
model: MODEL,
messages: history,
tools: [wikipediaToolSchema],
});
msg = resp.choices[0].message;
llm1.output(msg.content ?? '');
// record() sets usage, the priced model, and response id in one call.
llm1.record({
usage: {
inputTokens: resp.usage?.prompt_tokens,
outputTokens: resp.usage?.completion_tokens,
cacheReadInputTokens: resp.usage?.prompt_tokens_details?.cached_tokens,
},
responseId: resp.id,
responseModel: resp.model,
});
history.push(msg);
} finally {
llm1.end();
}
// If no tool was requested, the first LLM response is the answer.
if (!msg.tool_calls?.length) {
return msg.content ?? null;
}
// Execute each requested tool call.
for (const tc of msg.tool_calls) {
if (tc.type !== 'function') continue;
const tool = weave.startTool({
name: tc.function.name,
args: tc.function.arguments,
toolCallId: tc.id,
});
try {
const { query } = JSON.parse(tc.function.arguments);
tool.result = await wikipediaSearch(query);
history.push({ role: 'tool', tool_call_id: tc.id, content: tool.result });
} finally {
tool.end();
}
}
// LLM call 2: synthesize the final answer.
const llm2 = weave.startLLM({ model: MODEL, providerName: 'openai' });
try {
const resp = await openaiClient.chat.completions.create({
model: MODEL,
messages: history,
});
const msg2 = resp.choices[0].message;
llm2.output(msg2.content ?? '');
llm2.record({
usage: {
inputTokens: resp.usage?.prompt_tokens,
outputTokens: resp.usage?.completion_tokens,
cacheReadInputTokens: resp.usage?.prompt_tokens_details?.cached_tokens,
},
responseId: resp.id,
responseModel: resp.model,
});
history.push({ role: 'assistant', content: msg2.content });
return msg2.content ?? null;
} finally {
llm2.end();
}
} finally {
turn.end();
}
}
const conversation = weave.startConversation({ agentName: 'research-bot' });
try {
const history: any[] = [];
for (const question of [
'Who founded Anthropic?',
'What is Claude (the AI assistant)?',
'Summarize what we discussed in one sentence.',
]) {
console.log(`USER: ${question}`);
console.log(`AGENT: ${await runTurn(history, question)}\n`);
}
} finally {
conversation.end();
}
```
## Record token usage and cost
Each `chat` span carries token usage and a model id. Weave renders token counts from usage and derives cost from usage plus the model id, so an incomplete or unpriceable value shows `0 in / 0 out` tokens or `Cost -` even when the rest of the trace looks correct. `record(...)` sets these fields (along with `output_messages`, `response_id`, `reasoning`, and more) in one call. Only the fields you pass are applied.
Two things must be right for cost to appear:
* **Complete usage.** `input_tokens` is the *total* input, including any cached tokens. Weave prices cache reads and cache writes at their own rates and subtracts them from the input total, so `cache_read_input_tokens` and `cache_creation_input_tokens` must be reported *in addition to* a total `input_tokens` that includes them. For providers with prompt caching (for example, Anthropic), cached tokens routinely dominate the input, so omitting them makes usage and cost render as roughly zero.
* **A priceable model id.** Cost is a lookup on the model. Weave prefers `response_model` (the exact model the provider served) and falls back to the `model` you passed to `start_llm`. An alias such as `opus` or `sonnet` is not priceable and renders `Cost -`, so pass the concrete id the response returns (`resp.model`) as `response_model`.
OpenAI counts cached tokens inside `prompt_tokens`, so the example above maps directly. Anthropic reports cached tokens *separately* from `input_tokens`, so add them back into the total Weave prices against:
```python lines Python theme={"system"}
with weave.start_llm(model=MODEL, provider_name="anthropic") as llm:
resp = anthropic_client.messages.create(
model=MODEL, max_tokens=1024, messages=history,
)
u = resp.usage
llm.output(resp.content[0].text)
llm.record(
usage=weave.Usage(
input_tokens=u.input_tokens
+ u.cache_read_input_tokens
+ u.cache_creation_input_tokens,
output_tokens=u.output_tokens,
cache_read_input_tokens=u.cache_read_input_tokens,
cache_creation_input_tokens=u.cache_creation_input_tokens,
),
response_id=resp.id,
response_model=resp.model,
)
```
```typescript lines TypeScript twoslash theme={"system"}
// @noErrors
const u = resp.usage;
llm.record({
usage: {
inputTokens:
u.input_tokens + u.cache_read_input_tokens + u.cache_creation_input_tokens,
outputTokens: u.output_tokens,
cacheReadInputTokens: u.cache_read_input_tokens,
cacheCreationInputTokens: u.cache_creation_input_tokens,
},
responseId: resp.id,
responseModel: resp.model,
});
```
## See your agent traces in the Agents view
When `weave.init()` runs, it prints a link to your project where you can see:
* A row in the **Agents** tab for `research-bot`.
* One conversation containing three turns.
* Each turn (`invoke_agent`) with two `chat` spans and an `execute_tool` span nested inside.
* Token counts, latency, model, and the full message exchange on each `chat`.
Click any turn to inspect the inputs, outputs, tool arguments, and tool results.
## Link to a conversation from your app
To deep-link from your own UI to a conversation in the Weave Agents view, build the URL from your entity, project, and the conversation id. `weave.init()` returns a client that carries `entity` and `project`, and `start_conversation` exposes `conversation_id`.
```python lines Python theme={"system"}
from weave.trace.urls import agent_conversation_path
client = weave.init(f"{TEAM}/{PROJECT}")
with weave.start_conversation(agent_name="research-bot") as conversation:
# ... run turns ...
url = agent_conversation_path(
client.entity, client.project, conversation.conversation_id
)
print(f"View this conversation at {url}")
```
```plaintext TypeScript lines theme={"system"}
This feature is not available in TypeScript.
```
## Next steps
* Learn how to [trace agents with Weave](/products/wandb/weave/guides/tracking/trace-agents) and what features and options are available in the Weave SDK.
* See [Choose an agent integration](/products/wandb/weave/agent-integration-quickstart) for more options about how to integrate Weave with your agents.
# Send OpenTelemetry spans to the Agents view
Source: https://docs.coreweave.com/products/wandb/weave/guides/tracking/trace-agents-otel
Send OTLP trace data from any OpenTelemetry pipeline to the Weave agents endpoint, with no Weave SDK required.
> [!TIP]
>
> **This workflow is also available in CoreWeave Agent Lens.** Agent Lens is the Forge experience built for tracing, monitoring, and analyzing AI agents, with automated insights into agent failures and user intents. It uses the same trace data as Weights & Biases Weave, so the traces you already send appear there with nothing to migrate.
> [See how to do this in Agent Lens](/products/agent-lens/get-started/opentelemetry).
Weave agent observability is built on [OpenTelemetry (OTel)](https://opentelemetry.io/docs/concepts/) and provides a dedicated OTLP endpoint that ingests spans into the **Agents** view. This means that you can quickly redirect your OTel-instrumented application's spans to Weave by swapping your OTel exporter's current endpoint configuration with Weave's endpoint configuration. You do not need to install the Weave SDK to do this.
This page covers how to authenticate with and send spans to the endpoint.
## Authentication
You can send spans to the following endpoint:
* **Path**: `/agents/otel/v1/traces`
* **Method**: `POST`
* **Content-Type**: `application/x-protobuf`
* **Base URL**: `https://trace.wandb.ai` for Multi-tenant Cloud.
The endpoint accepts OTLP protobuf payloads only. Use a protobuf HTTP exporter such as `opentelemetry-exporter-otlp-proto-http` (Python) or `@opentelemetry/exporter-trace-otlp-proto` (TypeScript). The endpoint also accepts `gzip` and `deflate` content encoding.
To authenticate with the endpoint, set your [Forge API Key](https://forge.coreweave.com/settings) in your `OTLPSpanExporter` configuration using one of the following values:
* `wandb-api-key`: Your API Key as the value.
* `Authorization`: HTTP Basic authentication with `api` as the user and your API Key as the password. This form is useful when your exporter or collector only supports standard authorization headers.
## Project routing
Weave routes spans to a project using either OTel resource attributes or a request header:
* **Resource attributes** (recommended): Set `wandb.entity` to your CoreWeave Forge team or user name and `wandb.project` to the project name on your `TracerProvider` resource.
* **`project_id` header**: Set the value to `[YOUR-TEAM]/[YOUR-PROJECT]`.
If both are present, the resource attributes take precedence. Spans that arrive with no entity and project are dropped.
## Shape spans for the Agents view
Weave accepts any OTel span and stores all of its attributes, but the Agents view renders spans that follow the [OTel GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md). Set `gen_ai.operation.name` on each span to tell Weave what the span represents:
| `gen_ai.operation.name` | Represents | Rendered as |
| - | - | - |
| `invoke_agent` | One complete user-agent exchange. | A turn. |
| `chat` | One call to a language model API. | An LLM call within a turn. |
| `execute_tool` | One tool execution. | A tool call. |
Two more attributes control grouping:
* `gen_ai.conversation.id`: Groups turns into a conversation. Use a stable ID for the lifetime of the conversation.
* `gen_ai.agent.name`: Groups conversations under a named agent in the Agents tab.
Other GenAI semantic-convention attributes, such as `gen_ai.request.model`, `gen_ai.usage.input_tokens`, and `gen_ai.usage.output_tokens`, enrich the rendering with model names and token counts but are optional. For the agent data model that these spans map onto, see [Trace your agents](/products/wandb/weave/guides/tracking/trace-agents#the-agent-data-model).
## Configure with environment variables only
If your application or collector reads the standard OTel exporter environment variables, you can set the following variables and route spans to Weave without changing any code:
```bash lines theme={"system"}
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://trace.wandb.ai/agents/otel/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="wandb-api-key=$WANDB_API_KEY"
export OTEL_RESOURCE_ATTRIBUTES="wandb.entity=[YOUR-TEAM],wandb.project=[YOUR-PROJECT]"
```
## Example: emit agent spans without the Weave SDK
The following example instruments a minimal agent turn using only OTel packages. It emits an `invoke_agent` span for the turn, a `chat` span for the LLM call, and an `execute_tool` span for a tool call, then exports them to the Weave agents endpoint.
First, install the required dependencies:
```bash Python theme={"system"}
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```
```bash TypeScript theme={"system"}
npm install @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/resources @opentelemetry/exporter-trace-otlp-proto
```
Then set the `WANDB_API_KEY`, `ENTITY`, and `PROJECT` values in the following code, and then run it:
```python Python lines highlight="8,9,14" theme={"system"}
import os
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import BatchSpanProcessor
ENTITY = "[YOUR-TEAM]"
PROJECT = "[YOUR-PROJECT]"
WEAVE_AGENTS_OTLP_ENDPOINT = "https://trace.wandb.ai/agents/otel/v1/traces"
# Create an API key at https://forge.coreweave.com/settings
WANDB_API_KEY = [YOUR-WANDB-API-KEY]
exporter = OTLPSpanExporter(
endpoint=WEAVE_AGENTS_OTLP_ENDPOINT,
headers={"wandb-api-key": WANDB_API_KEY},
)
tracer_provider = trace_sdk.TracerProvider(resource=Resource({
"wandb.entity": ENTITY,
"wandb.project": PROJECT,
}))
tracer_provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(tracer_provider)
tracer = trace.get_tracer("my-agent")
CONVERSATION_ID = "conversation-001"
# One turn: a user asks a question, the agent calls a tool, then answers.
with tracer.start_as_current_span("invoke_agent my-agent") as turn:
turn.set_attribute("gen_ai.operation.name", "invoke_agent")
turn.set_attribute("gen_ai.agent.name", "my-agent")
turn.set_attribute("gen_ai.conversation.id", CONVERSATION_ID)
with tracer.start_as_current_span("chat gpt-4o") as llm:
llm.set_attribute("gen_ai.operation.name", "chat")
llm.set_attribute("gen_ai.conversation.id", CONVERSATION_ID)
llm.set_attribute("gen_ai.request.model", "gpt-4o")
# Replace with a real LLM call and record its token usage.
llm.set_attribute("gen_ai.usage.input_tokens", 100)
llm.set_attribute("gen_ai.usage.output_tokens", 20)
with tracer.start_as_current_span("execute_tool get_weather") as tool:
tool.set_attribute("gen_ai.operation.name", "execute_tool")
tool.set_attribute("gen_ai.conversation.id", CONVERSATION_ID)
tool.set_attribute("gen_ai.tool.name", "get_weather")
# Replace with a real tool execution.
tracer_provider.shutdown() # Flush all pending spans before exit.
```
```typescript TypeScript lines highlight="7,8,13" theme={"system"}
import { trace } from "@opentelemetry/api";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { Resource } from "@opentelemetry/resources";
const ENTITY = "[YOUR-TEAM]";
const PROJECT = "[YOUR-PROJECT]";
const WEAVE_AGENTS_OTLP_ENDPOINT = "https://trace.wandb.ai/agents/otel/v1/traces";
// Create an API key at https://forge.coreweave.com/settings
const WANDB_API_KEY = [YOUR-WANDB-API-KEY]!;
const exporter = new OTLPTraceExporter({
url: WEAVE_AGENTS_OTLP_ENDPOINT,
headers: { "wandb-api-key": WANDB_API_KEY },
});
const provider = new NodeTracerProvider({
resource: new Resource({
"wandb.entity": ENTITY,
"wandb.project": PROJECT,
}),
spanProcessors: [new BatchSpanProcessor(exporter)],
});
provider.register();
const tracer = trace.getTracer("my-agent");
const CONVERSATION_ID = "conversation-001";
// One turn: a user asks a question, the agent calls a tool, then answers.
tracer.startActiveSpan("invoke_agent my-agent", (turn) => {
turn.setAttribute("gen_ai.operation.name", "invoke_agent");
turn.setAttribute("gen_ai.agent.name", "my-agent");
turn.setAttribute("gen_ai.conversation.id", CONVERSATION_ID);
tracer.startActiveSpan("chat gpt-4o", (llm) => {
llm.setAttribute("gen_ai.operation.name", "chat");
llm.setAttribute("gen_ai.conversation.id", CONVERSATION_ID);
llm.setAttribute("gen_ai.request.model", "gpt-4o");
// Replace with a real LLM call and record its token usage.
llm.setAttribute("gen_ai.usage.input_tokens", 100);
llm.setAttribute("gen_ai.usage.output_tokens", 20);
tracer.startActiveSpan("execute_tool get_weather", (tool) => {
tool.setAttribute("gen_ai.operation.name", "execute_tool");
tool.setAttribute("gen_ai.conversation.id", CONVERSATION_ID);
tool.setAttribute("gen_ai.tool.name", "get_weather");
// Replace with a real tool execution.
tool.end();
});
llm.end();
});
turn.end();
});
await provider.shutdown(); // Flush all pending spans before exit.
```
To add more turns to the same conversation, emit more `invoke_agent` spans with the same `gen_ai.conversation.id`. Each turn is the root span of its own trace, so turns don't need to share a parent span.
After your application exports spans, open the **Agents** tab of your Weave project.
# Quickstart: Instrument and trace functions
Source: https://docs.coreweave.com/products/wandb/weave/quickstart
Get started with W&B Weave by adding tracing to your LLM application to debug and monitor model interactions.
[Try in Colab](https://colab.research.google.com/github/wandb/examples/blob/main/colabs/weave/quickstart.ipynb)
[GitHub source](https://github.com/wandb/examples/blob/main/colabs/weave/quickstart.ipynb)
Learn how to track LLM calls with Weave by adding tracing to your code. This quickstart walks you through tracing a request to OpenAI and viewing the results in the Weights & Biases UI. It's for developers who are new to Weave and want to instrument an LLM application for debugging and monitoring.
## What you'll learn
This guide shows you how to:
* Import and configure Weave in your code.
* Use the `weave.op` decorator to track your code.
* View traces in the Weights & Biases UI.
## Prerequisites
* A [CoreWeave Forge account](https://id.coreweave.com/signup)
* Python 3.10+ or Node.js 18+
* Required packages installed:
* **Python**: `pip install weave openai`
* **TypeScript**: `npm install weave openai`
* An [OpenAI API key](https://platform.openai.com/api-keys) set as an environment variable.
## Log a trace to a new project
In this section, you instrument a small application so that Weave can record its inputs, outputs, and code. To begin tracking your code and logging traces to Weave:
1. Import the `weave` library into your code.
2. Call `weave.init('your_wb_team/project_name')` in your code to send tracking information to your Forge [team](/products/wandb/platform/app/settings-page/teams) and [project](/products/wandb/platform/hosting/iam/org_team_struct#project). If you don't set a team, Weave sends the traces to your [default team](/products/wandb/platform/app/settings-page/user-settings#default-team). If the specified project doesn't exist in your team, Weave creates it.
3. Add the [`@weave.op()` decorator](/products/wandb/weave/guides/tracking/ops) to functions you want to track. Although Weave automatically tracks calls to supported LLMs, the decorator lets you track the inputs, outputs, and code of specific functions. The decorator uses the following syntax in TypeScript: `weave.op(your_function)`.
The following example code sends a request to OpenAI (requires an [OpenAI API key](https://platform.openai.com/docs/quickstart)), and Weave records the request's tracing information. The request asks the OpenAI model to extract dinosaur names from the input and identify each dinosaur's diet (herbivore or carnivore).
The following example code tracks your first project with Weave:
**Python**
```python lines {1-2,7-8,27-28} theme={"system"}
# Imports the Weave library
import weave
from openai import OpenAI
client = OpenAI()
# Weave automatically tracks the inputs, outputs and code of this function
@weave.op()
def extract_dinos(sentence: str) -> dict:
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """In JSON format extract a list of `dinosaurs`, with their `name`,
their `common_name`, and whether its `diet` is a herbivore or carnivore"""
},
{
"role": "user",
"content": sentence
}
],
response_format={ "type": "json_object" }
)
return response.choices[0].message.content
# Initializes Weave, and sets the team and project to log data to
weave.init('your-team/traces-quickstart')
sentence = """I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike), \
both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant \
Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below."""
result = extract_dinos(sentence)
print(result)
```
**TypeScript**
```typescript twoslash lines {1-2,7-8,20,24-25} theme={"system"}
// @noErrors
// Imports the Weave library
import * as weave from 'weave';
import OpenAI from 'openai';
const openai = new OpenAI();
// Weave automatically tracks the inputs, outputs and code of this function
async function extractDinos(input: string) {
const response = await openai.chat.completions.create({
model: 'gpt-4o',
messages: [
{
role: 'user',
content: `In JSON format extract a list of 'dinosaurs', with their 'name', their 'common_name', and whether its 'diet' is a herbivore or carnivore: ${input}`,
},
],
});
return response.choices[0].message.content;
}
const extractDinosOp = weave.op(extractDinos);
async function main() {
// Initializes Weave, and sets the team and project to log data to
await weave.init('your-team/traces-quickstart');
const result = await extractDinosOp(
'I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike), both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below.'
);
console.log(result);
}
main();
```
When you call the `extract_dinos` function, Weave outputs links in the terminal to view your traces. The output looks like this:
```text theme={"system"}
weave: $ pip install weave --upgrade
weave: Logged in as Weights & Biases user: example-username.
weave: View Weave data at https://wandb.ai/your-team/traces-quickstart/weave
weave: 🍩 https://wandb.ai/your-team/traces-quickstart/r/call/019ae171-7f32-7c96-8b42-931a32f900b7
{
"dinosaurs": [
{
"name": "Tyrannosaurus rex",
"common_name": "T. rex",
"diet": "carnivore"
},
{
"name": "Triceratops",
"common_name": "Trike",
"diet": "herbivore"
},
{
"name": "Brachiosaurus",
"common_name": "Brachi",
"diet": "herbivore"
}
]
}
```
## See traces of your application in your project
Now that your code has produced a trace, you can inspect it in the Weights & Biases UI. Click the link in your terminal or paste it into your browser to open the Weights & Biases UI. In the **Traces** panel of the Weights & Biases UI, click the trace to see its data, such as its input, output, latency, and token usage.
You have now logged and viewed your first Weave trace.
## Learn more about traces
* Learn how to [decorate your functions and retrieve call information](/products/wandb/weave/guides/tracking/tracing).
* Try the [Playground](/products/wandb/weave/guides/tools/playground) to test different models on logged traces.
* [Explore integrations](/products/wandb/weave/guides/integrations). Weave automatically tracks calls made to OpenAI, Anthropic, and many more LLM libraries. If your LLM library isn't one of the Weave integrations, you can track calls to other LLM libraries or frameworks by wrapping them with `@weave.op()`.
## Next steps
[Get started evaluating your app](/products/wandb/weave/tutorial-eval) and then see how to [evaluate a RAG application](/products/wandb/weave/tutorial-rag).
# Agent Lens
Source: https://docs.coreweave.com/redirects/forge/agent-lens
Overview of Agent Lens on CoreWeave
# ARIA
Source: https://docs.coreweave.com/redirects/forge/aria
AI research agent that analyzes experiments, explains results, and builds reports
# Infrastructure and Forge billing
Source: https://docs.coreweave.com/redirects/forge/billing
Choose between CoreWeave Infrastructure billing and Forge billing, and find where each invoice lives
# Serverless Inference
Source: https://docs.coreweave.com/redirects/forge/inference
Open-source foundation models through an OpenAI-compatible API
# Notebooks
Source: https://docs.coreweave.com/redirects/forge/notebooks
Overview of Notebooks on CoreWeave
# Platform
Source: https://docs.coreweave.com/redirects/forge/platform
Forge platform documentation: configuration, security and access, monitoring and usage, deployment options, and Launch
# Registry
Source: https://docs.coreweave.com/redirects/forge/registry
Centralized, versioned hub for models and datasets with lineage and deployment handoff
# Sandboxes
Source: https://docs.coreweave.com/redirects/forge/sandboxes
Isolated compute environments on serverless capacity or your own CKS cluster
# Security & Access
Source: https://docs.coreweave.com/redirects/forge/security
Choose between CoreWeave IAM and Weights & Biases access management, and find the platform security model
# Post-Training
Source: https://docs.coreweave.com/redirects/forge/training
Post-train and fine-tune LLMs with Serverless RL and SFT, and distill specialized models from real traffic
# Weights & Biases
Source: https://docs.coreweave.com/redirects/forge/wandb
W&B Models for experiment tracking, sweeps, artifacts, and reports; W&B Weave for tracing and evaluating LLM apps
# Introduction to Automated User Provisioning
Source: https://docs.coreweave.com/security/automated-user-provisioning
Overview of Automated User Provisioning (AUP) and the CoreWeave user provisioning system
This page introduces CoreWeave's user provisioning system: how user and group identities move from their source of truth into CoreWeave IAM (CW IAM) and, if you use SUNK, into your SUNK clusters. Users and groups enter CW IAM through two paths: automated federation from an identity provider (IdP) using Automated User Provisioning (AUP), or manual creation in the Cloud Console. Users can also add SSH public keys to their accounts, which SUNK User Provisioning (SUP) uses to enable SSH access to SUNK clusters. This page focuses on AUP and is intended for organization administrators who manage user access at scale.
AUP uses [SCIM (System for Cross-domain Identity Management)](https://scim.cloud/), an open standard that ensures changes to user permissions, additions, or removals made in the IdP appear immediately in CW IAM. This removes the need for manual updates or waiting for user-initiated events like SAML SSO logins, and it supports an efficient, accurate user management workflow.
Although SCIM can synchronize in two directions, CoreWeave AUP uses one-way synchronization: the IdP is the source of truth, and data flows only from the IdP to CW IAM.
The following diagram shows how all three elements (IdP federation via AUP, manual user and group management, and user SSH keys) connect through CW IAM to your SUNK clusters:
User provisioning flow from identity sources through CW IAM to SUNK clusters
* **Identity provider (IdP) via AUP**: Federate users and groups from an IdP such as Okta or Microsoft Entra using SCIM. Changes in the IdP propagate automatically to CW IAM. The rest of this page covers AUP in detail.
* **Manual user and group management**: Create and manage users and groups directly in the Cloud Console on the [Users](https://console.coreweave.com/organization/users) and [Groups](https://console.coreweave.com/organization/groups) pages.
* **User SSH keys**: Users add SSH public keys in their [Settings page](https://console.coreweave.com/account/settings) under Slurm Attributes. SUP uses these keys to enable SSH access to SUNK clusters.
## SUNK user provisioning
SUP automatically provisions CW IAM data into your SUNK clusters, whether that data came from an IdP through AUP or was created manually in the Cloud Console. The exception is users you [exempt from SCIM](#exempt-a-user-from-scim), which SUP doesn't provision. For setup procedures, see [Provision users in SUNK](/products/sunk/manage_sunk/manage_cluster_access/sunk_user_provisioning).
## How SCIM differs from SAML SSO
Because AUP builds on SAML SSO, it helps to understand how the two standards divide responsibilities.
SAML SSO lets users sign in securely and supports Just-In-Time (JIT) provisioning, which creates accounts on first sign-in. SCIM requires SAML SSO and goes further by syncing entire directories, including hundreds of users and their group memberships, in real time. Users and groups appear automatically in the Cloud Console without individual invitations or first-time SAML sign-ins. SCIM handles user provisioning and deprovisioning, while SAML handles authentication.
## Users and groups
AUP synchronizes two core resource types from your IdP to the Cloud Console:
* **Users**: Creates new users, updates profile fields (first name, last name, status), and deactivates users in the Cloud Console when they're deactivated in your IdP.
* **Groups**: Syncs group memberships and group definitions for better access control. AUP supports only flat groups. Nested groups (groups whose member lists include other groups) aren't supported and cause provisioning errors.
AUP also supports syncing custom attributes defined in the IdP.
### Key use cases
The following scenarios show how AUP keeps user state consistent between the IdP and the Cloud Console:
* **Provision users**: When you assign a user to the app in the IdP, AUP automatically creates the user in the Cloud Console.
* **Update profiles**: When you change attributes, first name, last name, or active status in the IdP, those changes overwrite the matching values in the Cloud Console. Email address changes aren't profile updates. For more information, see [Change a user's email address](#change-a-users-email-address).
* **Deactivate users**: When you remove a user from a group assigned to the application in the IdP, AUP deactivates the user in the Cloud Console, so only authorized users retain access.
* **Sync groups**: You can sync group memberships for application-specific group management beyond access control. To sync a group, you must explicitly add it to the push list in the IdP.
* **Force sync (Okta-specific)**: Okta's force sync feature lets admins manually push updates, which triggers a synchronization of user attributes between Okta and the Cloud Console. This updates user attributes but doesn't activate or deactivate accounts. For more information, see [Okta's guide](https://help.okta.com/oie/en-us/content/topics/apps/apps_app_integration_wizard_scim.htm).
### Change a user's email address
AUP updates first name, last name, and active status from the IdP. It doesn't rename an existing user when the email address changes.
Don't change a user's email address only in your IdP. SCIM treats an unrecognized email address as a new user and creates a second CoreWeave account. That account has no organization permissions, and the original account remains. If you use SUNK, the new account also receives a new POSIX UID and GID, which breaks SSH access that depends on the original IDs. SAML just-in-time (JIT) provisioning is disabled while AUP is enabled, so a new email address does not create a user at sign-in.
To change email addresses or migrate users to a new email domain, [contact CoreWeave Support](/support/contact). Support coordinates the change. For more information, see [Change a user's email address](/security/authn-authz/manage-users#change-a-users-email-address).
## SCIM-managed and exempt users
Users provisioned from your IdP through AUP are SCIM-managed: the IdP is the source of truth for their profile fields, group memberships, and active status. When AUP is enabled, the Cloud Console **Users** page shows a **SCIM Managed** column so you can see which users the IdP controls.
You can't deactivate a SCIM-managed user from the Cloud Console, even with the `IAM Admin` role. To remove a SCIM-managed user's access, deprovision them in your IdP and let AUP propagate the change, or mark them exempt from SCIM and then [deactivate them](/security/authn-authz/manage-users#deactivate-user-accounts) from the Console.
### Exempt a user from SCIM
Marking a user exempt from SCIM removes them from synchronization. The IdP no longer updates the user's attributes (name, group memberships, or status), and you must manage the account manually from the Cloud Console. If you later deactivate or unassign an exempt user in your IdP, their CoreWeave account stays active until you deactivate it in the Console.
An exemption doesn't block deletion requests. If your IdP sends a SCIM delete request for the user, as Microsoft Entra does when a user is permanently deleted from the tenant, AUP deletes the CoreWeave account as well.
Use an exemption when you need to control an account from the Cloud Console instead of the IdP: for example, to deactivate a SCIM-managed user or to manage a user's group memberships manually.
> [!WARNING]
>
> If your organization uses [SUNK User Provisioning](/products/sunk/manage_sunk/manage_cluster_access/sunk_user_provisioning), exempting a user also removes them from the user data that SUP syncs to your SUNK clusters, because SUP reads the SCIM user list and that list excludes exempt users. This can break their SSH access to those clusters. Don't exempt a user who still needs SUNK access.
To exempt a user, you need the `IAM Admin` role:
1. In the Cloud Console, go to the [**Users** page](https://console.coreweave.com/organization/users).
2. Click the vertical ellipsis (**More options**) next to the user, then click **Exempt from SCIM**.
3. Review the warning, then click **Yes, exempt user**.
The **SCIM Managed** column reflects the change, and you can now manage the user from the Cloud Console.
To return an exempt user to IdP control, in the same menu, click **Include in SCIM**, then confirm in the dialog that opens.
## Set up SCIM
After you configure SAML SSO, set up SCIM to synchronize users and groups from the IdP to the Cloud Console. The exact procedure depends on your IdP, but the overall flow is the same:
1. **Enable SCIM in the Cloud Console**: Because SCIM controls organization-wide user data, you must explicitly enable it.
2. **Enable SCIM in the IdP**: In your IdP, open the Provisioning section in the application configuration. This is usually next to the Single Sign-On (SSO) section.
3. **Configure the SCIM API base URL**: This URL is usually prefixed with the organization ID in the Cloud Console.
4. **Select synchronization options**: Choose to push new users, push profile updates, and push groups, depending on the level of synchronization you want.
5. **Set up authentication**: Configure a bearer token for secure communication between the IdP and the Cloud Console.
6. **Assign users and groups**: In the IdP, assign users and groups to the SCIM-enabled application to provision them in the Cloud Console.
After you complete these steps in your IdP, AUP syncs users and groups assigned to the application into the Cloud Console, and subsequent changes in the IdP propagate automatically.
## Next steps
To set up AUP, follow the procedure for your IdP:
* [Configure AUP with Okta](/security/automated-user-provisioning/configure-okta)
* [Configure AUP with Microsoft Entra](/security/automated-user-provisioning/configure-microsoft-entra)
# Configure Automated User Provisioning with Microsoft Entra
Source: https://docs.coreweave.com/security/automated-user-provisioning/configure-microsoft-entra
Configure CoreWeave Automated User Provisioning with Microsoft Entra
Automated User Provisioning (AUP) lets you instantly sync users and groups from your identity provider (IdP) to the Cloud Console, using the SCIM (System for Cross-domain Identity Management) standard. You no longer need to send invites or wait for users to sign in.
This guide shows how to set up AUP with Microsoft Entra as the IdP. When you finish, SAML SSO and one-way SCIM are configured, and you can assign Entra users and groups so they sync into CoreWeave IAM.
> [!WARNING]
>
> Don't change a user's email address only in Microsoft Entra. Entra matches users by email or UPN by default, and CoreWeave treats an unrecognized address as a new user instead of a rename. To change email addresses, [contact CoreWeave Support](/support/contact). For more information, see [Change a user's email address](/security/automated-user-provisioning#change-a-users-email-address).
## Prerequisites
You need the following access:
* Admin access to the CoreWeave Cloud Console.
* Admin access to the Microsoft Entra dashboard.
## Create SAML integration
Configure SAML SSO first, because AUP relies on it for authentication.
1. Open the Cloud Console and the Microsoft Entra dashboard in separate windows.
* In the Cloud Console: Go to the [SAML SSO](https://console.coreweave.com/organization/iam/sso) page and click **Configure SAML**.
* In Entra: Open the Microsoft Entra dashboard at [entra.microsoft.com](https://entra.microsoft.com).
* Click **Enterprise Apps**, select **+ New Application** in the top bar, and select **Create your own application**.
* Choose an appropriate name for your CoreWeave organization and select **Integrate any other application you don't find in the gallery (Non-gallery)**.
* When you return to the Enterprise Apps page, select your new application and click **Single sign on** under the **Manage** menu in the left bar.
For the next few steps, the Cloud Console displays details for your integration that you copy into Entra.
2. In Entra, select **SAML 2.0** under **Select a single sign-on method**.
3. Under field 1: **Basic SAML Configuration**, click **Edit**.
4. Copy the **ACS URL** from the Cloud Console and paste it into Entra's **Reply URL (Assertion Consumer Service URL)** field.
5. Copy the **Entity ID** from the Cloud Console and paste it into Entra's **Entity ID** field.
6. Under **field 2: Attributes and Claims**, click **Edit**.
7. For each claim under **Additional Claims**, map the following fields to each Azure value by clicking each claim and editing the top **Name** field:
| Name | Value |
| - | - |
| `first_name` | `user.givenname` |
| `last_name` | `user.surname` |
| `email` | `user.mail` |
8. Under section **3: SAML Certificates**, click **Edit**. On the following menu under **Signing Option**, set this to **Sign SAML Response and Assertion**. Click **Save** to save settings.
9. Return to section **3: SAML Certificates**. Copy the URL from the **App Federation Metadata URL** field into your clipboard.
10. Return to the CoreWeave Cloud Console. Click the **Metadata URL** field and paste the URL you copied from Entra.
11. In the Cloud Console, click **Next** and then click **Deploy SSO**.
12. Return to Entra, and scroll to the end of the SAML configuration page. Click **Test**, and then in the **Test Single Sign on with your \[Enterprise App Name]** dialog click **Test**. Entra prompts you to sign in through your Microsoft account into the CoreWeave Cloud Console. If the test succeeds, you complete sign-in and arrive at the Clusters page.
SAML SSO is now configured between Entra and the Cloud Console. Next, configure SCIM so Entra can push users and groups into CoreWeave IAM.
## Configure one-way SCIM
Set up one-way SCIM provisioning so the Cloud Console can receive user and group information from Entra:
1. In the Cloud Console, on the [SCIM Configuration](https://console.coreweave.com/organization/iam/scim) page, toggle **Enable SCIM API** and **Enable Automated User Provisioning**. Record the **SCIM Base URL**. You need it later.
2. In the Cloud Console, create a new SCIM API token with a name of up to 20 letters, numbers, and hyphens (for example, `entra-id`). Record the token value. You need it later.
3. In Entra:
* On your Enterprise App's left navigation bar, click **Provisioning** (located under **Single Sign On**).
* Click **Connect your application** under **Create configuration**.
* Copy the **SCIM Base URL** from the Cloud Console and paste this into the **Tenant URL** field in Entra.
* Copy the **SCIM Token** from the Cloud Console and paste it into the **Secret token** field in Entra.
* Click **Test connection**. If the test succeeds, a green alert appears in the top right corner of your browser window.
* Click **Create**.
## Assign users and groups
To complete the integration, assign users to a group in your IdP, and then assign the group to your application. Then test the integration by checking whether the users sync to the Cloud Console.
1. In the Cloud Console, go to the [Users](https://console.coreweave.com/organization/users) page to view a list of all users in your organization.
2. Assign users to a group in Entra:
* In Entra, click **Users and groups** on your Enterprise App's left menu under **Manage**.
* Click **Add user/group** to select the users and groups that should sync with CoreWeave Cloud IAM.
3. In Entra, toggle the **Provisioning Status** to **On** under the **Provisioning** section. (This step is [required when you enable provisioning for the first time](https://learn.microsoft.com/en-us/entra/identity/app-provisioning/configure-automatic-user-provisioning-portal#provisioning-status).)
4. In the Cloud Console, refresh the page showing your **Users** list. The users in the group you just assigned in Entra appear immediately in the Cloud Console.
> [!NOTE]
>
> Users provisioned through AUP are SCIM-managed: Entra controls their profile fields and active status. To deactivate a SCIM-managed user from the Cloud Console rather than from Entra, first mark them [exempt from SCIM](/security/automated-user-provisioning#exempt-a-user-from-scim).
## Map SSH keys for SUNK
If you use [SUNK User Provisioning (SUP)](/products/sunk/manage_sunk/manage_cluster_access/sunk_user_provisioning), you can sync SSH public keys from Entra to CoreWeave so that users can use SSH to connect to SUNK login nodes. To do this, create a custom extension attribute in Entra and map it to CoreWeave's SCIM endpoint so that keys flow from user profiles into IAM.
> [!NOTE]
>
> If you don't use SUNK, skip this section.
### Create the SSH key extension attribute
Entra doesn't include a built-in attribute for SSH keys. You must create a custom extension property on your Enterprise Application's backing application registration.
1. In PowerShell, install the Microsoft Graph module if you haven't already:
```powershell theme={"system"}
Install-Module Microsoft.Graph -Scope CurrentUser
```
2. Connect to Microsoft Graph. This operation requires the `Application.ReadWrite.All` permission:
```powershell theme={"system"}
Connect-MgGraph -Scopes "Application.ReadWrite.All"
```
3. Find the **Object ID** of the application registration that backs your CoreWeave Enterprise Application. In the Entra portal, go to **App registrations** > **All applications**, find your CoreWeave app, and copy the **Object ID** (not the Application ID).
4. Create the extension property. Replace `[APP-OBJECT-ID]` with that object ID:
```powershell theme={"system"}
New-MgApplicationExtensionProperty `
-ApplicationId [APP-OBJECT-ID] `
-Name "sunkSshKeys" `
-DataType "String" `
-TargetObjects @("User")
```
Entra generates a prefixed name like `extension_[APP-ID-NO-HYPHENS]_sunkSshKeys`. You reference this prefixed name within Entra (for example, in PowerShell commands and Terraform). CoreWeave's SCIM endpoint only receives the `sunkSshKeys` portion after the namespace mapping, so the prefix doesn't affect provisioning.
### Set the SSH key on a user profile
Writing custom extension attributes to user profiles requires the `User.ReadWrite.All` Microsoft Graph permission. The more limited `User.ReadWrite` scope isn't sufficient for custom extension attributes.
To set an SSH key on a user profile, replace `[USER-EMAIL]` with the user's email address, replace the extension attribute name with the prefixed name Entra generated, and replace `[SSH-PUBLIC-KEY]` with the full public key string:
```powershell theme={"system"}
Update-MgUser -UserId [USER-EMAIL] `
-AdditionalProperties @{
extension_[APP-ID-NO-HYPHENS]_sunkSshKeys = "[SSH-PUBLIC-KEY]"
}
```
For example, `[SSH-PUBLIC-KEY]` might look like `ssh-ed25519 AAAAC3Nza...`.
### Add the attribute mapping
With the extension property created and populated on user profiles, map it to the CoreWeave SCIM schema so the key syncs.
1. In Entra, go to your CoreWeave Enterprise Application.
2. Under **Manage**, select **Provisioning** > **Attribute Mapping (Preview)**.
3. Select **Provision Microsoft Entra ID Users**.
4. Click **Add New Mapping** and configure the following fields:
| Field | Value |
| - | - |
| Mapping type | Direct |
| Source attribute | `extension_[APP-ID-NO-HYPHENS]_sunkSshKeys` |
| Target attribute | `urn:coreweave:params:scim:schemas:extension:coreweave:2.0:CoreWeaveUser:sunkSshKeys` |
| Match objects using this attribute | No |
| Apply this mapping | Always |
5. Click **OK**, then click **Save** at the top of the attribute mappings page.
### Verify the sync
1. In Entra, trigger a provisioning cycle (or wait for the next automatic cycle).
2. In the Cloud Console, go to **Users**, click the three-dot menu next to the target user, and select **View Details**.
3. Under **Slurm Attributes**, confirm the **SSH Keys** field contains the public key you set in Entra.
If the key doesn't appear, check the Entra provisioning logs for errors. For more details about verifying SSH keys and configuring SUP, see [Provision users in SUNK](/products/sunk/manage_sunk/manage_cluster_access/sunk_user_provisioning).
## Adjust attribute mappings
### Remove the department attribute
The **department** attribute in Microsoft Entra can prevent groups from syncing properly with CoreWeave IAM. Before you sync groups, remove this attribute:
1. Click **Enterprise apps**, and select the app that represents your CoreWeave integration.
2. Under the **Manage** heading on the left menu, select **Provisioning** > **Attribute Mapping (Preview)**.
3. Select **Provision Microsoft Entra ID Users**.
4. Find the **department** attribute, and under **Remote** click **Delete**.
5. Go to the top of the page and click **Save** to save your attribute mapping.
This adjustment removes the **department** attribute from the attribute mapping used for syncing with CoreWeave, but doesn't change the attribute inside Entra itself.
## Sync groups
Use the following guidance to plan how Entra groups map to CoreWeave IAM groups before you assign them to your Enterprise App.
### Nested groups
CoreWeave SCIM doesn't support nested groups. If you assign a parent Entra group whose **Members** list includes references to other groups, provisioning fails for those nested group members. To avoid sync errors, use one of the following approaches:
* Assign only flat (leaf) groups to your Enterprise App. Don't assign parent groups that contain subgroups.
* Add a [scoping filter](https://learn.microsoft.com/en-us/entra/identity/app-provisioning/define-conditional-rules-for-provisioning-user-accounts) on the group provisioning mapping to exclude parent groups.
* Use an Entra [dynamic group](https://learn.microsoft.com/en-us/entra/identity/users/groups-dynamic-membership) to flatten nested memberships into a single flat group before syncing.
### Recommended group sync configuration
1. Configure a regular Entra group for all the users you want to push to CoreWeave.
2. Configure regular Entra groups for all the subgroups that you want to represent in the Cloud Console.
3. For legacy CoreWeave IAM deployments, ensure that your selected Entra groups and subgroups aren't named any of the default user groups, or for every push group with the same name as a default user group create a new user group with the appropriate default policies attached.
### Legacy default user groups
Legacy CoreWeave IAM deployments automatically provisioned a set of default user groups with specific policies attached. The policies attached to these groups were necessary for operating CoreWeave services. These legacy default user groups include:
* `admin`
* `metrics`
* `read`
* `write`
* `billing_viewer`
When you sync groups with legacy CoreWeave IAM deployments with SCIM, you must resolve the naming conflict by either avoiding syncing push groups with these names, or for each push group with the same name as a default user group:
* Create a new user group in CoreWeave IAM with a new preferred name.
* Assign the policies attached to a default user group. For example, for an administration group use the policies attached to the `admin` group.
* Delete the default user group before configuring a push group with the same name.
# Configure Automated User Provisioning with Okta
Source: https://docs.coreweave.com/security/automated-user-provisioning/configure-okta
Configure CoreWeave Automated User Provisioning with Okta
Automated User Provisioning (AUP) lets you sync users and groups from your identity provider (IdP) to the Cloud Console, using the SCIM (System for Cross-domain Identity Management) standard. You no longer need to send invites or wait for users to sign in.
This guide shows how to set up AUP with Okta as the IdP. When you finish, SAML SSO and one-way SCIM are configured, and you can assign Okta users and groups so they sync into CoreWeave IAM.
> [!WARNING]
>
> Don't change a user's email address only in Okta. That change creates a new CoreWeave user instead of renaming the existing one. To change email addresses, [contact CoreWeave Support](/support/contact). For more information, see [Change a user's email address](/security/automated-user-provisioning#change-a-users-email-address).
> [!WARNING]
>
> If you're migrating SUNK users from Authentik or another LDAP directory, follow [Preserve POSIX identities during migration](#preserve-posix-identities-during-migration). Export and populate the existing attributes before you assign users to the CoreWeave application.
## Prerequisites
You need the following access:
* Admin access to the CoreWeave Cloud Console.
* Admin access to the Okta dashboard.
For SUNK migrations from Authentik or another LDAP directory, you also need:
* A system with `ldapsearch` and network access to the LDAP endpoint.
* An LDAP bind account with permission to search the configured LDAP search base. The account must also be able to read the configured POSIX username attribute, `uidNumber`, `gidNumber`, `homeDirectory`, `loginShell`, and `sshPublicKey`.
## Create SAML SSO integration
Users created through AUP must use SAML SSO for authentication, so you must [configure SAML SSO first](/security/authn-authz/saml-sso/configure-saml-sso).
1. Open the Cloud Console and the Okta dashboard in separate windows.
* In the Cloud Console: Navigate to the [SAML SSO](https://console.coreweave.com/organization/iam/sso) page and click **Configure SAML**.
* In Okta: Go to **Applications**, click **Create App Integration**, and select **SAML 2.0**.
For the next few steps, the Cloud Console displays details for your integration that you need to copy into Okta.
2. Copy the **ACS URL** from the Cloud Console and paste it into Okta's **Single sign-on URL** field. Leave the checkbox checked (for **Use this for Recipient URL and Destination URL**).
3. Copy the **Entity ID** from the Cloud Console and paste it into Okta's **Audience URI (SP Entity ID)** field.
4. In Okta, within the **SAML Attributes > Sign-on** tab, add the following SAML SSO attributes in the **Attribute Statements** section:
| Name | Value |
| - | - |
| `first_name` | `user.firstName` |
| `last_name` | `user.lastName` |
| `email` | `user.email` |
Click **Next**.
5. For **App Type**, select **This is an internal app we have created**, and click **Finish**.
6. The Okta dashboard displays a page with details for your integration. This time, you copy values from the Okta dashboard and paste them into the **Identity Provider** section in the Cloud Console. In the Okta dashboard, ensure the **Sign-on** tab is selected. Under **Sign-on methods**, under **Metadata details**, expand the **More details** caret to reveal more sign-on details.
* Copy the **Sign-on URL**, and then paste the value into the **SSO URL** field in the Cloud Console.
* Copy the **Issuer**, and then paste the value into the **Entity ID** field in the Cloud Console.
* Copy the **Signing Certificate**, and then paste the value into the **Signing certificate** field in the Cloud Console.
7. In the Cloud Console, click **Next** and then click **Deploy SSO**.
## Configure one-way SCIM
Set up one-way SCIM provisioning so the Cloud Console can receive user and group information from Okta.
1. In the Cloud Console, on the [SCIM Configuration](https://console.coreweave.com/organization/iam/scim) page, toggle **Enable SCIM API** and **Enable Automated User Provisioning**.
2. In the Cloud Console, create a new SCIM API token with a name of up to 20 letters, numbers, and hyphens (for example, `okta-id`).
3. In Okta:
* On your integration's **General** tab, click **Edit** in the **App Settings** card.
* Set **Provisioning** to **SCIM** radio button, and click **Save**.
4. Go to the **Provisioning** tab in Okta:
* Click **Edit** in the **SCIM Connection** section.
* Set **SCIM Connector base URL** to `https://api.coreweave.com/scim/[ORG-USERID]`. Find your org ID in the ACS URL (from the Cloud Console) `https://console.coreweave.com/accounts/saml/[ORG-USERID]/acs`.
* Set **Unique identifier field for users** to `userName`.
* For **Supported provisioning actions**: select **Push New Users**, **Push Profile Updates**, and **Push Groups**. Do not enable any import options.
* Set **Authentication mode** to **HTTP Header**.
* For **Authorization**, paste the bearer token from the Cloud Console.
After setup, the **Settings** sidebar populates two new tabs: **To App** and **To Okta**. Because this is a one-way sync, **To Okta** shows **Import Not Available**.
5. In the **To App** tab:
* Click **Edit** under **Provisioning to App**.
* Enable: **Create Users**, **Update User Attributes**, and **Deactivate Users**.
* Do not enable **Sync Password** because SAML SSO handles authentication.
* Click **Save**.
> [!NOTE]
>
> The **Deactivate Users** option controls whether deactivation flows from Okta to CoreWeave: when you deprovision a user in Okta, AUP deactivates them in the Cloud Console. To deactivate a SCIM-managed user from the Cloud Console instead, first mark them [exempt from SCIM](/security/automated-user-provisioning#exempt-a-user-from-scim).
## Assign users and groups
To complete the integration, assign users to a group in your IdP, and then assign the group to your application. Then test the integration by checking whether the users sync to the Cloud Console.
1. In the Cloud Console, navigate to the [Users](https://console.coreweave.com/organization/users) page to view a list of all users in your organization.
2. Assign users to a group in Okta:
* Go to **Directory > Groups** and click the name of a group.
* With the **People** tab selected, click **Assign people**.
* Click the **+** icon to add individuals to the group, and ensure you include the **Org Admin**.
3. Assign the group to your application in Okta:
* Go to **Applications > Applications** and click the name of your application.
* On your application's **Assignments** tab, click the **Assign** dropdown, then click **Assign to Groups**.
* Find the name of the group, then click **Assign**.
* Click **Save and go back**, then click **Done**.
* The name of that group appears in the assignments list for your application.
4. In the Cloud Console, refresh the page showing your **Users** list. The users in the group you assigned in Okta appear in the Cloud Console.
## Sync groups
The following sections describe how Okta syncs groups to CoreWeave and the options available for managing nested group structures.
### Nested groups
CoreWeave SCIM does not support nested groups. If you push a parent Okta group whose member list includes references to other groups, provisioning fails for those nested group members. To avoid sync errors, use one of the following approaches:
* Push only flat (leaf) groups. Do not push parent groups that contain sub-groups.
* Add a [group membership filter](https://help.okta.com/en-us/content/topics/users-groups-profiles/usgp-about-group-push.htm) to exclude parent groups from being pushed.
* Use an Okta [group rule](https://help.okta.com/en-us/content/topics/users-groups-profiles/usgp-create-group-rules.htm) to flatten nested memberships into a single flat group before pushing.
Okta uses *push groups* that are configured to push group memberships to applications like the Cloud Console, through a feature called [Group Push](https://help.okta.com/en-us/content/topics/users-groups-profiles/usgp-about-group-push.htm).
You can sync groups from Okta to the Cloud Console in two ways:
* **Recommended path**: Configure a regular Okta group for all the users you want to push to CoreWeave, along with all regular groups that you want to represent in the Cloud Console. Then configure push groups for any subgroups you want to manage within CoreWeave. Only those subgroups sync with CoreWeave automatically. Removing a user from a subgroup does not remove the user from the "all CoreWeave users" group.
* **Manual path**: Do not create an "all CoreWeave users" group. Configure push groups in a one-to-one mapping with every regular group. When you make changes, manually run **force sync** in Okta for each update.
## Map SUNK attributes
If you use [SUNK User Provisioning (SUP)](/products/sunk/manage_sunk/manage_cluster_access/sunk_user_provisioning), you can sync SSH public keys and POSIX user information from Okta to CoreWeave. This requires creating custom attributes in your Okta user profile and mapping each one to CoreWeave's SCIM endpoint.
> [!NOTE]
>
> If you are not using SUNK, skip this section.
### SUNK attribute reference
Create the following custom attributes in your Okta user profile and map each one to the corresponding CoreWeave SCIM attribute. Only `sunkSshKeys` is required for SSH access. The other attributes are optional: map them only if you want to set those values from Okta instead of letting CoreWeave IAM generate them. For how SUNK uses each attribute, see [SUNK SCIM attribute reference](/products/sunk/manage_sunk/manage_cluster_access/sunk_user_provisioning#sunk-scim-attribute-reference).
The steps in this section use the SSH keys attribute as a worked example. Repeat the same process for each attribute you map.
| Display name | Variable name | Data type | CoreWeave SCIM attribute |
| - | - | - | - |
| SUNK SSH Keys | `sunk_ssh_keys` | Array | `sunkSshKeys` |
| SUNK POSIX Username | `sunk_posix_username` | String | `sunkPosixUsername` |
| SUNK Preferred Home Directory | `sunk_preferred_home_directory` | String | `sunkPreferredHomeDirectory` |
| SUNK POSIX Group ID | `sunk_posix_group_id` | Number | `sunkPosixGroupId` |
| SUNK POSIX User ID | `sunk_posix_user_id` | Number | `sunkPosixUserId` |
| SUNK Login Shell | `sunk_login_shell` | String | `sunkLoginShell` |
### Add the SUNK attributes to the Okta user profile
1. In the Okta Admin Console, navigate to **Directory** > **Profile Editor**.
2. Click **Okta** in the Filters list, then click **Profile** for **Okta User (default)**.
3. Click **Add Attribute** and configure the fields for the SSH keys attribute:
| Field | Value |
| - | - |
| Data type | Array |
| Display name | SUNK SSH Keys |
| Variable name | `sunk_ssh_keys` |
| Description | SSH public keys for SUNK login |
4. Click **Save**.
5. Repeat the previous two steps for each other attribute you map from the [SUNK attribute reference table](#sunk-attribute-reference), using the **Display name**, **Variable name**, and **Data type** values from that table.
### Set the SSH key on a user profile
1. In the Okta Admin Console, navigate to **Directory** > **People**.
2. Select the target user and click the **Profile** tab.
3. Click **Edit** and enter the user's SSH public key in the **SUNK SSH Keys** field (for example, `ssh-ed25519 AAAAC3Nza...`).
4. Click **Save**.
### Add the attribute mapping
1. In the Okta Admin Console, navigate to **Applications** > **Applications** and select your CoreWeave application.
2. Click the **Provisioning** tab, then select **To App** in the settings list.
3. Scroll to the **Attribute Mappings** section and click **Edit**.
4. Find `sunk_ssh_keys` in the list of Okta user profile attributes. In the target attribute dropdown, select the CoreWeave SCIM attribute `sunkSshKeys`. Okta auto-discovers this attribute from CoreWeave's SCIM schema. The attribute appears under the `urn:coreweave:params:scim:schemas:extension:coreweave:2.0:CoreWeaveUser` namespace.
5. Repeat the previous step for each other attribute you map from the [SUNK attribute reference table](#sunk-attribute-reference), matching the Okta source attribute (**Variable name**) to the corresponding **CoreWeave SCIM attribute**.
6. Click **Save**.
### Verify the sync
1. In Okta, trigger a manual push for the target user (or wait for the next provisioning cycle).
2. In the Cloud Console, navigate to **Users**, click the three-dot menu next to the target user, and select **View Details**.
3. Under **Slurm Attributes**, confirm the **SSH Keys** field contains the public key you set in Okta.
If the key does not appear, check the Okta provisioning logs under **Reports** > **System Log** for errors. For more details about verifying SSH keys and configuring SUP, see [Provision users in SUNK](/products/sunk/manage_sunk/manage_cluster_access/sunk_user_provisioning).
## Preserve POSIX identities during migration
For a SUNK migration, use this procedure before the standard provisioning sequence.
When you migrate an existing SUNK environment from Authentik or another LDAP directory, preserve each user's numeric UID and GID. File ownership and permissions use these numeric IDs, so allowing CoreWeave IAM to assign new values can prevent users from accessing their existing files.
Complete the migration in this order:
1. From a system that can reach your LDAP endpoint, export the existing LDAP attributes as close to the cutover as possible. Set `[LDAP-URI]` to the LDAP endpoint URI and `[LDAP-BIND-DN]` to the bind distinguished name. Set `[LDAP-SEARCH-BASE]` to the search base and `[LDAP-USER-FILTER]` to the user search filter. Set `[LDAP-USERNAME-ATTRIBUTE]` to the attribute configured as the POSIX username in SUNK:
```bash theme={"system"}
export LDAP_URI="[LDAP-URI]"
export LDAP_BIND_DN="[LDAP-BIND-DN]"
export LDAP_SEARCH_BASE="[LDAP-SEARCH-BASE]"
export LDAP_USER_FILTER="[LDAP-USER-FILTER]"
export LDAP_USERNAME_ATTRIBUTE="[LDAP-USERNAME-ATTRIBUTE]"
ldapsearch -x \
-H "${LDAP_URI}" \
-D "${LDAP_BIND_DN}" \
-W \
-b "${LDAP_SEARCH_BASE}" \
"${LDAP_USER_FILTER}" \
"${LDAP_USERNAME_ATTRIBUTE}" \
uidNumber gidNumber homeDirectory loginShell sshPublicKey \
> authentik-sunk-users.ldif
```
`ldapsearch` prompts for the bind password instead of storing it in your shell history. The exported file contains user identity data and SSH public keys, so protect it according to your organization's policies.
To find the username attribute, check `userNameAttr` in the existing System Security Services Daemon (SSSD) configuration or `ldap_uidattr` in the existing `nsscache` configuration. Authentik deployments can use `cn`, but the configured attribute can differ.
2. Complete the [SAML SSO integration](#create-saml-sso-integration) and [one-way SCIM configuration](#configure-one-way-scim), but don't assign users or groups to the CoreWeave application yet.
3. In Okta, [create the SUNK attributes](#add-the-sunk-attributes-to-the-okta-user-profile) and [map them to the CoreWeave SCIM attributes](#add-the-attribute-mapping).
4. Before you assign users to the CoreWeave application, populate each Okta user profile with the corresponding exported values:
| LDAP attribute | Okta variable |
| - | - |
| Configured POSIX username attribute | `sunk_posix_username` |
| `uidNumber` | `sunk_posix_user_id` |
| `gidNumber` | `sunk_posix_group_id` |
| `homeDirectory` | `sunk_preferred_home_directory` |
| `loginShell` | `sunk_login_shell` |
| `sshPublicKey` | `sunk_ssh_keys` |
If a user has multiple `sshPublicKey` values, add each key to the `sunk_ssh_keys` array.
5. [Assign the users and groups](#assign-users-and-groups).
6. On the **SCIM Configuration** page in the Cloud Console, confirm that you've enabled **Enable SCIM API**, **Enable automated user provisioning**, and **Enable SUNK user provisioning**.
7. Coordinate with your CoreWeave account team to test the SCIM-backed [`nsscache` configuration](/products/sunk/manage_sunk/manage_cluster_access/nsscache#provision-cluster-access-to-groups-of-users) against the LDAP export. After the comparison succeeds, your account team can update SUNK to use SCIM instead of LDAP.
8. Compare each user's POSIX username, UID, GID, home directory, login shell, and SSH keys with the exported LDAP values. Use the `nsscache` output or SCIM response for the complete comparison. You can verify the POSIX username, UID, GID, and SSH keys on the Cloud Console **Users** page. Also confirm that assigned users and groups appear in the Cloud Console.
> [!WARNING]
>
> Until the exported POSIX attributes and SSH keys match and you've verified user and group synchronization, don't disable or remove Authentik.
## Adjust attribute mappings
The following sections describe attribute mapping adjustments you must make so that groups sync correctly, along with the recommended group sync configuration.
### Remove the Department attribute
The **Department** attribute in Okta can prevent groups from syncing properly with CoreWeave IAM. Before syncing groups, remove this attribute from the attribute mappings:
1. Navigate to your CoreWeave application in the Okta dashboard.
2. Click the **Provisioning** tab.
3. Locate the **Attribute Mapping** section.
4. Remove the **Department** attribute, and save your changes.
This adjustment removes the **Department** attribute from the attribute mapping used for syncing with CoreWeave, but does not change the attribute inside Okta itself.
### Recommended group sync configuration
To sync groups correctly between Okta and CoreWeave IAM, follow these steps:
1. Configure a regular Okta group for all the users you want to push to CoreWeave.
2. Configure regular Okta groups for all the subgroups that you want to represent in the Cloud Console.
3. For legacy CoreWeave IAM deployments, ensure that your selected Okta groups and subgroups are not named any of the [default user groups](#legacy-default-user-groups). For every push group with the same name as a default user group, create a new user group with the appropriate default policies attached.
4. Use Group Push to [sync your Okta groups](https://help.okta.com/wf/en-us/content/topics/workflows/access-control/access-control-sync-group.htm) for all except the "all CoreWeave users" group.
### Legacy default user groups
Legacy CoreWeave IAM deployments automatically provisioned a set of default user groups with specific policies attached. The attached policies were necessary for operating CoreWeave services. These legacy default user groups included:
* `admin`
* `metrics`
* `read`
* `write`
* `billing_viewer`
When syncing groups with legacy CoreWeave IAM deployments with SCIM, you must resolve the naming conflict by either avoiding syncing push groups with these names, or for each push group with the same name as a default user group:
* Create a new user group in CoreWeave IAM with a new preferred name.
* Assign the policies attached to a default user group. For example, for an administration group use the policies attached to the `admin` group.
* Delete the default user group before configuring a push group with the same name.
# Support access management
Source: https://docs.coreweave.com/security/support-access-management
Control and audit CoreWeave employee access to your CKS environment
## Overview
Support Access Management gives your organization visibility and control over when and how CoreWeave employees access your CKS environment for support purposes. This page describes how the feature works, what systems it covers, how approvers in your organization review access requests, and what logs are available for auditing. With Support Access Management enabled, every CoreWeave-initiated access to your environment is:
* **Request-based**: CoreWeave employees must submit a Teleport access request before accessing any resource.
* **Customer-approved**: Access is granted only after a member of your organization with the `Access Request Approver` role approves the request.
* **Time-bound**: Approved access expires automatically after 8 hours.
* **Auditable**: CoreWeave logs all sessions and the actions taken within them.
## Included systems
CoreWeave employees may request access to different systems within a CKS environment. The following table describes the specific systems and their typical methods of access:
| System | Typical method of access |
| - | - |
| CoreWeave Kubernetes Service (CKS) | `kubectl` CLI |
| Physical servers underlying a CKS cluster | SSH |
CoreWeave never impersonates specific individuals within your organization.
### Access to CKS clusters
When CoreWeave requests access to your CKS cluster, the request specifies the level of access needed. When you review the request, you see the requested role and can deny the request if the permissions are inappropriate. CoreWeave employees may request the following roles:
| ClusterRole | Permissions |
| - | - |
| `teleport-cluster-viewers` | Read-only access to most resources, but no access to `Secret` resources |
| `teleport-cluster-super-admins` | Full access to all Kubernetes resources |
### Access to physical servers
In some situations, CoreWeave employees may need to access your physical servers through SSH. This lets CoreWeave run low-level networking diagnostics, troubleshoot issues with Kubelets, and more. CoreWeave typically requests login as the `acc` user, which has sudo access.
## Manage access requests
When a CoreWeave employee needs to access your environment, your organization receives a notification and reviews the request before CoreWeave grants any access. The following sections describe how CoreWeave delivers notifications and how approvers in your organization respond to each request.
### Receive access request notifications
When a CoreWeave employee requests access to a system within your CKS environment, you receive a notification. This notification includes a link to the [Cloud Console](https://console.coreweave.com/access-requests), where you can review the details of the request. These details include the resources and permissions that the CoreWeave employee is requesting access to, and the reason for the request.
CoreWeave delivers notifications by email to members of your organization with the [`Access Request Approver`](/security/iam/access-policies/roles) role. Only users with this role can approve or deny access requests.
If you enable Support Access Management, CoreWeave requires that you monitor your notification channel at all times to review and approve access requests promptly.
New access requests appear in the `Pending` state in the Cloud Console:
### Approve requests
After reviewing an access request, you can approve it in the Console to grant CoreWeave access to your environment. Approved requests display the date and time of approval and identify the person in your organization who approved them:
By default, approved access requests remain valid for 8 hours, after which CoreWeave revokes the employee's access and terminates the session.
### Deny requests
After reviewing an access request, you can deny it. Denied requests display the date and time of denial and identify the person in your organization who denied them:
When you deny a request, CoreWeave doesn't receive access to your environment until CoreWeave submits a subsequent request and your organization approves it.
### Expired requests
After receiving an access request, you have 2 hours to either approve or deny it. Requests that don't receive a response within 2 hours automatically expire and don't grant CoreWeave any access to your environment. If this occurs, CoreWeave informs you that it won't access the requested resource, and may send a new request or attempt to troubleshoot without access to the system.
> [!WARNING]
>
> Any time between CoreWeave delivering the notification that access has been requested and you approving the request doesn't count as downtime against CoreWeave SLAs.
>
> If any component involved in the access request and approval flow experiences downtime, and immediate access to your cluster or physical server is required, CoreWeave requests written approval through Slack or email.
## Logs and forwarding
CoreWeave can provide three types of logs related to support access activity:
| | Teleport audit logs | Kubernetes audit logs | Teleport session logs |
| - | - | - | - |
| **Contents** | Every access request, approval, and denial | User, resource, and authorization details of all Kubernetes API interactions | Commands executed during approved access periods |
| **Delivery** | Forwarded in real time through CoreWeave Telemetry Relay | Available in CoreWeave Grafana, forwarded in real time through Telemetry Relay | Provided on request with a 24-hour turnaround |
> [!WARNING]
>
> CoreWeave makes no commitment to retain Teleport audit logs or Kubernetes audit logs for the purpose of providing them to you at a later date.
### Teleport audit logs
Teleport audit logs contain every access request, approval, and denial related to the in-scope resources. CoreWeave can forward Teleport audit logs automatically through [CoreWeave Telemetry Relay](/observability/telemetry-forwarding/relay). CoreWeave labels these audit logs to distinguish them from Kubernetes logs.
After forwarding, you fully control audit log management. Because CoreWeave may not be able to resend previously forwarded logs, follow best practices for log retention within your organization.
### Kubernetes audit logs
Kubernetes audit logs contain user actions, resource changes, authentication details, and metadata associated with all CKS cluster API interactions. CoreWeave Grafana includes a Kubernetes audit log dashboard to view and query these logs in a tabular format. CoreWeave can also forward Kubernetes audit logs automatically through Telemetry Relay.
### Teleport session logs
Teleport session logs contain detailed records of the commands the CoreWeave employee executes during their approved period of access.
CoreWeave can provide session logs on request, with a turnaround time of 24 hours. Before providing the logs, CoreWeave verifies that a user with the `Access Request Approver` role initiated or approved any requests for session logs.
Session logs remain available for forwarding for 12 months after the start of the session.
## System architecture
CoreWeave employees can only access Teleport through the secure corporate access network, using a CoreWeave-monitored device such as a company-issued laptop. When you enable Support Access Management, only users with the `Access Request Approver` role in your organization can approve Teleport requests. The following diagram details the system architecture for Support Access Management:
## Get started
Support Access Management is an opt-in feature. To enable it for your organization, contact [Support](/support).
After CoreWeave enables Support Access Management, CoreWeave recommends assigning the [`Access Request Approver`](/security/iam/access-policies/roles) role to multiple members of your organization across different time zones to ensure requests can be approved at any time. Access requests expire after 2 hours without a response.
To enable log forwarding for support access activity, [configure Telemetry Relay](/observability/telemetry-forwarding/relay#configure-telemetry-relay).
# Compliance programs
Source: https://docs.coreweave.com/security/trust-compliance/compliance-programs
CoreWeave's alignment with SOC 2, ISO 27001, ISO 27017, and ISO 27018 compliance standards
CoreWeave aligns its security and compliance programs with recognized regulatory standards so you can run sensitive and regulated workloads on the CoreWeave Cloud. Our controls, processes, and independent assessments support enterprise due diligence, third-party risk reviews, and formal compliance programs across industries.
At our [Compliance Documentation Portal](https://trust.coreweave.com/), you can request and review detailed security and compliance evidence, including policies, control descriptions, and (where available) third-party assessments that map to major frameworks and regulations such as:
* SOC 2
* ISO 27001
* ISO 27017
* ISO 27018
# Trust Center
Source: https://docs.coreweave.com/security/trust-compliance/trust-center
Access the CoreWeave Trust Center for security architecture, privacy, resilience, and compliance information
CoreWeave prioritizes privacy, safety, and transparency for AI workloads. Customers retain full ownership of their data, which we process only to operate and secure the platform, with controls designed to support enterprise and regulated use cases.
For details, visit the [CoreWeave Trust Center](https://www.coreweave.com/trust), where you'll find:
* Security architecture and advisories: How security is built into every layer, plus platform security advisories and the shared responsibility model.
* Privacy and data handling: Privacy policies, terms, and how we protect data confidentiality, integrity, and regulatory alignment (including GDPR and CCPA).
* Resilience and status: Service status, regions and availability zones, and how we engineer for high availability and fault tolerance.
* Compliance and evidence: Overview of certifications, assessments, and how to request audit reports and compliance documentation under NDA.
# CKS Workload Federation for AI Object Storage
Source: https://docs.coreweave.com/security/tutorials/cks-object-storage-authentication
Authenticate CKS workloads to AI Object Storage using OIDC Workload Identity Federation
This guide describes how to request access tokens for CoreWeave AI Object Storage operations through federation between CKS clusters (operating as OIDC Providers) and an organization's Object Storage infrastructure.
## Prerequisites
* A [CKS cluster](/products/cks/clusters/create) with [OIDC Workload Identity](/products/cks/auth-access/workload-identity/introduction) enabled.
* An AI Object Storage bucket. For setup instructions, see [Get started with AI Object Storage](/products/storage/object-storage/get-started-caios).
* The roles or granular permissions required by the setup tasks. The automatic and manual guides list both options.
## Authentication methods
CKS Workload Federation supports two methods for authenticating to AI Object Storage:
* [Automatic](/security/tutorials/cks-object-storage-authentication/automatic) using the Pod Identity Webhook, which is recommended for most use cases.
* [Manual](/security/tutorials/cks-object-storage-authentication/manual) by configuring the OIDC Workload Federation manually, which is useful for debugging or when the Pod Identity Webhook isn't available.
## How the three authorization systems relate
CoreWeave uses separate authorization systems that apply at different stages of this setup:
| System | What it controls | When it applies in this flow |
| - | - | - |
| [CoreWeave IAM Access Policies](/security/iam/access-policies) | Who can manage the platform and Object Storage control plane (create clusters, edit org policies, create WIF config). The **Object Storage Admin** role doesn't grant S3 data access. | Roles cover broad administrative tasks such as creating the cluster, configuring OIDC Workload Federation, and initially managing organization access policies. |
| [Kubeconfig](/products/cks/auth-access/managed-auth/kubeconfig) and Kubernetes RBAC | A kubeconfig is a config file with the cluster server URL and an API key (the same type used for other CoreWeave APIs). No separate kubeconfig permissions are required. [Kubernetes RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) controls what you can do inside the cluster (for example, creating Pods and service accounts). | You use a kubeconfig to run `kubectl` and create the workload. The API key authenticates you to the cluster, and Kubernetes RBAC authorizes your requests. |
| [AI Object Storage org and bucket access policies](/products/storage/object-storage/auth-access/organization-policies/about) | Who can perform Object Storage control-plane and S3 operations. Principals include CoreWeave users and federated OIDC roles such as `role/[ISSUER]:system:serviceaccount:namespace:name`. | An existing organization policy can delegate granular `cwobject:` actions to users. When a Pod uses OIDC, grant its WIF principal `cwobject:CreateAccessKeyOIDC` plus only the S3 actions that the workload needs. |
In short: IAM governs your ability to manage the platform and create the cluster. The API key in your kubeconfig (the same type as for other CoreWeave APIs) authenticates you to the cluster, and Kubernetes RBAC governs what you can do inside it. Org and bucket policies govern what each identity (including the Pod's OIDC role) can do with AI Object Storage data.
# Automatic authentication with the Pod Identity Webhook
Source: https://docs.coreweave.com/security/tutorials/cks-object-storage-authentication/automatic
Configure automatic OIDC credential injection for CoreWeave AI Object Storage using the Pod Identity Webhook on CKS
This tutorial shows you how to automate CoreWeave Kubernetes Service (CKS) Workload Federation for CoreWeave AI Object Storage by configuring the Pod Identity Webhook. With the webhook in place, your Pods automatically receive the configuration needed to obtain temporary Object Storage credentials through their ServiceAccount. Your workloads can then access buckets without managing long-lived access keys. By the end, you'll have a CKS cluster federated with Object Storage and a sample workload that uses an annotated ServiceAccount to obtain credentials on demand.
## Prerequisites
* A [CKS cluster](/products/cks/clusters/create) with [OpenID Connect (OIDC) Workload Identity](/products/cks/auth-access/workload-identity/introduction) enabled.
* [`helm`](https://helm.sh/docs/intro/install/) version 3
* [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) installed and configured for your cluster.
The following table separates roles that cover the administrative tasks from the narrower permissions used by this tutorial:
| Task | Role that covers the task | Granular permissions |
| - | - | - |
| Create or manage the CKS cluster | **CKS Admin** | After the cluster exists, Kubernetes RBAC controls whether you can install the Helm chart and create the tutorial's Namespace, ServiceAccount, and Pods. |
| Create an OIDC Workload Federation configuration if the automatic configuration is missing | **IAM Admin** | CoreWeave IAM Access Policies assign roles rather than individual IAM actions. |
| Create and view the Object Storage organization access policy | **Object Storage Admin** | `cwobject:EnsureAccessPolicy` and `cwobject:ListAccessPolicy` granted to your user through an existing organization access policy. Add `cwobject:DeleteAccessPolicy` if you must delete the policy. |
| Exchange the ServiceAccount token and run the verification command | No CoreWeave IAM role applies to the federated ServiceAccount. | Grant `cwobject:CreateAccessKeyOIDC` and `s3:ListAllMyBuckets` to the WIF principal in an Object Storage organization access policy, with `resources: ["*"]`. |
An Object Storage Admin must create the first organization access policy that delegates granular `cwobject:` permissions. The **Object Storage Admin** role governs the Object Storage control plane; it doesn't grant S3 data access.
## Install the Pod Identity Webhook
Install the Pod Identity Webhook to use Object Storage. The following steps install the webhook from [CoreWeave Charts](/products/cks/clusters/coreweave-charts/introduction):
1. Set environment variables for your organization ID and Availability Zone:
* Replace `[ORG-ID]` with your organization's ID. You can find your organization ID on the [CoreWeave Console settings page](https://console.coreweave.com/account/settings).
* Replace `[AVAILABILITY-ZONE]` with an Availability Zone that supports Object Storage.
```bash theme={"system"}
export ORG_ID=[ORG-ID]
export REGION=[AVAILABILITY-ZONE]
```
> **Availability Zones that support AI Object Storage**
>
> **US-CENTRAL**
>
> * `US-CENTRAL-05A`
> * `US-CENTRAL-06A`
> * `US-CENTRAL-07A`
> * `US-CENTRAL-08A`
> * `US-CENTRAL-08B`
>
> **US-EAST**
>
> * `US-EAST-01A`
> * `US-EAST-02A`
> * `US-EAST-03A`
> * `US-EAST-04A`
> * `US-EAST-04B`
> * `US-EAST-06A`
> * `US-EAST-08A`
> * `US-EAST-13A`
> * `US-EAST-14A`
> * `US-EAST-15A`
> * `US-EAST-17A`
>
> **US-WEST**
>
> * `RNO2A`
> * `US-WEST-01A`
> * `US-WEST-04A`
> * `US-WEST-09B`
> * `US-WEST-10A`
>
> **CA-EAST**
>
> * `CA-EAST-01A`
>
> **EU-NORTH**
>
> * `EU-NORTH-05A`
>
> **EU-SOUTH**
>
> * `EU-SOUTH-03B`
> * `EU-SOUTH-04A`
>
> Learn more about [Regions and Availability Zones](/platform/regions/about-regions-and-azs).
2. Add the CoreWeave Charts repository and install the Pod Identity Webhook:
```bash theme={"system"}
helm repo add coreweave https://charts.core-services.ingress.coreweave.com
helm install pod-identity-webhook coreweave/pod-identity-webhook \
--namespace pod-identity-webhook \
--create-namespace \
--wait \
--set config.orgID=$ORG_ID \
--set config.region=$REGION
```
The output resembles the following:
```text theme={"system"}
"coreweave" has been added to your repositories
NAME: pod-identity-webhook
LAST DEPLOYED: Fri Mar 13 20:05:00 2026
NAMESPACE: pod-identity-webhook
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
```
With the `--wait` flag, Helm waits until the release resources are ready. The Pod Identity Webhook can then inject credential configuration into annotated Pods.
## Verify OIDC Workload Federation
Before the Pod Identity Webhook can obtain credentials, Object Storage must trust the OIDC tokens your CKS cluster issues. When you create a CKS cluster with OIDC Workload Identity enabled, CoreWeave registers this trust automatically. CoreWeave creates an OIDC configuration for the cluster with the audience `https://coreweave.com/iam`. This audience matches the default audience that the Pod Identity Webhook injects, so you don't need to create another OIDC configuration.
1. In the Cloud Console, retrieve the OIDC Issuer URL for your CKS cluster:
* Go to the [Clusters](https://console.coreweave.com/clusters) page.
* On the **Clusters** page, click the name of your cluster. A cluster details panel opens on the right.
* In the **Cluster access** section, copy the **OIDC Issuer URL**. You need this URL again when you configure access policies.
2. On the [Workload Federation](https://console.coreweave.com/organization/iam/workload-federation/oidc) page, verify that the automatically created OIDC configuration for your cluster exists:
* On the **Workload Federation** page, find the row that corresponds to your cluster. The automatically created configuration's description begins with "Auto-created OIDC workload federation config" and contains the cluster name. In the search field, enter the cluster name to filter for matching OIDC configurations.
* Confirm that its **Issuer** matches your cluster's OIDC Issuer URL.
* Confirm that its **Client ID (Audience)** is `https://coreweave.com/iam`.
### Configure OIDC workload identity federation manually
If the automatically created configuration is missing, create an OIDC workload identity federation (WIF) configuration manually:
1. In the Cloud Console, go to the [Create OIDC Configuration](https://console.coreweave.com/organization/iam/workload-federation/oidc/create) page.
2. Enter the following values:
* **Configuration name**: A unique name for this configuration
* **Issuer URL**: Your cluster's OIDC Issuer URL
* **Client ID (Audience)**: `https://coreweave.com/iam`
3. On the **Create OIDC Configuration** page, click **Create**.
### Configure OIDC workload identity federation with a custom audience
If a workload needs a custom audience, create an additional OIDC configuration. Set **Issuer URL** to your cluster's OIDC Issuer URL. Set **Client ID (Audience)** to the custom audience. Then annotate the workload's ServiceAccount with `caios.coreweave.com/audience` set to that same custom audience, as described in [Use your ServiceAccount](#use-your-serviceaccount).
To change the default audience for every workload that uses the webhook, set the chart's `config.audience` value instead. See the [Pod Identity Webhook chart values](/products/cks/clusters/coreweave-charts/pod-identity-webhook) for details.
With the OIDC configuration in place, Object Storage trusts tokens issued by your CKS cluster and can map them to access policies.
## Use your ServiceAccount
The Pod Identity Webhook injects the configuration your Pods need to obtain Object Storage credentials. Add one of these annotations to the ServiceAccount that your workload uses:
| Audience type | Annotation syntax |
| - | - |
| Default audience | `caios.coreweave.com/inject: "true"` |
| Custom audience | `caios.coreweave.com/audience: custom` (use when you need to match a specific WIF audience) |
Create a ServiceAccount with the appropriate annotation, then run your workload with that ServiceAccount.
1. Create the following manifest and save it to a file named `object-storage-sa.yaml`. This example uses the default audience annotation. For a custom audience, adjust the annotation:
```yaml title="object-storage-sa.yaml" highlight={6-7} theme={"system"}
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-object-storage-sa
namespace: default
annotations:
caios.coreweave.com/inject: "true"
```
2. Apply the manifest to create the ServiceAccount:
```bash theme={"system"}
kubectl apply -f object-storage-sa.yaml
```
3. Verify that the ServiceAccount exists and has the appropriate annotation:
```bash theme={"system"}
kubectl get serviceaccounts my-object-storage-sa -o jsonpath='{.metadata.annotations.caios\.coreweave\.com/inject}'
true
```
The `true` response confirms that the ServiceAccount has the default audience annotation.
## Grant the ServiceAccount access to Object Storage
You must grant the ServiceAccount's federated identity access to Object Storage through an [organization access policy](/products/storage/object-storage/auth-access/organization-policies/about). To construct the WIF principal, use the OIDC Issuer URL you copied earlier and the ServiceAccount name:
Replace `[OIDC-ISSUER-URL]` with the OIDC Issuer URL from your cluster's details panel.
```text theme={"system"}
role/[OIDC-ISSUER-URL]:system:serviceaccount:default:my-object-storage-sa
```
The verification Pod requires only `cwobject:CreateAccessKeyOIDC` to exchange its OIDC token and `s3:ListAllMyBuckets` to run `aws s3api list-buckets`. Both are global actions and require `resources: ["*"]`. For a production workload, replace `s3:ListAllMyBuckets` with the specific S3 actions and bucket resources that the workload needs.
The following example policy grants only the actions required by the verification Pod:
```json theme={"system"}
{
"policy": {
"version": "v1alpha1",
"name": "wif-service-account-access",
"statements": [
{
"name": "allow-s3-access-for-cks-workload",
"effect": "Allow",
"actions": ["cwobject:CreateAccessKeyOIDC", "s3:ListAllMyBuckets"],
"resources": ["*"],
"principals": ["role/[OIDC-ISSUER-URL]:system:serviceaccount:default:my-object-storage-sa"]
}
]
}
}
```
Create this policy using the [Cloud Console](https://console.coreweave.com/object-storage/access-policies), the [CoreWeave Terraform Provider](https://registry.terraform.io/providers/coreweave/coreweave/latest/docs/resources/object_storage_organization_access_policy), or the Object Storage API. Before you deploy your workload, replace `s3:ListAllMyBuckets` with its required S3 actions and resources. For examples, see [Organization access policy examples](/products/storage/object-storage/auth-access/organization-policies/examples).
After the policy is in place, it authorizes the ServiceAccount's federated identity to call Object Storage.
## Create a workload that uses the ServiceAccount
After you configure federation, the ServiceAccount, and an access policy, you can deploy a workload that obtains credentials through the webhook. This example creates a Pod that uses the ServiceAccount you created earlier.
1. Configure the Pod's S3 endpoints:
To work with AI Object Storage, configure your Pod's S3 endpoints to use virtual addressing style and set the appropriate endpoint URL.
> **Configure S3 endpoints for Pods**
>
> Because the environment variables handle credential exchange, you only need to configure the S3 endpoint and addressing style in your application. For example, with the AWS CLI:
>
> ```bash theme={"system"}
> aws configure set s3.addressing_style virtual
> aws configure set endpoint_url "http://cwlota.com"
> ```
>
> Alternatively, you can mount this configuration file into the container from a ConfigMap and set the `AWS_CONFIG_FILE` variable to the location of the mounted file.
>
> ```ini theme={"system"}
> [default]
> s3 =
> addressing_style = virtual
> endpoint_url = http://cwlota.com
> ```
>
> When your workloads run inside a CKS cluster, use the LOTA endpoint (`cwlota.com`) for optimal performance. For more configuration options (Boto3, s3cmd, Multi-Storage Client), see [Attaching endpoints](/products/storage/object-storage/using-object-storage/configure-endpoints#full-configuration-examples).
2. Create the following manifest and save it to a file named `object-storage-pod.yaml`. Make the following adjustments:
* Replace `[CONTAINER-IMAGE]` with the container image that you want to use for your workload.
* Reference the same ServiceAccount in your Pod so the Pod Identity Webhook can inject credential configuration:
```yaml title="object-storage-pod.yaml" highlight={7, 10} theme={"system"}
apiVersion: v1
kind: Pod
metadata:
name: my-workload
namespace: default
spec:
serviceAccountName: my-object-storage-sa
containers:
- name: app
image: "[CONTAINER-IMAGE]"
# Your app uses the injected env or volume to obtain S3 credentials.
```
3. Apply the manifest to create the workload:
```bash theme={"system"}
kubectl apply -f object-storage-pod.yaml
```
Pods that use this ServiceAccount receive the necessary environment or volume configuration to call Object Storage.
## Optional: Verify the webhook injection
Before you deploy your real workload, you can confirm that the Pod Identity Webhook injects credential configuration correctly. Create a test Pod that uses the `amazon/aws-cli:2.33.2` image and run an S3 command against Object Storage.
1. Create the following manifest and save it to a file named `test-webhook-pod.yaml`. The test Pod uses the ServiceAccount and lists your buckets:
```yaml title="test-webhook-pod.yaml" theme={"system"}
apiVersion: v1
kind: Pod
metadata:
name: test-webhook
namespace: default
spec:
restartPolicy: Never
serviceAccountName: my-object-storage-sa
containers:
- name: test
image: amazon/aws-cli:2.33.2
command:
- sh
- -c
- |
aws configure set s3.addressing_style virtual
aws configure set endpoint_url http://cwlota.com
aws s3api list-buckets
```
2. Apply the manifest:
```bash theme={"system"}
kubectl apply -f test-webhook-pod.yaml
```
3. Check the Pod logs for the response:
```bash theme={"system"}
kubectl logs test-webhook
```
When the webhook injection and your access policies are configured correctly, the logs contain a JSON response that lists your buckets. If you see an `AccessDenied` error, verify the organization policy. It must grant `cwobject:CreateAccessKeyOIDC` and `s3:ListAllMyBuckets` to the correct WIF principal with `resources: ["*"]`.
4. Clean up the test Pod:
```bash theme={"system"}
kubectl delete pod test-webhook
```
## Next steps
* To configure OIDC Workload Federation manually or for debugging, see [Manual OIDC token exchange](/security/tutorials/cks-object-storage-authentication/manual).
* Learn more about [Workload Identity Federation for Object Storage](/products/storage/object-storage/auth-access/workload-identity-federation/use-oidc-tokens).
# Manual OIDC token exchange for Object Storage
Source: https://docs.coreweave.com/security/tutorials/cks-object-storage-authentication/manual
Manually configure OIDC Workload Identity Federation to authenticate CKS workloads to CoreWeave AI Object Storage
This guide describes how to manually configure OpenID Connect (OIDC) Workload Identity Federation between a CoreWeave Kubernetes Service (CKS) cluster and CoreWeave AI Object Storage. This configuration lets CKS workloads authenticate to Object Storage without long-lived static credentials. The resulting Pod receives a projected service account token that it exchanges for temporary S3 credentials at runtime.
You can do this automatically with minimal configuration by installing the [Pod Identity Webhook](/security/tutorials/cks-object-storage-authentication/automatic). When you need full control over the token exchange or need to debug it, use this guide.
## Prerequisites
Before you begin, make sure you have the following in place.
In your CoreWeave organization, you must have:
* A deployed [CKS cluster with OIDC Workload Identity enabled](/products/cks/auth-access/workload-identity/introduction).
The following table separates roles that cover the administrative tasks from the narrower permissions used by this guide:
| Task | Role that covers the task | Granular permissions |
| - | - | - |
| Create or manage the CKS cluster | **CKS Admin** | After the cluster exists, Kubernetes RBAC controls whether you can create the guide's ServiceAccount and Pods. |
| Create the OIDC Workload Federation configuration | **IAM Admin** | CoreWeave IAM Access Policies assign roles rather than individual IAM actions. |
| Create and view the Object Storage organization access policy | **Object Storage Admin** | `cwobject:EnsureAccessPolicy` and `cwobject:ListAccessPolicy` granted to your user through an existing organization access policy. Add `cwobject:DeleteAccessPolicy` if you must delete the policy. |
| Exchange the ServiceAccount token and run the verification command | No CoreWeave IAM role applies to the federated ServiceAccount. | Grant `cwobject:CreateAccessKeyOIDC` and `s3:ListAllMyBuckets` to the WIF principal in an Object Storage organization access policy, with `resources: ["*"]`. |
An Object Storage Admin must create the first organization access policy that delegates granular `cwobject:` permissions. The **Object Storage Admin** role governs the Object Storage control plane; it doesn't grant S3 data access.
In your local environment, you must have:
* A terminal. This tutorial is written for bash and zsh.
* [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) installed and accessible from your terminal path.
* An S3 command-line interface (CLI) client installed and accessible from your terminal path. This guide uses the [AWS CLI](https://aws.amazon.com/cli/). If you use another S3 CLI, adapt the commands as needed.
## Configure the CKS OIDC provider
Register your CKS cluster as a trusted OIDC identity provider for Object Storage. This lets Pods running in the cluster exchange their projected service account tokens for temporary S3 credentials.
1. Launch your CKS cluster.
2. In the Cloud Console, retrieve the **OIDC Issuer URL** of your CKS cluster:
* Go to the [Clusters](https://console.coreweave.com/clusters) page.
* Click the name of your cluster. A cluster details panel opens on the right.
* In the **Cluster access** section, copy the **OIDC Issuer URL**.
3. Configure OIDC Workload Identity Federation in the Cloud Console using this OIDC Issuer URL.
4. On the [Workload Federation](https://console.coreweave.com/organization/iam/workload-federation/oidc) page, create a new OIDC configuration.
* Click **Create OIDC configuration**. The configuration form opens.
* Set the configuration name (choose a name that indicates it is for federating CKS tokens with AI Object Storage).
* Enter the CKS cluster's OIDC Issuer URL that you copied earlier into the **Issuer URL** field.
* Set the **Client ID (Audience)** field to `https://coreweave.com/iam` (recommended) or choose another audience.
* Click **Create** to save the configuration.
## Configure `kubectl` and create a Kubernetes ServiceAccount
1. Download a kubeconfig for your CKS cluster using one of the following methods:
* **Get a kubeconfig for an existing cluster**: Go to the [Clusters](https://console.coreweave.com/clusters) page. Find the cluster, click the vertical ellipsis next to its name, and then click **Get kubeconfig**.
* **Create an API access token and kubeconfig**: Go to the [Tokens](https://console.coreweave.com/tokens) page, and click **Create Token**. Store your token secret securely, set the context by selecting your existing cluster, and then click **Download** to get the kubeconfig.
Save the kubeconfig file to your local machine.
2. To let `kubectl` locate the kubeconfig that you downloaded, export an environment variable. Replace `[PATH-TO-KUBECONFIG-FILE]` with the path to the downloaded file:
```bash theme={"system"}
export KUBECONFIG=[PATH-TO-KUBECONFIG-FILE]
```
3. Test that your `kubectl` installation works:
```bash theme={"system"}
kubectl config get-contexts
kubectl get nodes
```
If the environment variable is set, the downloaded context appears. If the cluster has a ready Node Pool, `kubectl get nodes` also shows its Nodes.
4. Create a Kubernetes ServiceAccount by saving the following data to a file named `test-service-account.yaml`:
```yaml title="test-service-account.yaml" theme={"system"}
apiVersion: v1
kind: ServiceAccount
metadata:
name: test-service-account
namespace: default
```
5. Create the account:
```bash theme={"system"}
kubectl apply -f test-service-account.yaml
```
6. Verify the account:
```bash theme={"system"}
kubectl get serviceaccounts
```
If successful, `test-service-account` is visible.
## Grant the service account access to Object Storage
You must grant the service account's federated identity access to Object Storage through an organization access policy. Use the OIDC Issuer URL you copied earlier and the service account name to construct the workload identity federation (WIF) principal. Replace `[OIDC-ISSUER-URL]` with the OIDC Issuer URL from your cluster's details panel:
```text theme={"system"}
role/[OIDC-ISSUER-URL]:system:serviceaccount:default:test-service-account
```
The verification Pod requires only `cwobject:CreateAccessKeyOIDC` to exchange its OIDC token and `s3:ListAllMyBuckets` to run `aws s3api list-buckets`. Both are global actions and require `resources: ["*"]`. For a production workload, replace `s3:ListAllMyBuckets` with the specific S3 actions and bucket resources that the workload needs.
The following example policy grants only the actions required by the verification Pod:
```json theme={"system"}
{
"policy": {
"version": "v1alpha1",
"name": "wif-service-account-access",
"statements": [
{
"name": "allow-s3-access-for-cks-workload",
"effect": "Allow",
"actions": ["cwobject:CreateAccessKeyOIDC", "s3:ListAllMyBuckets"],
"resources": ["*"],
"principals": ["role/[OIDC-ISSUER-URL]:system:serviceaccount:default:test-service-account"]
}
]
}
}
```
Create this policy with the Cloud Console, the [CoreWeave Terraform Provider](https://registry.terraform.io/providers/coreweave/coreweave/latest/docs/resources/object_storage_organization_access_policy), or the Object Storage API. Before you deploy your workload, replace `s3:ListAllMyBuckets` with its required S3 actions and resources. For examples, see [Organization access policy examples](/products/storage/object-storage/auth-access/organization-policies/examples).
## Set up the workload
With the service account created and granted access, reference it from your Object Storage policies and from your workload Pod.
* You can define Object Storage policies that reference a role of the form `[ISSUER]:[SUBJECT]`, where the issuer is discovered through the well-known URL and the subject follows a standard pattern for Kubernetes service accounts, based on namespace and service account name.
* You can get the OIDC well-known URL from the Cloud Console. On the **Clusters** page, select your cluster, and then copy the OIDC Issuer URL.
When you create Object Storage organization and bucket policies for the service account, use the following principal syntax:
| Principal type | Syntax |
| - | - |
| Principal | `[OIDC-ISSUER-URL]:system:serviceaccount:default:test-service-account` |
| Organization policy `"principals"` | `["role/[OIDC-ISSUER-URL]:system:serviceaccount:default:test-service-account"]` |
| Bucket policy `"Principal"` | `{"AWS": "arn:aws:iam::[ORG-ID]:role/[OIDC-ISSUER-URL]:system:serviceaccount:default:test-service-account"}` |
## Create the workload Pod
Create a Pod manifest that mounts a projected service account token with the correct audience. The Pod uses this token to exchange for temporary Object Storage credentials at runtime.
Replace `[ORG-ID]` with your CoreWeave organization ID and replace `[CONTAINER-IMAGE]` with your application's container image:
To work with AI Object Storage, configure your Pod's S3 endpoints to use virtual addressing style and set the appropriate endpoint URL.
> **Configure S3 endpoints for Pods**
>
> Because the environment variables handle credential exchange, you only need to configure the S3 endpoint and addressing style in your application. For example, with the AWS CLI:
>
> ```bash theme={"system"}
> aws configure set s3.addressing_style virtual
> aws configure set endpoint_url "http://cwlota.com"
> ```
>
> Alternatively, you can mount this configuration file into the container from a ConfigMap and set the `AWS_CONFIG_FILE` variable to the location of the mounted file.
>
> ```ini theme={"system"}
> [default]
> s3 =
> addressing_style = virtual
> endpoint_url = http://cwlota.com
> ```
>
> When your workloads run inside a CKS cluster, use the LOTA endpoint (`cwlota.com`) for optimal performance. For more configuration options (Boto3, s3cmd, Multi-Storage Client), see [Attaching endpoints](/products/storage/object-storage/using-object-storage/configure-endpoints#full-configuration-examples).
```yaml title="workload-pod.yaml" theme={"system"}
apiVersion: v1
kind: Pod
metadata:
name: my-workload
namespace: default
spec:
serviceAccountName: test-service-account
containers:
- name: app
image: "[CONTAINER-IMAGE]"
env:
- name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
value: /var/run/secrets/tokens/oidc-token
- name: AWS_CONTAINER_CREDENTIALS_FULL_URI
value: "https://api.coreweave.com/v1/cwobject/temporary-credentials/oidc/[ORG-ID]"
- name: AWS_DEFAULT_REGION
value: "US-EAST-04A"
volumeMounts:
- name: oidc-token
mountPath: /var/run/secrets/tokens
readOnly: true
volumes:
- name: oidc-token
projected:
sources:
- serviceAccountToken:
path: oidc-token
audience: "https://coreweave.com/iam" # Or a custom audience
expirationSeconds: 3600
```
* Apply the manifest:
```bash theme={"system"}
kubectl apply -f workload-pod.yaml
```
The Pod mounts the projected token at `/var/run/secrets/tokens/oidc-token`. Supported versions of AWS software development kits (SDKs) and the AWS CLI use two environment variables: `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE` and `AWS_CONTAINER_CREDENTIALS_FULL_URI`. The software uses these variables to exchange the OIDC token for temporary S3 credentials. Kubernetes rotates the projected OIDC token according to its time to live, and the AWS credential provider refreshes the temporary S3 credentials before they expire.
## Optional: Verify the setup
To confirm the projected token exchange works before you deploy your real workload, create a test Pod that uses the `amazon/aws-cli:2.33.2` image:
```yaml title="test-manual-pod.yaml" theme={"system"}
apiVersion: v1
kind: Pod
metadata:
name: test-manual-oidc
namespace: default
spec:
restartPolicy: Never
serviceAccountName: test-service-account
containers:
- name: test
image: amazon/aws-cli:2.33.2
command:
- sh
- -c
- |
aws configure set s3.addressing_style virtual
aws configure set endpoint_url http://cwlota.com
aws s3api list-buckets
env:
- name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
value: /var/run/secrets/tokens/oidc-token
- name: AWS_CONTAINER_CREDENTIALS_FULL_URI
value: "https://api.coreweave.com/v1/cwobject/temporary-credentials/oidc/[ORG-ID]"
- name: AWS_DEFAULT_REGION
value: "US-EAST-04A"
volumeMounts:
- name: oidc-token
mountPath: /var/run/secrets/tokens
readOnly: true
volumes:
- name: oidc-token
projected:
sources:
- serviceAccountToken:
path: oidc-token
audience: "https://coreweave.com/iam" # Or a custom audience
expirationSeconds: 3600
```
1. Apply the test Pod:
```bash theme={"system"}
kubectl apply -f test-manual-pod.yaml
```
2. Check the Pod logs for the response:
```bash theme={"system"}
kubectl logs test-manual-oidc
```
When your OIDC configuration and access policies are configured correctly, the logs contain a JSON response that lists your buckets. If you see an `AccessDenied` error, verify the organization policy. It must grant `cwobject:CreateAccessKeyOIDC` and `s3:ListAllMyBuckets` to the correct WIF principal with `resources: ["*"]`.
3. Clean up the test Pod:
```bash theme={"system"}
kubectl delete pod test-manual-oidc
```
## Optional: Validate locally
You can also validate the OIDC token exchange from your local machine, outside a Kubernetes Pod. Create a service account token and call the credential endpoint directly. For one-time testing and debugging, this approach isolates the credential exchange from the rest of your workload.
You need the following additional tools for local validation:
* `curl` for calling the credential endpoint.
* `jq` for parsing JSON responses.
* [jwt.io](https://www.jwt.io/) (optional) for inspecting OIDC token contents.
### Test that the service account token can authenticate
Create a service account token with the same audience as your WIF configuration and exchange it for temporary Object Storage credentials. The following example uses the recommended audience, `https://coreweave.com/iam`. In the command, replace `[ORG-ID]` with your CoreWeave organization ID:
```bash theme={"system"}
OIDC_TOKEN=$(kubectl create token -n default test-service-account --audience=https://coreweave.com/iam)
printf 'header = "Authorization: %s"\n' "$OIDC_TOKEN" | curl --config - \
"https://api.coreweave.com/v1/cwobject/temporary-credentials/oidc/[ORG-ID]" | jq
unset OIDC_TOKEN
```
If authentication succeeds, you see a response similar to the following:
```json theme={"system"}
{
"AccessKeyId": "CWAKIAEXAMPLE1234567",
"SecretAccessKey": "wJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEY",
"Token": "",
"Expiration": "2026-03-14T21:44:43Z",
"attributes": {
"cw-role": "https://oidc.example.coreweave.com/abc123:system:serviceaccount:default:test-service-account"
}
}
```
### Test access with the AWS CLI
Configure a local AWS CLI profile that uses a credential process to fetch temporary credentials. To use the [CoreWeave Intelligent CLI](https://github.com/coreweave/cwic) (`cwic`) instead of a custom script, see [Share credentials across processes](/products/storage/object-storage/using-object-storage/share-credentials-across-processes#exchange-an-oidc-token). The Kubernetes Pod example on that page uses the same projected token this tutorial sets up.
To use a custom script, replace `[CREDENTIAL-SCRIPT-PATH]` with the path to your credential script (for example, `/bin/cw-auth.sh`). The script must obtain a service account token, exchange it at the credential endpoint, and return JSON in the format the AWS CLI expects. For a full walkthrough of writing this script, see [Workload Identity Federation with a credential process](/products/storage/object-storage/using-object-storage/s3-clients/aws#workload-identity-federation-with-a-credential-process).
The following commands set the virtual addressing style, the credential process, the region, and the endpoint URL in a new profile called `cw-test`:
```bash theme={"system"}
export AWS_CONTAINER_AUTHORIZATION_TOKEN=$(kubectl create token -n default test-service-account --audience=https://coreweave.com/iam)
export AWS_CONTAINER_CREDENTIALS_FULL_URI="https://api.coreweave.com/v1/cwobject/temporary-credentials/oidc/[ORG-ID]"
aws configure set s3.addressing_style virtual --profile cw-test
aws configure set credential_process "[CREDENTIAL-SCRIPT-PATH]" --profile cw-test
aws configure set region "US-EAST-04A" --profile cw-test
aws configure set endpoint_url "https://cwobject.com" --profile cw-test
```
> [!NOTE]
>
> Use the primary endpoint (`https://cwobject.com`) for local testing because the LOTA endpoint (`http://cwlota.com`) is accessible only from within CoreWeave clusters.
Run an S3 command to verify the credential exchange works:
```bash theme={"system"}
aws s3api list-buckets --profile cw-test
```
If the credential exchange succeeds, the command lists the buckets that the federated identity can access.
## Next steps
After you validate the setup, consider the following options:
* To use automatic authentication when you don't need manual control, see [Automatic authentication with the Pod Identity Webhook](/security/tutorials/cks-object-storage-authentication/automatic).
* To review the CKS configuration, see [OIDC Workload Identity for CKS](/products/cks/auth-access/workload-identity/introduction).
* Review [Workload Identity Federation for Object Storage](/products/storage/object-storage/auth-access/workload-identity-federation/use-oidc-tokens).
# Deploy network policies with CKS
Source: https://docs.coreweave.com/security/tutorials/deploying-network-policies
Enforce Pod-level network segmentation on CKS clusters.
This tutorial demonstrates how to implement basic network policies on CoreWeave Kubernetes Service (CKS) clusters. These policies segment and secure Pod-to-Pod communication. By the end, you have a working default-deny network policy and a targeted allow rule, validated with test Pods. You learn the rationale behind each step, CoreWeave-specific best practices, and how to validate your configuration.
This tutorial is for platform engineers and cluster operators who need to enforce Pod-level network segmentation on CKS.
## Prerequisites
* **CKS cluster:** You need access to a CKS cluster. CKS clusters use either the Cilium or Calico container network interface (CNI). Cilium enforces network policies with extended Berkeley Packet Filter (eBPF).
* **`kubectl` access:** Ensure `kubectl` is installed and configured for your cluster identity and namespace.
### Create or use an existing namespace
Namespaces provide logical segmentation and isolation in Kubernetes. They are foundational for multi-tenancy and enforcing network policies scoped to individual teams or workloads. This step ensures your resources don't interfere with others and that network policies apply only within your segment.
Replace `[NAMESPACE]` in the following examples with a name relevant to your application.
```bash theme={"system"}
kubectl create ns [NAMESPACE]
```
With a namespace ready, the next step is to deploy the Pods that your network policies govern.
## Deploy sample workloads
Deploy two Pods and a Service:
* `backend`: an NGINX server exposing port `80`, labeled `app: backend`.
* `backend` Service: a stable endpoint that sends port `80` traffic to the `backend` Pod.
* `frontend`: a BusyBox Pod running `sleep`, labeled `app: frontend`.
These two Pods let you demonstrate segmentation: by restricting which Pods can reach `backend`, you exercise least privilege for service access.
```bash theme={"system"}
kubectl apply -n [NAMESPACE] -f - < [!NOTE]
>
> The following code snippets are meant to be run in order.
## Create an artifact
Create an artifact with four lines of code:
1. Create a [W\&B run](/products/wandb/runs).
2. Create an artifact object with [`wandb.Artifact`](/products/wandb/ref/python/experiments/artifact).
3. Add one or more files, such as a model file or dataset, to the artifact object with `wandb.Artifact.add_file()`.
4. Log your artifact to Weights & Biases with `wandb.Run.log_artifact()`.
For example, the following code snippet shows how to log a file called `dataset.h5` to an artifact called `example_artifact`:
```python theme={"system"}
import wandb
with wandb.init(project="artifacts-example", job_type="add-dataset") as run:
artifact = wandb.Artifact(name="example_artifact", type="dataset")
artifact.add_file(local_path="./dataset.h5", name="training_dataset")
run.log_artifact(artifact)
```
* The `type` of the artifact affects how it appears in Weights & Biases. If you do not specify a `type`, it defaults to `unspecified`.
* Each label of the dropdown represents a different `type` parameter value. In the above code snippet, the artifact's `type` is `dataset`.
> [!NOTE]
>
> See the [track external files](/products/wandb/artifacts/track-external-files) page for information on how to add references to files or directories stored in external object storage, like an Amazon S3 bucket.
## Download an artifact
Indicate the artifact you want to mark as input to your run with the [`wandb.Run.use_artifact()`](/products/wandb/ref/python/experiments/run#use_artifact) method.
Continuing from the previous code snippet, the following code example shows how to use the artifact called `example_artifact` that was created earlier:
```python theme={"system"}
with wandb.init(project="artifacts-example", job_type="add-dataset") as run:
artifact = run.use_artifact("training_dataset:latest") # returns a run object using the "my_data" artifact
```
This returns an artifact object.
Next, use the returned object to download all contents of the artifact:
```python theme={"system"}
datadir = artifact.download() # downloads the full `my_data` artifact to the default directory.
```
> [!NOTE]
>
> You can pass a custom path into the `root` [parameter](/products/wandb/ref/python/experiments/artifact) to download an artifact to a specific directory. For alternate ways to download artifacts and to see additional parameters, see the guide on [downloading and using artifacts](/products/wandb/artifacts/download-and-use-an-artifact).
## Next steps
* Learn how to [version](/products/wandb/artifacts/create-a-new-artifact-version) and [update](/products/wandb/artifacts/update-an-artifact) artifacts.
* Learn how to trigger downstream workflows or notify a Slack channel in response to changes to your artifacts with [automations](/products/wandb/automations).
* Learn about the [registry](/products/registry), a space that houses trained models.
* Explore the [Python SDK](/products/wandb/ref/python/experiments/artifact) and [CLI](/products/wandb/ref/cli/wandb-artifact) reference guides.
# Tutorial: Create, track, and use a dataset artifact
Source: https://docs.coreweave.com/products/wandb/artifacts/artifacts-walkthrough
Create, track, and use a dataset artifact with Weights & Biases.
This walkthrough demonstrates how to create, track, and use a dataset artifact.
## 1. Log into Weights & Biases
Import the W\&B library and log in to Weights & Biases. You will need to sign up for a free Weights & Biases account if you have not done so already.
```python theme={"system"}
import wandb
wandb.login()
```
## 2. Initialize a run
Use [`wandb.init()`](/products/wandb/ref/python/functions/init) to initialize a run. This generates a background process to sync and log data. Provide a project name and a job type:
```python theme={"system"}
# Create a W&B Run. Here we specify 'dataset' as the job type since this example
# shows how to create a dataset artifact.
with wandb.init(project="artifacts-example", job_type="upload-dataset") as run:
# Your code here
```
## 3. Create an artifact object
Create an artifact object with the [`wandb.Artifact()`](/products/wandb/ref/python/experiments/artifact). Provide a name for the artifact and a description of the file type for the `name` and `type` parameters, respectively.
For example, the following code snippet demonstrates how to create an artifact called `‘bicycle-dataset’` with a `‘dataset’` label:
```python theme={"system"}
artifact = wandb.Artifact(name="bicycle-dataset", type="dataset")
```
For more information about how to construct an artifact, see [Construct artifacts](/products/wandb/artifacts/construct-an-artifact).
## 4. Add the dataset to the artifact
Add a file to the artifact. Common file types include models and datasets. The following example adds a dataset named `dataset.h5` that is saved locally on our machine to the artifact:
```python theme={"system"}
# Add a file to the artifact's contents
artifact.add_file(local_path="dataset.h5")
```
Replace the filename `dataset.h5` in the previous code snippet with the path to the file you want to add to the artifact.
## 5. Log the dataset
Use the W\&B run objects `wandb.Run.log_artifact()` method to both save your artifact version and declare the artifact as an [output of the run](/products/wandb/artifacts/explore-and-traverse-an-artifact-graph).
```python theme={"system"}
# Save the artifact version to W&B and mark it
# as the output of this run
run.log_artifact(artifact)
```
A `'latest'` [alias](/products/wandb/artifacts/create-a-custom-alias) is created by default when you log an artifact. For more information about artifact aliases and versions, see [Create a custom alias](/products/wandb/artifacts/create-a-custom-alias) and [Create new artifact versions](/products/wandb/artifacts/create-a-new-artifact-version), respectively.
Putting this together, you script so far should look like this:
```python theme={"system"}
import wandb
wandb.login()
with wandb.init(project="artifacts-example", job_type="upload-dataset") as run:
artifact = wandb.Artifact(name="bicycle-dataset", type="dataset")
artifact.add_file(local_path="dataset.h5")
run.log_artifact(artifact)
```
## 6. Download and use the artifact
The following code example demonstrates the steps you can take to use an artifact you have logged and saved to the Weights & Biases servers.
1. First, initialize a new run object with **`wandb.init()`.**
2. Second, use the run objects [`wandb.Run.use_artifact()`](/products/wandb/ref/python/experiments/run#use_artifact) method to tell Weights & Biases what artifact to use. This returns an artifact object.
3. Third, use the artifacts [`wandb.Artifact.download()`](/products/wandb/ref/python/experiments/artifact#download) method to download the contents of the artifact.
```python theme={"system"}
# Create a W&B Run. Here we specify 'training' for 'type'
# because we will use this run to track training.
with wandb.init(project="artifacts-example", job_type="training") as run:
# Query W&B for an artifact and mark it as input to this run
artifact = run.use_artifact("bicycle-dataset:latest")
# Download the artifact's contents
artifact_dir = artifact.download()
```
Alternatively, you can use the Public API (`wandb.Api`) to export (or update data) data already saved in Weights & Biases outside of a Run. See [Track external files](/products/wandb/artifacts/track-external-files) for more information.
# Create an artifact
Source: https://docs.coreweave.com/products/wandb/artifacts/construct-an-artifact
Create and log a W&B Artifact. Learn how to add one or more files or a URI reference to an Artifact.
Use the W\&B Python SDK to construct artifacts from [W\&B Runs](/products/wandb/ref/python/experiments/run). You can add [files, directories, URIs, and files from parallel runs to artifacts](#add-files-to-an-artifact). After you add a file to an artifact, save the artifact to the W\&B Server or [your own private server](/products/wandb/platform/hosting/hosting-options/self-managed). Each artifact is associated with a run.
For information on how to track external files, such as files stored in Amazon S3, see the [Track external files](/products/wandb/artifacts/track-external-files) page.
## Construct an artifact
Construct a [W\&B Artifact](/products/wandb/ref/python/experiments/artifact) in three steps:
1. [Create an artifact Python object with `wandb.Artifact()`](/products/wandb/artifacts/construct-an-artifact#create-an-artifact-python-object-with-wandb-artifact)
2. [Add one or more files to the artifact](/products/wandb/artifacts/construct-an-artifact#add-one-or-more-files-to-the-artifact)
3. [Save your artifact to the W\&B server](/products/wandb/artifacts/construct-an-artifact#save-your-artifact-to-the-w\&b-server)
### Create an artifact Python object with `wandb.Artifact()`
Initialize the [`wandb.Artifact()`](/products/wandb/ref/python/experiments/artifact) class to create an artifact object. Specify the following parameters:
* **Name**: The name of your artifact. The name should be unique, descriptive, and memorable.
* **Type**: The type of artifact. The type should be simple, descriptive, and correspond to a single step of your machine learning pipeline. Common artifact types include `'dataset'` or `'model'`.
> [!NOTE]
>
> Weights & Biases uses the "name" and "type" you provide to create a directed acyclic graph in the W\&B App. See the [Explore and traverse artifact graphs](/products/wandb/artifacts/explore-and-traverse-an-artifact-graph) for more information.
Copy and paste the following code snippet to create an artifact object. Replace the `"artifact-name"` and `"artifact-type"` placeholders with your own values:
```python theme={"system"}
import wandb
# Create an artifact object
artifact = wandb.Artifact(name="artifact-name", type="artifact-type")
```
> [!NOTE]
>
> Artifacts can't share the same name, even if they have different types. For example, you can't create an artifact named `cats` with type `dataset` and another artifact named `cats` with type `model`.
You can optionally provide a description, metadata, digest algorithm, and other parameters when you initialize an artifact object.
For example, the following code snippet creates an artifact object with a name, type, description, metadata, and digest algorithm:
```python theme={"system"}
import wandb
# Create an artifact object
artifact = wandb.Artifact(
name="my-dataset",
type="dataset",
description="This is my dataset artifact",
metadata={"source": "generated", "version": 1},
digest_algorithm="XXH128"
)
```
> [!NOTE]
>
> MD5 remains the default digest algorithm. With W\&B Python SDK 0.29.0 or later, set `digest_algorithm="XXH128"` to opt in to faster hashing for new artifacts. Existing artifact collections that use MD5 continue to use MD5. SDK versions before 0.29.0 cannot verify artifacts that contain XXH128 entries with `Artifact.verify()`.
For more information on available attributes and parameters, see the [`wandb.Artifact`](/products/wandb/ref/python/experiments/artifact) Class definition in the Python SDK Reference Guide.
### Add one or more files to the artifact
[Add files, directories, external URI references (such as Amazon S3) and more](/products/wandb/artifacts/construct-an-artifact#add-files-to-an-artifact) to your artifact object.
To add a single file, use the artifact object's [`Artifact.add_file()`](/products/wandb/ref/python/experiments/artifact#add_file) method:
```python theme={"system"}
artifact.add_file(local_path="path/to/file.txt", name="artifact-name")
```
To add a directory, use the [`Artifact.add_dir()`](/products/wandb/ref/python/experiments/artifact#add_dir) method:
```python theme={"system"}
artifact.add_dir(local_path="path/to/directory", name="artifact-name")
```
See the next section, [Add files to an artifact](/products/wandb/artifacts/construct-an-artifact#add-files-to-an-artifact), for more information on how to add different file types to an artifact.
### Save your artifact to the W\&B server
Save your artifact to the W\&B server. Use the run object's [`wandb.Run.log_artifact()`](/products/wandb/ref/python/experiments/run#log_artifact) method to save the artifact.
```python theme={"system"}
with wandb.init(project="", job_type="") as run:
run.log_artifact(artifact)
```
> [!TIP]
>
> **When to use to use `wandb.Run.log_artifact()` or `Artifact.save()`**
>
> * Use `wandb.Run.log_artifact()` to create a new artifact and associate it with a specific run.
> * Use `Artifact.save()` to update an existing artifact without creating a new run.
Putting this all together, the following code snippet demonstrates how to create a dataset artifact, add a file to the artifact, and save the artifact to Weights & Biases:
```python theme={"system"}
import wandb
artifact = wandb.Artifact(name="artifact-name", type="artifact-type")
artifact.add_file(local_path="path/to/file.txt", name="filename")
artifact.add_dir(local_path="path/to/directory", name="filename")
with wandb.init(project="", job_type="") as run:
run.log_artifact(artifact)
```
Each time you log an artifact with the same name and type, Weights & Biases creates a new version of that artifact. For more information, see [Create a new artifact version](/products/wandb/artifacts/create-a-new-artifact-version).
> [!WARNING]
>
> Weights & Biases performs calls `wandb.Run.log_artifact()` asynchronously for performant uploads. This can cause surprising behavior when logging artifacts in a loop. For example:
>
> ```python theme={"system"}
> with wandb.init() as run:
> for i in range(10):
> a = wandb.Artifact(name = "race",
> type="dataset",
> metadata={
> "index": i,
> },
> )
> # ... add files to artifact a ...
> run.log_artifact(a)
> ```
>
> The artifact version **v0** might not have an index of 0 in its metadata because artifacts may be logged in an arbitrary order.
## Add files to an artifact
The following sections demonstrate how to add different types of objects to an artifact. Assume you have a directory with the following structure as you read through the examples:
```text theme={"system"}
root-directory
| - hello.txt
| - images/
| -- | cat.png
| -- | dog.png
| - checkpoints/
| -- | model.h5
| - models/
| -- | model.h5
```
### Add a single file
Use [`wandb.Artifact.add_file()`](/products/wandb/ref/python/experiments/artifact#method-artifact-add-file) to add a single local file to an artifact. Provide the local path to the file as the `local_path` parameter:
```python theme={"system"}
import wandb
# Initialize an artifact object
artifact = wandb.Artifact(name="artifact-name", type="artifact-type")
# Add a single file
artifact.add_file(local_path="path/file.format")
```
For example, suppose you had a file called `'hello.txt'` in your working local directory.
```python theme={"system"}
artifact.add_file("hello.txt")
```
The artifact now has the following content:
```text theme={"system"}
hello.txt
```
Optionally, pass a different name to the `name` parameter to rename the file within the artifact object itself. Continuing the previous example:
```python theme={"system"}
artifact.add_file(
local_path="hello.txt",
name="new/path/hello_world.txt"
)
```
The artifact is stored as:
```text theme={"system"}
new/path/hello_world.txt
```
The following table shows how different API calls produce different artifact contents:
| API Call | Resulting artifact |
| - | - |
| `artifact.new_file('hello.txt')` | `hello.txt` |
| `artifact.add_file('model.h5')` | `model.h5` |
| `artifact.add_file('checkpoints/model.h5')` | `model.h5` |
| `artifact.add_file('model.h5', name='models/mymodel.h5')` | `models/mymodel.h5` |
### Add multiple files
Use the [`wandb.Artifact.add_dir()`](/products/wandb/ref/python/experiments/artifact#method-artifact-add-dir) method to add multiple files from a local directory to an artifact. Provide the local path to the directory as the `local_path` parameter.
```python theme={"system"}
import wandb
# Initialize an artifact object
artifact = wandb.Artifact(name="artifact-name", type="artifact-type")
# Add a local directory to the artifact
artifact.add_dir(local_path="path/file.format", name="optional-prefix")
```
The following table show how different API calls produce different artifact contents:
| API Call | Resulting artifact |
| - | - |
| `artifact.add_dir('images')` | `cat.png``dog.png` |
| `artifact.add_dir('images', name='images')` | `images/cat.png``images/dog.png` |
### Add a URI reference
Artifacts track checksums and other information for reproducibility if the URI has a scheme that the W\&B library supports.
Add an external URI reference to an artifact with the [`wandb.Artifact.add_reference()`](/products/wandb/ref/python/experiments/artifact#method-artifact-add-reference) method. Replace the `'uri'` string with your own URI. Optionally pass the desired path within the artifact for the name parameter.
```python theme={"system"}
# Add a URI reference
artifact.add_reference(uri="uri", name="optional-name")
```
Artifacts support the following URI schemes:
* `http(s)://`: A path to a file accessible over HTTP. The artifact will track checksums in the form of etags and size metadata if the HTTP server supports the `ETag` and `Content-Length` response headers.
* `s3://`: A path to an object or object prefix in S3. The artifact will track checksums and versioning information (if the bucket has object versioning enabled) for the referenced objects. Object prefixes are expanded to include the objects under the prefix, up to a maximum of 10,000 objects.
* `gs://`: A path to an object or object prefix in GCS. The artifact will track checksums and versioning information (if the bucket has object versioning enabled) for the referenced objects. Object prefixes are expanded to include the objects under the prefix, up to a maximum of 10,000 objects.
The following table shows how different API calls produce different artifact contents:
| API call | Resulting artifact contents |
| - | - |
| `artifact.add_reference('s3://my-bucket/model.h5')` | `model.h5` |
| `artifact.add_reference('s3://my-bucket/checkpoints/model.h5')` | `model.h5` |
| `artifact.add_reference('s3://my-bucket/model.h5', name='models/mymodel.h5')` | `models/mymodel.h5` |
| `artifact.add_reference('s3://my-bucket/images')` | `cat.png``dog.png` |
| `artifact.add_reference('s3://my-bucket/images', name='images')` | `images/cat.png``images/dog.png` |
### Add files to artifacts from parallel runs
For large datasets or distributed training, multiple parallel runs might need to contribute to a single artifact.
```python theme={"system"}
import wandb
import time
# This example uses Ray to runs in parallel
# for demonstration purposes.
import ray
ray.init()
artifact_type = "dataset"
artifact_name = "parallel-artifact"
table_name = "distributed_table"
parts_path = "parts"
num_parallel = 5
# Each batch of parallel writers should have its own
# unique group name.
group_name = "writer-group-{}".format(round(time.time()))
@ray.remote
def train(i):
"""
Our writer job. Each writer will add one image to the artifact.
"""
with wandb.init(group=group_name) as run:
artifact = wandb.Artifact(name=artifact_name, type=artifact_type)
# Add data to a wandb table.
table = wandb.Table(columns=["a", "b", "c"], data=[[i, i * 2, 2**i]])
# Add the table to folder in the artifact
artifact.add(table, "{}/table_{}".format(parts_path, i))
# Upserting the artifact creates or appends data to the artifact
run.upsert_artifact(artifact)
# Launch your runs in parallel
result_ids = [train.remote(i) for i in range(num_parallel)]
# Join on all the writers to make sure their files have
# been added before finishing the artifact.
ray.get(result_ids)
# Once all the writers are finished, finish the artifact
# to mark it ready.
with wandb.init(group=group_name) as run:
artifact = wandb.Artifact(artifact_name, type=artifact_type)
# Create a "PartitionTable" pointing to the folder of tables
# and add it to the artifact.
artifact.add(wandb.data_types.PartitionedTable(parts_path), table_name)
# Finish artifact finalizes the artifact, disallowing future "upserts"
# to this version.
run.finish_artifact(artifact)
```
## Find path for logged artifacts and other metadata
The following code snippet shows how to use the [W\&B Public API](/products/wandb/ref/python/public-api) to list the files in a run, including their names and URLs. Replace the `` placeholder with your own values:
```python theme={"system"}
from wandb.apis.public.files import Files
from wandb.apis.public.api import Api
# Example run object
run = Api().run("")
# Create a Files object to iterate over files in the run
files = Files(api.client, run)
# Iterate over files
for file in files:
print(f"File Name: {file.name}")
print(f"File URL: {file.url}")
print(f"Path to file in the bucket: {file.direct_url}")
```
See the [File](/products/wandb/ref/python/public-api/file) Class for more information on available attributes and methods.
# Create an artifact alias
Source: https://docs.coreweave.com/products/wandb/artifacts/create-a-custom-alias
Create and manage custom aliases to reference specific W&B artifact versions by meaningful names like best or production.
Use aliases as pointers to specific versions. By default, `wandb.Run.log_artifact()` adds the `latest` alias to the logged version.
Weights & Biases creates an artifact version `v0` and attaches it to your artifact when you log that artifact for the first time. Weights & Biases checksums the contents when you log again to the same artifact. If the artifact changed, Weights & Biases saves a new version `v1`.
For example, if you want your training script to pull the most recent version of a dataset, specify `latest` when you use that artifact. The following code example demonstrates how to download a recent dataset artifact named `bike-dataset` that has an alias, `latest`:
```python theme={"system"}
import wandb
with wandb.init(project="") as run:
artifact = run.use_artifact("bike-dataset:latest")
artifact.download()
```
You can also apply a custom alias to an artifact version. For example, if you want to mark that model checkpoint is the best on the metric AP-50, you could add the string `'best-ap50'` as an alias when you log the model artifact.
```python theme={"system"}
with wandb.init(project="") as run:
artifact = wandb.Artifact("run-3nq3ctyy-bike-model", type="model")
artifact.add_file("model.h5")
run.log_artifact(artifact, aliases=["latest", "best-ap50"])
```
# Create an artifact version
Source: https://docs.coreweave.com/products/wandb/artifacts/create-a-new-artifact-version
Create a new artifact version from a single run or from a distributed process.
Create a new artifact version with a single [run](/products/wandb/runs) or collaboratively with distributed runs. You can optionally create a new artifact version from a previous version known as an [incremental artifact](#create-a-new-artifact-version-from-an-existing-version).
> [!NOTE]
>
> We recommend that you create an incremental artifact when you need to apply changes to a subset of files in an artifact, where the size of the original artifact is significantly larger.
## Create new artifact versions from scratch
There are two ways to create a new artifact version: from a single run and from distributed runs. They are defined as follows:
* **Single run**: A single run provides all the data for a new version. This is the most common case and is best suited when the run fully recreates the needed data. For example: outputting saved models or model predictions in a table for analysis.
* **Distributed runs**: A set of runs collectively provides all the data for a new version. This is best suited for distributed jobs which have multiple runs generating data, often in parallel. For example: evaluating a model in a distributed manner, and outputting the predictions.
Weights & Biases will create a new artifact and assign it a `v0` alias if you pass a name to the `wandb.Artifact` API that does not exist in your project. Weights & Biases checksums the contents when you log again to the same artifact. If the artifact changed, Weights & Biases saves a new version `v1`.
Weights & Biases will retrieve an existing artifact if you pass a name and artifact type to the `wandb.Artifact` API that matches an existing artifact in your project. The retrieved artifact will have a version greater than 1.
### Single run
Log a new version of an Artifact with a single run that produces all the files in the artifact. This case occurs when a single run produces all the files in the artifact.
Based on your use case, select one of the tabs below to create a new artifact version:
> [!TIP]
>
> Use `wandb.Run.log_artifact()` to explicitly mark an artifact as an output of a run.
>
> Use `Artifact.save()` as a convenience method. If a run is active, Weights & Biases logs the artifact to that run. Otherwise, Weights & Biases creates an automatic run and logs it there.
**Track as run output**
Use `wandb.Run.log_artifact()` to explicitly associate the artifact with a run:
1. Create a run with `wandb.init()`.
2. Create a new artifact or retrieve an existing one with `wandb.Artifact`.
3. Add files to the artifact with `wandb.Artifact.add_file()`.
4. Log the artifact to the run with `wandb.Run.log_artifact()`.
```python theme={"system"}
with wandb.init() as run:
artifact = wandb.Artifact("artifact_name", "artifact_type")
# Add Files and Assets to the artifact using
# `.add`, `.add_file`, `.add_dir`, and `.add_reference`
artifact.add_file("image1.png")
run.log_artifact(artifact)
```
**Save directly**
Use `wandb.Artifact.save()` as a convenience method to persist the artifact from the current context:
1. Create a new artifact or retrieve an existing one with `wandb.Artifact`.
2. Add files to the artifact with `wandb.Artifact.add_file()`.
3. Save the artifact with `wandb.Artifact.save()`.
```python theme={"system"}
artifact = wandb.Artifact("artifact_name", "artifact_type")
# Add Files and Assets to the artifact using
# `.add`, `.add_file`, `.add_dir`, and `.add_reference`
artifact.add_file("image1.png")
artifact.save()
```
### Distributed runs
Allow a collection of runs to collaborate on a version before committing it. This is in contrast to single run mode described above where one run provides all the data for a new version.
> [!NOTE]
>
> 1. Each run in the collection needs to be aware of the same unique ID (called `distributed_id`) in order to collaborate on the same version. By default, if present, Weights & Biases uses the run's `group` as set by `wandb.init(group=GROUP)` as the `distributed_id`.
> 2. There must be a final run that "commits" the version, permanently locking its state.
> 3. Use `upsert_artifact` to add to the collaborative artifact and `finish_artifact` to finalize the commit.
Consider the following example. Different runs (labelled below as **Run 1**, **Run 2**, and **Run 3**) add a different image file to the same artifact with `upsert_artifact`.
#### Run 1
```python theme={"system"}
with wandb.init() as run:
artifact = wandb.Artifact("artifact_name", "artifact_type")
# Add Files and Assets to the artifact using
# `.add`, `.add_file`, `.add_dir`, and `.add_reference`
artifact.add_file("image1.png")
run.upsert_artifact(artifact, distributed_id="my_dist_artifact")
```
#### Run 2
```python theme={"system"}
with wandb.init() as run:
artifact = wandb.Artifact("artifact_name", "artifact_type")
# Add Files and Assets to the artifact using
# `.add`, `.add_file`, `.add_dir`, and `.add_reference`
artifact.add_file("image2.png")
run.upsert_artifact(artifact, distributed_id="my_dist_artifact")
```
#### Run 3
Must run after Run 1 and Run 2 complete. The Run that calls `wandb.Run.finish_artifact()` can include files in the artifact, but does not need to.
```python theme={"system"}
with wandb.init() as run:
artifact = wandb.Artifact("artifact_name", "artifact_type")
# Add Files and Assets to the artifact
# `.add`, `.add_file`, `.add_dir`, and `.add_reference`
artifact.add_file("image3.png")
run.finish_artifact(artifact, distributed_id="my_dist_artifact")
```
## Create a new artifact version from an existing version
Create a new artifact version by adding, modifying, or removing files from an existing artifact version. Weights & Biases re-indexes only the files that changed. The resulting version is known as an *incremental* artifact.
To create an incremental artifact, create a *draft* from any previously committed artifact version. You can then modify the draft by using methods such as `Artifact.add_file()`, `Artifact.add_dir()`, and `Artifact.remove()`. For a complete list of methods, see the [`Artifact` reference documentation](/products/wandb/ref/python/experiments/artifact).
After you finish modifying the draft, log or save it to create a new artifact version.
> [!NOTE]
>
> You can create a new artifact from scratch instead of creating an incremental artifact. However, this approach requires all artifact files to be available on your local disk. With an incremental artifact, you can add, remove, or modify individual files while reusing unchanged files from the previous artifact version.
You can create an incremental artifact using a single run or across multiple runs that contribute to the new version. The following sections describe how to create an incremental artifact in each case.
### Single run
**Track as run input**
Use `wandb.Run.use_artifact()` when the current run consumes the artifact. Weights & Biases records the artifact as an input to the run.
1. Retrieve the artifact version with `wandb.Run.use_artifact()`.
2. Create a draft with `Artifact.new_draft()`.
3. Add, remove, or modify entries in the draft.
4. Log the draft with `wandb.Run.log_artifact()` to create a new artifact version.
```python lines theme={"system"}
with wandb.init(job_type="modify dataset") as run:
saved_artifact = run.use_artifact(
"my_artifact:latest"
) # fetch artifact and input it into your run
draft_artifact = saved_artifact.new_draft() # create a draft version
# modify a subset of files in the draft version
draft_artifact.add_file("file_to_add.txt")
draft_artifact.remove("dir_to_remove/")
run.log_artifact(
draft_artifact
) # log your changes to create a new version and mark it as output to your run
```
**Fetch directly**
Use the Public API to retrieve an artifact without recording it as a run input.
1. Retrieve the artifact version with `wandb.Api.artifact()`.
2. Create a draft with `Artifact.new_draft()`.
3. Add, remove, or modify entries in the draft.
4. Save the draft version with `Artifact.save()` to create a new artifact version.
```python theme={"system"}
client = wandb.Api()
saved_artifact = client.artifact("my_artifact:latest") # load your artifact
draft_artifact = saved_artifact.new_draft() # create a draft version
# modify a subset of files in the draft version
draft_artifact.remove("deleted_file.txt")
draft_artifact.add_file("modified_file.txt")
draft_artifact.save() # commit changes to the draft
```
### Distributed runs
Use `Artifact.upsert_artifact()` and `Artifact.finish_artifact()` when multiple runs contribute files to one artifact version.
The following example shows how to create a new artifact version from distributed runs in the case where the artifact is used as [an input to the runs](/products/wandb/artifacts/create-a-new-artifact-version#use-as-run-input). The same approach applies when the artifact is [fetched directly](/products/wandb/artifacts/create-a-new-artifact-version#fetch-directly) with the Public API.
```python theme={"system"}
import wandb
project = ""
artifact_name = ""
distributed_id = "dataset-build-001"
worker_files = [
("data/shards/worker-0.txt", "shards/worker-0.txt"),
("data/shards/worker-1.txt", "shards/worker-1.txt"),
]
for local_path, artifact_path in worker_files:
with wandb.init(project=project, group=distributed_id) as run:
artifact = wandb.Artifact(artifact_name, type="dataset")
artifact.add_file(local_path, name=artifact_path)
run.upsert_artifact(artifact, distributed_id=distributed_id)
with wandb.init(project=project, group=distributed_id) as run:
artifact = wandb.Artifact(artifact_name, type="dataset")
logged = run.finish_artifact(
artifact,
aliases=["latest"],
distributed_id=distributed_id,
)
logged.wait()
```
> [!TIP]
>
> Use `Artifact.new_draft()` when you are modifying a committed artifact version.
>
> Use `Artifact.upsert_artifact()` and `Artifact.finish_artifact()` when multiple runs need to collaborate on one artifact version before it is finalized.
# Artifact data privacy and compliance
Source: https://docs.coreweave.com/products/wandb/artifacts/data-privacy-and-compliance
Learn where Weights & Biases files are stored by default. Explore how to save, store sensitive information.
When you log artifacts, Weights & Biases uploads the files to a Google Cloud bucket. The contents of the bucket are encrypted at rest and in transit. Only users with access to the project can see the files.
When you delete a version of an artifact, it is marked for soft deletion in our database and removed from your storage cost. When you delete an entire artifact, it is queued for permanent deletion and all of its contents are removed from the Weights & Biases bucket. If you have specific needs around file deletion, reach out to [Customer Support](mailto:forge-support@coreweave.com).
By default, deleted artifacts are retained for 7 days and can be restored during this period, which is configurable for Dedicated Cloud. Learn more about data retention in [Multi-tenant Cloud](/products/wandb/platform/hosting/hosting-options/multi_tenant_cloud#data-retention-policy) or [Dedicated Cloud](/products/wandb/platform/hosting/hosting-options/dedicated-cloud#data-retention-policy).
For sensitive datasets that cannot reside in a multi-tenant environment, you can use [W\&B Dedicated Cloud](/products/wandb/platform/hosting/hosting-options/dedicated-cloud) or [reference artifacts](/products/wandb/artifacts/track-external-files). Reference artifacts track references to private buckets without sending file contents to Weights & Biases. Reference artifacts maintain links to files on your buckets or servers. Weights & Biases only keeps track of the metadata associated with the files, not the files themselves.
For more information about deployment options, see [Dedicated Cloud](/products/wandb/platform/hosting/hosting-options/dedicated-cloud) or [Self-Managed](/products/wandb/platform/hosting/hosting-options/self-managed). To discuss your specific requirements, [contact CoreWeave](https://www.coreweave.com/contact-us).
# Delete an artifact
Source: https://docs.coreweave.com/products/wandb/artifacts/delete-artifacts
Delete artifacts interactively with the App UI or programmatically with the W&B Python SDK.
Delete artifacts interactively with the W\&B App or programmatically with the W\&B Python SDK. When you delete an artifact, Weights & Biases marks that artifact as a *soft-delete*. In other words, the artifact is marked for deletion but files are not immediately deleted from storage.
The contents of the artifact remain as a soft-delete, or pending deletion state, until a regularly run garbage collection process reviews all artifacts marked for deletion. The garbage collection process deletes associated files from storage if the artifact and its associated files are not used by a previous or subsequent artifact versions.
> [!NOTE]
>
> Garbage collection is **best-effort**. Weights & Biases does not guarantee how quickly freed space appears in your object storage after you delete an artifact. Large deployments or backlogs can take longer than expected. For how this fits with run data, retention settings, and optional operator actions, see [Manage bucket storage and costs](/products/wandb/platform/hosting/managing-bucket-storage).
## Artifact garbage collection workflow
The following diagram illustrates the complete artifact garbage collection process:
```mermaid theme={"system"}
graph TB
Start([Artifact Deletion Initiated]) --> DeleteMethod{Deletion Method}
DeleteMethod -->|UI| UIDelete[Delete via W&B App]
DeleteMethod -->|SDK| SDKDelete[Delete via W&B Python SDK]
DeleteMethod -->|TTL| TTLDelete[TTL Policy Expires]
UIDelete --> SoftDelete[Artifact Marked as
'Soft Delete'] SDKDelete --> SoftDelete TTLDelete --> SoftDelete SoftDelete --> GCWait[(Wait for
best-effort
Garbage Collection)] GCWait --> GCRun[Garbage Collection
Process Runs
- Reviews all soft-deleted artifacts
- Checks file dependencies] GCRun --> CheckUsage{Are files used by
other artifact versions?} CheckUsage -->|Yes| KeepFiles[Files Kept in Storage
- Artifact marked deleted
- Files remain for other versions] CheckUsage -->|No| DeleteFiles[Files Deleted from Storage
- Artifact fully removed
- Storage space reclaimed] KeepFiles --> End([End]) DeleteFiles --> End style Start fill:#e1f5fe,stroke:#333,stroke-width:2px,color:#000 style SoftDelete fill:#fff3e0,stroke:#333,stroke-width:2px,color:#000 style GCRun fill:#f3e5f5,stroke:#333,stroke-width:2px,color:#000 style KeepFiles fill:#e8f5e9,stroke:#333,stroke-width:2px,color:#000 style DeleteFiles fill:#ffebee,stroke:#333,stroke-width:2px,color:#000 style End fill:#e0e0e0,stroke:#333,stroke-width:2px,color:#000 ``` You can schedule when artifacts are deleted from Weights & Biases with TTL policies. For more information, see [Manage data retention with Artifact TTL policy](/products/wandb/artifacts/ttl). > [!NOTE] > > Artifacts deleted by a TTL policy, the W\&B Python SDK, or the W\&B App are first soft-deleted. Soft-deleted artifacts are then garbage-collected before they are permanently deleted. > [!NOTE] > > Deleting an entity, project, or artifact collection triggers the artifact deletion process described on this page. When you delete a run and choose to delete its associated artifacts, those artifacts follow the same soft-delete and garbage collection workflow. ## Delete an artifact version Delete an artifact version interactively with the W\&B App or programmatically with the W\&B Python SDK. **W\&B App** To delete an artifact version: 1. Navigate to the project that contains the artifact version you want to delete. 2. Select the **Artifacts** tab. 3. From the list of artifact types, select the type of artifact that contains the version you want to delete. 4. Click the **action ()** menu next to the artifact version you want to delete. 5. From the dropdown, choose **Delete Version**. **W\&B Python SDK** Delete an artifact version programmatically with the [wandb.Artifact.delete()](/products/wandb/ref/python/experiments/artifact#delete) method. Provide the full name of the artifact. The full name consists of `//:`. Set the `delete_aliases` parameter to `True` to delete the artifact even if it has one or more aliases associated with it.
```python theme={"system"}
import wandb
api = wandb.Api()
# Get the artifact by its path
artifact = api.artifact("//:")
# Delete the artifact version along with any aliases
artifact.delete(delete_aliases=True)
```
## Delete multiple artifact versions
The following code example shows how to delete multiple artifact versions. Provide the entity, project name, and run ID that created the artifact as arguments to `wandb.Api.run()`. This returns a run object that you can use to access all artifact versions created by that run. Next, iterate through the artifact versions and delete the ones that match your criteria.
> [!TIP]
>
> Set the `delete_aliases` parameter to `True` (`wandb.Artifact.delete(delete_aliases=True)`) to delete an artifact version and any aliases associated with it.
Replace the ``, ``, ``, and `` placeholders with your own values:
```python theme={"system"}
import wandb
# Initialize W&B API
api = wandb.Api()
# Get the run by its path. Consists of //
run = api.run("//")
# Specify the artifact name to delete versions for
artifact_name = ""
# Search and delete artifact versions with the specified name
for artifact in run.logged_artifacts():
print(f"Found artifact: {artifact.name}") # Example name run_4dfbufgq_model:v0
# Grab only the artifact name without the version with split()
if artifact.name.split(":")[0] == artifact_name:
print(f"Deleting artifact version: {artifact.name}")
artifact.delete(delete_aliases=True)
```
## Delete multiple artifact versions with a specific alias
The following code demonstrates how to delete multiple artifact versions that have a specific alias.
Replace the ``, ``, ``, ``, and `` placeholders with your own values:
```python theme={"system"}
import wandb
# Initialize W&B API
api = wandb.Api()
# Get the run by its path. Consists of //
run = api.run("//")
# Specify the artifact name to delete versions for
artifact_name = ""
# Specify the alias to filter artifact versions for deletion
desired_alias = ""
# Delete artifacts logged to run with alias 'v3' and 'v4
for artifact in run.logged_artifacts():
print(f"Found artifact: {artifact.name}")
if (artifact.name.split(":")[0] == artifact_name) and (desired_alias in artifact.aliases):
artifact.delete(delete_aliases=True)
```
## Delete an artifact collection
**W\&B App**
To delete an artifact collection:
1. Navigate to the artifact collection you want to delete.
2. Select the **action ()** menu next to the artifact collection name.
3. From the dropdown menu, select **Delete**.
**W\&B Python SDK**
Delete artifact collection programmatically with the [wandb.Artifact.delete()](/products/wandb/ref/python/experiments/artifact#delete) method.
Provide the full path of the artifact collection to `wandb.Api.artifact_collection(name="")`. The full path consists of `//`.
```python theme={"system"}
import wandb
# Initialize W&B API
api = wandb.Api()
# Get the artifact collection by its path. Consists of
# //
collection = api.artifact_collection(
type_name = "",
name = "//"
)
collection.delete()
```
## Protected aliases and deletion permissions
Artifacts with protected aliases have special deletion restrictions. [Protected aliases](/products/registry/aliases#protected-aliases) are aliases in the Registry that registry admins can set to prevent unauthorized deletion.
> [!NOTE]
>
> **Important considerations for protected aliases:**
>
> * Artifacts with protected aliases cannot be deleted by non-registry admins.
> * Within a registry, registry admins can unlink protected artifact versions and delete collections/registries that contain protected aliases.
> * For source artifacts: if a source artifact is linked to a registry with a protected alias, it cannot be deleted by any user
> * Registry admins can remove the protected aliases from source artifacts and then delete them.
## Enable garbage collection based on how Weights & Biases is hosted
> [!NOTE]
>
> Garbage collection timing is not guaranteed. See [Manage bucket storage and costs](/products/wandb/platform/hosting/managing-bucket-storage) for details.
Garbage collection is active by default if you use W\&B Multi-tenant Cloud. In W\&B Dedicated and Self-Managed, you might need to take these additional steps to activate garbage collection.
1. **W\&B Self-Managed**: Set `GORILLA_ARTIFACT_GC_ENABLED=true`.
2. **Dedicated Cloud**: Contact support to verify that garbage collection is active.
3. Enable bucket versioning if you use [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html), [Google Cloud](https://cloud.google.com/storage/docs/object-versioning) or any other storage provider such as [Minio](https://min.io/docs/minio/linux/administration/object-management/object-versioning.html#enable-bucket-versioning). If you use Azure, [enable soft deletion](https://learn.microsoft.com/azure/storage/blobs/soft-delete-blob-overview), which is equivalent to bucket versioning.
The following table describes how to satisfy requirements to enable garbage collection based on your deployment type.
The `X` indicates you must satisfy the requirement:
| | Environment variable | Enable versioning |
| - | - | - |
| Multi-tenant Cloud | | |
| Multi-tenant Cloud with [BYOB storage](/products/wandb/platform/hosting/data-security/secure-storage-connector) | | X |
| Dedicated Cloud | | |
| Dedicated Cloud with [BYOB storage](/products/wandb/platform/hosting/data-security/secure-storage-connector) | | X |
| Self-Managed | X | X |
> [!NOTE]
>
> note
> Secure storage connector is currently only available for Google Cloud Platform and Amazon Web Services.
# Download and use artifacts
Source: https://docs.coreweave.com/products/wandb/artifacts/download-and-use-an-artifact
Download and use Artifacts from multiple projects.
Download and use an artifact that is already stored on the W\&B server or construct an artifact object and pass it in for de-duplication as necessary.
## Download and use an artifact stored on Weights & Biases
Download an artifact as a tracked run input when it is a run dependency, or fetch it directly for workflows that don’t require lineage tracking.
> [!NOTE]
>
> References that have schemes that Weights & Biases knows how to handle get downloaded just like artifact files. For more information, see [Track external files](/products/wandb/artifacts/track-external-files).
You can download all files in an artifact or download individual files:
* Use [`Artifact.download()`](/products/wandb/ref/python/experiments/artifact#download) to download all files.
* Use [`Artifact.get_entry()`](/products/wandb/ref/python/experiments/artifact#get_entry) to retrieve a specific entry, and then call `.download()` to download it.
**Track run input**
Use `wandb.Run.use_artifact()` to download an artifact as part of a W\&B run. Weights & Biases records the artifact as an input to the run.
1. Import the W\&B Python SDK and initialize a W\&B [Run](/products/wandb/ref/python/experiments/run).
2. Retrieve the artifact version with `wandb.Run.use_artifact()`.
3. Download all files or a specific file from the returned `Artifact` object.
```python theme={"system"}
import wandb
with wandb.init(project="project-name", job_type="job-type") as run:
# Indicate the artifact to use. Format is "name:alias"
artifact = run.use_artifact("bike-dataset:latest")
# Download the entire artifact
datadir = artifact.download()
# Download a specific file
entry = artifact.get_entry("bike.png")
file_path = entry.download() # downloads only bike.png
```
**Fetch directly**
Use the Public API to download an artifact without creating a run or recording a run input relationship.
1. Import the W\&B Python SDK and create a client object with `wandb.Api()`.
2. Retrieve the artifact version with `wandb.Api.artifact()`.
3. Download all files or a specific file from the returned `Artifact` object.
Specify the artifact path in the format `entity/project/artifact:alias`.
```python theme={"system"}
import wandb
api = wandb.Api()
artifact = api.artifact("entity/project/artifact:alias")
artifact.download()
```
**W\&B CLI**
Use the `wandb artifact get` command to download an artifact from the W\&B server.
```
$ wandb artifact get project/artifact:alias --root mnist/
```
## Partially download an artifact
You can optionally download part of an artifact based on a prefix. Use the `path_prefix=` (`wandb.Artifact.download(path_prefix=)`) parameter to download a single file or the content of a sub-folder.
```python theme={"system"}
with wandb.init(project="project-name", job_type="job-type") as run:
# Indicate the artifact to use. Format is "name:alias"
artifact = run.use_artifact("bike-dataset:latest")
# Download a specific file or sub-folder
artifact.download(path_prefix="bike.png") # downloads only bike.png
```
Alternatively, you can download files from a certain directory. To do so, specify the directory within the `path_prefix=` parameter. Continuing from the previous code snippet:
```python theme={"system"}
# downloads files in the images/bikes directory
artifact.download(path_prefix="images/bikes/")
```
## Verify downloaded artifact files
After you download an artifact, call `wandb.Artifact.verify()` to confirm that the local files on disk match the artifact manifest.
The following example downloads an artifact as part of a run and verifies the downloaded files:
```python theme={"system"}
import wandb
with wandb.init(project="project-name", job_type="job-type") as run:
artifact = run.use_artifact("entity/project/my-dataset:latest")
artifact_dir = artifact.download()
artifact.verify(artifact_dir)
```
[`wandb.Artifact.verify()`](/products/wandb/ref/python/experiments/artifact#method-artifact-verify) recomputes checksums for local files and compares them with the checksums in the artifact manifest. Verification fails if a file is missing or modified, or if the directory contains files that are not included in the artifact.
> [!NOTE]
>
> **Artifact verification considerations**
>
> * `wandb.Artifact.verify()` does not verify [reference entries](/products/wandb/artifacts/track-external-files).
>
> * Artifacts created with `digest_algorithm="XXH128"` require W\&B Python SDK version 0.29.0 or newer to use `wandb.Artifact.verify()`. Earlier SDK versions do not support XXH128 manifest entries, so verification fails.
## Use an artifact from a different project
Specify the name of the artifact along with its project name to reference an artifact. You can also reference artifacts across entities by specifying the name of the artifact with its entity name.
The following code example demonstrates how to query an artifact from another project as input to the current W\&B run.
```python theme={"system"}
with wandb.init(project="project-name", job_type="job-type") as run:
# Query W&B for an artifact from another project and mark it
# as an input to this run.
artifact = run.use_artifact("my-project/artifact:alias")
# Use an artifact from another entity and mark it as an input
# to this run.
artifact = run.use_artifact("my-entity/my-project/artifact:alias")
```
## Construct and use an artifact simultaneously
Simultaneously construct and use an artifact. Create an artifact object and pass it to use\_artifact. This creates an artifact in Weights & Biases if it does not exist yet. The [`wandb.Run.use_artifact()`](/products/wandb/ref/python/experiments/run#use_artifact) API is idempotent, so you can call it as many times as you like.
```python theme={"system"}
import wandb
with wandb.init(project="project-name", job_type="job-type") as run:
artifact = wandb.Artifact("reference model")
artifact.add_file("model.h5")
run.use_artifact(artifact)
```
For more information about constructing an artifact, see [Construct an artifact](/products/wandb/artifacts/construct-an-artifact).
# Explore artifact lineage graphs
Source: https://docs.coreweave.com/products/wandb/artifacts/explore-and-traverse-an-artifact-graph
View and traverse artifact lineage graphs to track the inputs and outputs of W&B runs as a directed acyclic graph.
Weights & Biases tracks the inputs and outputs of runs using directed acyclic graphs (DAGs) called *lineage graphs*. Lineage graphs are visual representations of the relationships between artifacts and runs in an ML experiment. They show how data and models
flow through different stages of the ML lifecycle, from raw data ingestion to model training and evaluation.
Tracking artifact lineage provides several key advantages:
* **Reproducibility**: Enables teams to reproduce experiments, models, and results for debugging, experimentation, and validation.
* **Version control**: Tracks changes to artifacts over time, allowing teams to revert to previous data or model versions when needed.
* **Auditing**: Maintains a detailed record of artifacts and transformations to support compliance and governance.
* **Collaboration**: Helps to improve teamwork by making experiment history transparent, reducing duplicated effort, and accelerating development.
## View an artifact's lineage graph
To view an artifact's lineage graph:
1. Navigate to your project's workspace in the W\&B App.
2. Click on the **Artifacts** tab in the project sidebar.
3. Select an artifact, then click the **Lineage** tab.
## Navigate lineage graphs
The lineage graph is a visual representation of the relationships between artifacts and runs in an ML experiment. Use the W\&B App UI or the Python SDK to explore and traverse an artifact's lineage graph.
**W\&B App UI**
Nodes with green icons represent runs. Nodes with blue icons represent artifacts. Arrows between nodes indicate the input and output of a run or artifact.
Artifact nodes display the artifact's name along with the version of the artifact in the form `:`. An artifact's type is displayed above the name of the artifact.
> [!NOTE]
>
> You can view the type and the name of artifact in both the left sidebar and in the lineage graph node.
Run nodes display the run's name.
Click any individual run to get more information about that runs such as the run's: start time, time duration, author, job type, and more. Click any individual artifact to get more information about the artifact's: aliases, creation time, type, version, description, the run that logged the artifact, file size, and more.
Runs that create multiple versions of the same artifact are grouped together in a cluster. Click on a specific artifact version listed within the cluster to view specific information about that artifact version.
Click and drag a node to rearrange the graph to customize the layout. You can also zoom in and out of the graph to get a better view of the nodes and their relationships.
Hover your mouse over a node and click on the eye icon to hide or show a node in the graph. This is useful for decluttering the graph to focus on specific nodes and their relationships.
**W\&B Python SDK**
Programmatically navigate a graph using the W\&B Python SDK. Use an artifact object's
[`logged_by()`](/products/wandb/ref/python/experiments/artifact#method-artifact-logged-by) and [`used_by()`](/products/wandb/ref/python/experiments/artifact#method-artifact-used-by) methods to walk the graph:
```python theme={"system"}
with wandb.init() as run:
artifact = run.use_artifact("artifact_name:latest")
# Walk up and down the graph from an artifact:
producer_run = artifact.logged_by()
consumer_runs = artifact.used_by()
```
## Enable lineage graph tracking
To enable lineage graph tracking, you need to mark artifacts as [inputs](/products/wandb/artifacts/explore-and-traverse-an-artifact-graph) or
[outputs](/products/wandb/artifacts/explore-and-traverse-an-artifact-graph#track-the-output-of-a-run) of a run using the W\&B Python SDK.
### Track the input of a run
Mark an artifact as the input (or dependency) of a run with the [`wandb.Run.use_artifact()`](/products/wandb/ref/python/experiments/run#method-runuse_artifact)
method. Specify the name of the artifact and an optional alias to reference a specific version of that artifact. The name of the
artifact is in the format `:` or `:`.
Replace values enclosed in angle brackets (`< >`) with your values:
```python theme={"system"}
import wandb
# Initialize a run
with wandb.init(entity="", project="") as run:
# Get artifact, mark it as a dependency
artifact = run.use_artifact(artifact_or_name="", aliases="")
```
### Track the output of a run
Use [`wandb.Run.log_artifact()`](/products/wandb/ref/python/experiments/run#log_artifact) to declare an artifact as an output of a run. First,
create an artifact with the [`wandb.Artifact()`](/products/wandb/ref/python/experiments/artifact#wandb.Artifact) constructor. Then, log the
artifact as an output of the run with `wandb.Run.log_artifact()`.
Replace values enclosed in angle brackets (`< >`) with your values:
```python theme={"system"}
import wandb
# Initialize a run
with wandb.init(entity="", project="") as run:
# Create an artifact
artifact = wandb.Artifact(name = "", type = "")
artifact.add_file(local_path = "", name="")
# Log the artifact as an output of the run
run.log_artifact(artifact_or_path = artifact)
```
## Artifact clusters
When a level of the graph has five or more runs or artifacts, it creates a cluster. A cluster has a search bar to find specific versions of runs or artifacts and pulls an individual node from a cluster to continue investigating the lineage of a node inside a cluster.
Clicking on a node opens a preview with an overview of the node. Clicking on the arrow extracts the individual run or artifact so you can examine the lineage of the extracted node.
# Manage artifact storage and memory allocation
Source: https://docs.coreweave.com/products/wandb/artifacts/storage
Manage storage, memory allocation of W&B Artifacts.
Weights & Biases stores artifact files in a private Google Cloud Storage bucket located in the United States by default. All files are encrypted at rest and in transit.
For sensitive files, we recommend you set up [Private Hosting](/products/wandb/platform/hosting) or use [reference artifacts](/products/wandb/artifacts/track-external-files).
During training, Weights & Biases locally saves logs, artifacts, and configuration files in the following local directories:
| File | Default location | To change default location set: |
| - | - | - |
| logs | `./wandb` | `dir` in `wandb.init()` or set the `WANDB_DIR` environment variable |
| artifacts | `~/.cache/wandb` | the `WANDB_CACHE_DIR` environment variable |
| configs | `~/.config/wandb` | the `WANDB_CONFIG_DIR` environment variable |
| staging artifacts for upload | `~/.cache/wandb-data/` | the `WANDB_DATA_DIR` environment variable |
| downloaded artifacts | `./artifacts` | the `WANDB_ARTIFACT_DIR` environment variable |
For a complete guide to using environment variables to configure Weights & Biases, see the [environment variables reference](/products/wandb/track/environment-variables).
> [!WARNING]
>
> Depending on the machine on `wandb` is initialized on, these default folders may not be located in a writeable part of the file system. This might trigger an error.
### Clean up local artifact cache
Weights & Biases caches artifact files to speed up downloads across versions that share files in common. Over time this cache directory can become large. Run the [`wandb artifact cache cleanup`](/products/wandb/ref/cli/wandb-artifact/wandb-artifact-cache) command to prune the cache and to remove any files that have not been used recently.
The following code snippet demonstrates how to limit the size of the cache to 1GB. Copy and paste the code snippet into your terminal:
```bash theme={"system"}
$ wandb artifact cache cleanup 1GB
```
# Track external files
Source: https://docs.coreweave.com/products/wandb/artifacts/track-external-files
Track files saved in an external bucket, HTTP file server, or an NFS share.
Use *reference artifacts* to track and use files saved outside of Weights & Biases servers. Common external storage solutions include: CoreWeave AI Object Storage, Backblaze B2 Cloud Storage, an Amazon Simple Storage Service (Amazon S3) bucket, GCS bucket, Azure blob, HTTP file server, or NFS share.
Reference artifacts behave similar to non-reference artifacts. The key difference is that the reference artifacts only consists of metadata about the files, such as their sizes and MD5 checksums. The files themselves never leave your system.
You can interact with reference artifact similarly to non-reference artifacts. In the W\&B App, you can browse the contents of the reference artifact using the file browser, explore the full dependency graph, and scan through the versioned history of your artifact. However, the UI cannot render rich media such as images, audio, because the data itself is not contained within the artifact.
> [!NOTE]
>
> If you log an artifact that does not track external files, Weights & Biases saves the artifact's files to Weights & Biases servers. This is the default behavior when you log artifacts with the W\&B Python SDK.
>
> If you log an artifact that tracks external files, Weights & Biases logs metadata about the object, such as the object's ETag and size. If object versioning is enabled on the bucket, the version ID is also logged.
The following sections describe how to track external reference artifacts.
## Track an artifact in an external bucket
Use the W\&B Python SDK to track references to files stored outside of Weights & Biases.
1. Initialize a run with `wandb.init()`.
2. Create an artifact object with `wandb.Artifact()`.
3. Specify the reference to the bucket path with the artifact object's `wandb.Artifact.add_reference()` method.
4. Log the artifact's metadata with `run.log_artifact()`.
```python theme={"system"}
import wandb
# Initialize a W&B run
with wandb.init(project="my-project") as run:
# Create an artifact object
artifact = wandb.Artifact(name="name", type="type")
# Add a reference to the bucket path
artifact.add_reference(uri = "uri/to/your/bucket/path")
# Log the artifact's metadata
run.log_artifact(artifact)
```
As an example, suppose your bucket has the following directory structure:
```text theme={"system"}
s3://my-bucket
|datasets/
|-- mnist/
|models/
|-- cnn/
```
The `datasets/mnist/` directory contains a collection of images. To track the image `datasets/mnist/` directory as a dataset artifact, specify:
1. Provide a name for the artifact, such as `"mnist"`.
2. Set the `type` parameter to `"dataset"` when you construct the artifact object (`wandb.Artifact(type="dataset")`).
3. Provide the path to the `datasets/mnist/` directory as an Amazon S3 URI (`s3://my-bucket/datasets/mnist/`) when you call `wandb.Artifact.add_reference()`.
4. Log the artifact with `run.log_artifact()`.
The following code sample creates a reference artifact `mnist:latest`:
```python theme={"system"}
import wandb
with wandb.init(project="my-project") as run:
artifact = wandb.Artifact(name="mnist", type="dataset")
artifact.add_reference(uri="s3://my-bucket/datasets/mnist")
run.log_artifact(artifact)
```
Within the W\&B App, you can look through the contents of the reference artifact using the file browser, [explore the full dependency graph](/products/wandb/artifacts/explore-and-traverse-an-artifact-graph), and scan through the versioned history of your artifact. The W\&B App does not render rich media such as images, audio, and so forth because the data itself is not contained within the artifact.
> [!NOTE]
>
> W\&B Artifacts support any Amazon S3-compatible interface, including CoreWeave Storage, Backblaze B2 Cloud Storage, and MinIO. The scripts described below work as-is with these providers when you set the `AWS_S3_ENDPOINT_URL` environment variable to point at your S3-compatible server (for example, `https://s3.us-west-001.backblazeb2.com` for Backblaze B2).
> [!WARNING]
>
> By default, Weights & Biases imposes a 10,000 object limit when adding an object prefix. You can adjust this limit by specifying `max_objects=` when you call `wandb.Artifact.add_reference()`.
## Download an artifact from an external bucket
Weights & Biases retrieves the files from the underlying bucket when it downloads a reference artifact using the metadata recorded when the artifact is logged. If your bucket has object versioning enabled, Weights & Biases retrieves the object version that corresponds to the state of the file at the time an artifact was logged. As you evolve the contents of your bucket, you can always point to the exact version of your data a given model was trained on, because the artifact serves as a snapshot of your bucket during the training run.
The following code sample shows how to download a reference artifact. The APIs for downloading artifacts are the same for both reference and non-reference artifacts:
```python theme={"system"}
import wandb
with wandb.init(project="my-project") as run:
artifact = run.use_artifact("mnist:latest", type="dataset")
artifact_dir = artifact.download()
```
> [!NOTE]
>
> Weights & Biases recommends that you enable 'Object Versioning' on your storage buckets if you overwrite files as part of your workflow.
>
> If versioning is enabled, Weights & Biases can always retrieve the correct version of the file when you download an artifact, even if the file has been overwritten since the artifact was logged.
>
> Based on your use case, read the instructions to enable object versioning: [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html), [Google Cloud](https://cloud.google.com/storage/docs/using-object-versioning#set), [Azure](https://learn.microsoft.com/azure/storage/blobs/versioning-enable).
## Add and download an external from a bucket
The following code sample uploads a dataset to an Amazon S3 bucket, tracks it with a reference artifact, then downloads it:
```python theme={"system"}
import boto3
import wandb
with wandb.init() as run:
# Training here...
s3_client = boto3.client("s3")
s3_client.upload_file(file_name="my_model.h5", bucket="my-bucket", object_name="models/cnn/my_model.h5")
# Log the model artifact
model_artifact = wandb.Artifact("cnn", type="model")
model_artifact.add_reference("s3://my-bucket/models/cnn/")
run.log_artifact(model_artifact)
```
At a later point, you can download the model artifact. Specify the name of the artifact and its type:
```python theme={"system"}
import wandb
with wandb.init() as run:
artifact = run.use_artifact(artifact_or_name = "cnn", type="model")
datadir = artifact.download()
```
> [!NOTE]
>
> See the following reports for an end-to-end walkthrough on how to track artifacts by reference for Google Cloud or Azure:
>
> * [Guide to Tracking Artifacts by Reference with Google Cloud](https://forge.coreweave.com/wandb/stacey/artifacts/reports/Tracking-Artifacts-by-Reference--Vmlldzo1NDMwOTE)
> * [Working with Reference Artifacts in Microsoft Azure](https://forge.coreweave.com/wandb/andrea0/azure-2023/reports/Efficiently-Harnessing-Microsoft-Azure-Blob-Storage-with-Weights-Biases--Vmlldzo0NDA2NDgw)
## Cloud storage credentials
Weights & Biases uses the default mechanism to look for credentials based on the cloud provider you use. Read the documentation from your cloud provider to learn more about the credentials used:
| Cloud provider | Credentials Documentation |
| - | - |
| CoreWeave AI Object Storage | [CoreWeave AI Object Storage documentation](/products/storage/object-storage/auth-access/manage-access-keys/create-keys) |
| Backblaze B2 Cloud Storage | [Backblaze B2 Application Keys documentation](https://www.backblaze.com/docs/cloud-storage-application-keys) |
| AWS | [Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) |
| Google Cloud | [Google Cloud documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc) |
| Azure | [Azure documentation](https://learn.microsoft.com/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) |
For AWS, if the bucket is not located in the configured user's default region, you must set the `AWS_REGION` environment variable to match the bucket region.
> [!WARNING]
>
> Rich media such as images, audio, video, and point clouds may fail to render in the App UI depending on the CORS configuration of your bucket. Allow listing **app.wandb.ai** in your bucket's CORS settings will allow the W\&B App to properly render such rich media.
>
> If rich media such as images, audio, video, and point clouds does not render in the App UI, ensure that `app.wandb.ai` is allowlisted in your bucket's CORS policy.
## Track an artifact in a filesystem
A common pattern for accessing datasets is to expose an NFS mount point to a remote filesystem on all machines running training jobs. This can be an alternative solution to a cloud storage bucket because from the perspective of the training script, the files appear local to your filesystem.
To track an artifact in a filesystem:
1. Initialize a run with `wandb.init()`.
2. Create an artifact object with `wandb.Artifact()`.
3. Specify the reference to the filesystem path with the artifact object's `wandb.Artifact.add_reference()` method.
4. Log the artifact's metadata with `run.log_artifact()`.
Copy and paste the following code snippet to track files in a mounted filesystem. Replace the values enclosed in angle brackets (`< >`) with your own values.
```python theme={"system"}
import wandb
# Initialize a run
with wandb.init(entity="", project="") as run:
# Create an artifact object
artifact = wandb.Artifact(name="", type="")
# Add a reference to the filesystem path
artifact.add_reference("file:///path/to/dataset/")
# Log the artifact (metadata only)
run.log_artifact(artifact)
```
Note the triple slash in the URL. The first component is the `file://` prefix that denotes the use of filesystem references. The second component is the root `/` of the filesystem. The remaining components are the path to the directory or file you want to track.
As an example, suppose you have a filesystem mounted at `/mount` with the following structure:
```text theme={"system"}
mount
|datasets/
|-- mnist/
|models/
|-- cnn/
```
You want to track the `datasets/mnist/` directory as a dataset artifact. To track it, you could use the following code snippet.
```python theme={"system"}
import wandb
with wandb.init() as run:
artifact = wandb.Artifact("mnist", type="dataset")
artifact.add_reference("file:///mount/datasets/mnist/")
run.log_artifact(artifact)
```
This creates a reference artifact `mnist:latest` that points to the files stored under `/mount/datasets/mnist/`.
> [!WARNING]
>
> By default, Weights & Biases imposes a 10,000 file limit when adding a reference to a directory. You can adjust this limit by specifying `max_objects=` when you call `wandb.Artifact.add_reference()`.
Similarly, to track a model stored at `models/cnn/my_model.h5`, you could use the following code snippet:
```python theme={"system"}
import wandb
with wandb.init() as run:
# Training here...
# Write model to disk
# Create an artifact object
model_artifact = wandb.Artifact("cnn", type="model")
# Add a reference to the model file path
model_artifact.add_reference("file:///mount/cnn/my_model.h5")
# Log the artifact to W&B
run.log_artifact(model_artifact)
```
## Download an artifact from an external filesystem
Download files from a referenced filesystem using the same APIs as non-reference artifacts:
1. Initialize a run with `wandb.init()`.
2. Use the `wandb.Run.use_artifact()` method to indicate the artifact you want to download.
3. Call the artifact's `wandb.Artifact.download()` method to download the files from the referenced filesystem
```python theme={"system"}
with wandb.init() as run:
artifact = run.use_artifact("entity/project/mnist:latest", type="dataset")
artifact_dir = artifact.download()
```
Weights & Biases copies the contents of `/mount/datasets/mnist` to the `artifacts/mnist:v0/` directory.
> [!NOTE]
>
> `Artifact.download()` throws an error if it cannot reconstruct the artifact. For example, if an artifact contains a reference to a file that was overwritten, `Artifact.download()` will throw an error because the artifact can no longer be reconstructed.
# Manage artifact data retention
Source: https://docs.coreweave.com/products/wandb/artifacts/ttl
Set time-to-live (TTL) policies on W&B artifacts to schedule automatic deletion and manage storage consumption.
[Try in Colab](https://colab.research.google.com/github/wandb/examples/blob/kas-artifacts-ttl-colab/colabs/wandb-artifacts/WandB_Artifacts_Time_to_live_TTL_Walkthrough.ipynb)
Schedule when artifacts are deleted from Weights & Biases with a W\&B Artifact time-to-live (TTL) policy. When you delete an artifact, Weights & Biases marks that artifact as a *soft-delete*. In other words, the artifact is marked for deletion but files are not immediately deleted from storage. For more information on how Weights & Biases deletes artifacts, see the [Delete artifacts](/products/wandb/artifacts/delete-artifacts) page.
Watch a [Managing data retention with Artifacts TTL](https://www.youtube.com/watch?v=hQ9J6BoVmnc) video tutorial to learn how to manage data retention with Artifacts TTL in the W\&B App.
> [!NOTE]
>
> * Only team admins can view a [team's settings](/products/wandb/platform/app/settings-page/teams) and access team level TTL settings such as (1) permitting who can set or edit a TTL policy or (2) setting a team default TTL.
> * If you do not see the option to set or edit a TTL policy in an artifact's details in the W\&B App UI or if setting a TTL programmatically does not successfully change an artifact's TTL property, your team admin has not given you permissions to do so.
## Auto-generated Artifacts
Only user-generated artifacts can use TTL policies. Artifacts auto-generated by Weights & Biases cannot have TTL policies set for them.
The following Artifact types indicate an auto-generated Artifact:
* `run_table`
* `code`
* `job`
* Any Artifact type starting with: `wandb-*`
You can check an Artifact's type on the [Weights & Biases UI](/products/wandb/artifacts/explore-and-traverse-an-artifact-graph) or programmatically:
```python theme={"system"}
import wandb
with wandb.init(project="") as run:
artifact = run.use_artifact(artifact_or_name="")
print(artifact.type)
```
Replace the values enclosed with `<>` with your own.
## Define who can edit and set TTL policies
Define who can set and edit TTL policies within a team. You can either grant TTL permissions only to team admins, or you can grant both team admins and team members TTL permissions.
> [!NOTE]
>
> Only team admins can define who can set or edit a TTL policy.
1. Navigate to your team’s profile page.
2. Select the **Settings** tab.
3. Navigate to the **Artifacts time-to-live (TTL) section**.
4. From the **TTL permissions dropdown**, select who can set and edit TTL policies.
5. Click on **Review and save settings**.
6. Confirm the changes and select **Save settings**.
## Create a TTL policy
Set a TTL policy for an artifact either when you create the artifact or retroactively after the artifact is created.
For all the code snippets below, replace the content wrapped in `<>` with your information to use the code snippet.
### Set a TTL policy when you create an artifact
Use the W\&B Python SDK to define a TTL policy when you create an artifact. TTL policies are typically defined in days.
> [!NOTE]
>
> Defining a TTL policy when you create an artifact is similar to how you normally [create an artifact](/products/wandb/artifacts/construct-an-artifact). With the exception that you pass in a time delta to the artifact's `ttl` attribute.
The steps are as follows:
1. [Create an artifact](/products/wandb/artifacts/construct-an-artifact).
2. [Add content to the artifact](/products/wandb/artifacts/construct-an-artifact#add-files-to-an-artifact) such as files, a directory, or a reference.
3. Define a TTL time limit with the [`datetime.timedelta`](https://docs.python.org/3/library/datetime.html) data type that is part of Python's standard library.
4. [Log the artifact](/products/wandb/artifacts/construct-an-artifact#3-save-your-artifact-to-the-wb-server).
The following code snippet demonstrates how to create an artifact and set a TTL policy.
```python theme={"system"}
import wandb
from datetime import timedelta
with wandb.init(project="", entity="") as run:
artifact = wandb.Artifact(name="", type="")
artifact.add_file("")
artifact.ttl = timedelta(days=30) # Set TTL policy
run.log_artifact(artifact)
```
The preceding code snippet sets the TTL policy for the artifact to 30 days. In other words, Weights & Biases deletes the artifact after 30 days.
### Set or edit a TTL policy after you create an artifact
Use the W\&B App UI or the W\&B Python SDK to define a TTL policy for an artifact that already exists.
> [!NOTE]
>
> When you modify an artifact's TTL, the time the artifact takes to expire is still calculated using the artifact's `createdAt` timestamp.
**W\&B Python SDK**
1. [Fetch your artifact](/products/wandb/artifacts/download-and-use-an-artifact).
2. Pass in a time delta to the artifact's `ttl` attribute.
3. Update the artifact with the [`save`](/products/wandb/ref/python/experiments/run#save) method.
The following code snippet shows how to set a TTL policy for an artifact:
```python theme={"system"}
import wandb
from datetime import timedelta
artifact = run.use_artifact("")
artifact.ttl = timedelta(days=365 * 2) # Delete in two years
artifact.save()
```
The preceding code example sets the TTL policy to two years.
**W\&B App**
1. Navigate to your Weights & Biases project in the Weights & Biases App.
2. Select the artifact icon in the project sidebar.
3. From the list of artifacts, expand the artifact type you
4. Select on the artifact version you want to edit the TTL policy for.
5. Click on the **Version** tab.
6. From the dropdown, select **Edit TTL policy**.
7. Within the modal that appears, select **Custom** from the TTL policy dropdown.
8. Within the **TTL duration** field, set the TTL policy in units of days.
9. Select the **Update TTL** button to save your changes.
### Set default TTL policies for a team
> [!NOTE]
>
> Only team admins can set a default TTL policy for a team.
Set a default TTL policy for your team. Default TTL policies apply to all existing and future artifacts based on their respective creation dates. Artifacts with existing version-level TTL policies are not affected by the team's default TTL.
1. Navigate to your team’s profile page.
2. Select the **Settings** tab.
3. Navigate to the **Artifacts time-to-live (TTL) section**.
4. Click on the **Set team's default TTL policy**.
5. Within the **Duration** field, set the TTL policy in units of days.
6. Click on **Review and save settings**.
7/ Confirm the changes and then select **Save settings**.
### Set a TTL policy outside of a run
Use the public API to retrieve an artifact without fetching a run, and set the TTL policy. TTL policies are typically defined in days.
The following code sample shows how to fetch an artifact using the public API and set the TTL policy.
```python theme={"system"}
api = wandb.Api()
artifact = api.artifact("entity/project/artifact:alias")
artifact.ttl = timedelta(days=365) # Delete in one year
artifact.save()
```
## Deactivate a TTL policy
Use the W\&B Python SDK or W\&B App UI to deactivate a TTL policy for a specific artifact version.
**W\&B Python SDK**
1. [Fetch your artifact](/products/wandb/artifacts/download-and-use-an-artifact).
2. Set the artifact's `ttl` attribute to `None`.
3. Update the artifact with the [`save`](/products/wandb/ref/python/experiments/run#save) method.
The following code snippet shows how to turn off a TTL policy for an artifact:
```python theme={"system"}
artifact = run.use_artifact("")
artifact.ttl = None
artifact.save()
```
**W\&B App**
1. Navigate to your Weights & Biases project in the Weights & Biases App.
2. Select the artifact icon in the project sidebar.
3. From the list of artifacts, expand the artifact type you
4. Select on the artifact version you want to edit the TTL policy for.
5. Click on the Version tab.
6. Click the **action ()** menu next to the **Link to registry** button.
7. From the dropdown, select **Edit TTL policy**.
8. Within the modal that appears, select **Deactivate** from the TTL policy dropdown.
9. Select the **Update TTL** button to save your changes.
## View TTL policies
View TTL policies for artifacts with the Python SDK or with the W\&B App UI.
**W\&B Python SDK**
Use a print statement to view an artifact's TTL policy. The following example shows how to retrieve an artifact and view its TTL policy:
```python theme={"system"}
artifact = run.use_artifact("")
print(artifact.ttl)
```
**W\&B App**
View a TTL policy for an artifact with the W\&B App UI.
1. Navigate to the [W\&B App](https://forge.coreweave.com/wandb).
2. Navigate to your Weights & Biases project.
3. Within your project, select the Artifacts tab in the project sidebar.
4. Click on a collection.
Within the collection view you can see all of the artifacts in the selected collection. Within the `Time to Live` column you will see the TTL policy assigned to that artifact.
## TTL policies for linked artifact versions
Weights & Biases deactivates TTL policies for linked artifact versions to prevent artifacts used in production workflows from expiring.
When you link an artifact version to a Registry collection, Weights & Biases deactivates its TTL policy and sets its TTL to `None`. If you later unlink the artifact version, Weights & Biases does not restore the previous TTL policy or its duration. Set a new TTL policy after unlinking if the artifact version still requires one.
# Update an artifact
Source: https://docs.coreweave.com/products/wandb/artifacts/update-an-artifact
Update an existing artifact while a run is active or using only the Public API.
Pass desired values to update the `description`, `metadata`, and `alias` of an artifact. Update a run previously logged to Weights & Biases with the W\&B Public API with ([`wandb.Api`](/products/wandb/ref/python/public-api/api)). Use `wandb.Run.save()` to update an artifact when is first initialized and still active.
> [!NOTE]
>
> **When to use wandb.Artifact.save() or wandb.Run.log\_artifact()**
>
> * Use `Artifact.save()` to update an existing artifact without starting a new run.
> * Use `wandb.Run.log_artifact()` to create a new artifact and associate it with a specific run.
Use the W\&B Public API ([`wandb.Api`](/products/wandb/ref/python/public-api/api)) to update an artifact. Use the wandb.Artifact ([`wandb.Artifact`](/products/wandb/ref/python/experiments/artifact)) Class while a run is active.
> [!WARNING]
>
> You can not update the alias of artifact linked to a model in Model Registry.
**During a run**
The following code example demonstrates how to update the description of an artifact using the [`wandb.Artifact`](/products/wandb/ref/python/experiments/artifact) API:
```python theme={"system"}
import wandb
with wandb.init(project="") as run:
artifact = run.use_artifact(":")
artifact.description = ""
artifact.save()
```
**W\&B Public API**
The following example updates an artifact with [`wandb.Api`](/products/wandb/ref/python/public-api/api):
```python theme={"system"}
import wandb
api = wandb.Api()
artifact = api.artifact("entity/project/artifact:alias")
# Update the description
artifact.description = "My new description"
# Selectively update metadata keys
artifact.metadata["oldKey"] = "new value"
# Replace the metadata entirely
artifact.metadata = {"newKey": "new value"}
# Add an alias
artifact.aliases.append("best")
# Remove an alias
artifact.aliases.remove("latest")
# Completely replace the aliases
artifact.aliases = ["replaced"]
# Persist all artifact modifications
artifact.save()
```
For more information, see the Weights & Biases [Artifact API](/products/wandb/ref/python/experiments/artifact).
**With collections**
You can also update an Artifact collection in the same way as a singular artifact:
```python theme={"system"}
import wandb
with wandb.init(project="") as run:
api = wandb.Api()
artifact = api.artifact_collection(type="", collection="")
artifact.name = ""
artifact.description = ""
artifact.save()
```
For more information, see the [Artifacts Collection](/products/wandb/ref/python/public-api/api) reference.
# Automations overview
Source: https://docs.coreweave.com/products/wandb/automations
Use W&B Automations for triggering workflows based on events in Weights & Biases
*Automations* send Slack notifications, run webhooks, or send prompts to ARIA in response to specific events in Weights & Biases. You can scope an automation to an organization, a team, project, registry, or collection. An automation observes the event across everything within its scope, including projects, registries, and collections created later. The steps to create an automation and the available events depend on the scope. For event types by scope, see [Automation events and scopes](/products/wandb/automations/automation-events).
> [!NOTE]
>
> Team-scoped and Organization-scoped automations and the [Automations hub](#automations-hub) are available on W\&B Multi-tenant Cloud and on W\&B Dedicated Cloud (Server v0.83.0 and later).
W\&B Automations follow this pattern: when an **event** occurs and optional **conditions** are met, an **action** runs automatically. For example:
* When a run fails (event), notify a Slack channel (action).
* When the `production` alias is added to an artifact (event), call a webhook to trigger deployment (action).
* When a new model version is linked (event), send a prompt to ARIA to compare it against production (action).
Events and available conditions differ for automations scoped to a [project](/products/wandb/automations/automation-events#project) or a [registry](/products/wandb/automations/automation-events#registry). See [Automation events and scopes](/products/wandb/automations/automation-events).
```mermaid theme={"system"}
flowchart LR
Event[Event] --> Condition[Condition]
Condition --> Action[Action]
```
**Example:** Run fails (event) and optional run name filter (condition) then Slack notification (action). Or: alias `production` added (event) then webhook (action).
## Where to create automations
* **In the Automations hub**: Select **Automations** in the left navigation to see every automation you can access across all scopes: organization, team, project, registry, or collection. You choose the scope when you create an automation. See [Automations hub](/products/wandb/automations#automations-hub).
* **In a project**: Open the project, then select the **Automations** tab in the project sidebar.
* **In a registry**: Open the registry, then select the **Automations** tab.
## Automations hub
The **Automations** hub is a global surface in the left navigation that aggregates every automation you can access, across all scopes and projects, in a single list. Use it to discover, inspect, and manage automations without navigating to each project or registry.
From the **Automations** hub, you can:
* Create an automation and choose its scope, without needing to know the scoped entry point first.
* View all automations you can access in one list, then filter and sort them (for example, by scope, trigger, or action).
* Click an automation's name to view its full configuration details.
* View an automation's [execution history](/products/wandb/automations/view-automation-history) to check whether it succeeded or failed.
* Use the quick filter to switch between automations you created and all automations in your organization.
* Select multiple automations and delete them at once.
The **Automations** hub complements the **Automations** tab in each project and registry; those scoped tabs remain available for quick targeted access.
## Use cases
* **Run monitoring and alerting**: Notify the team when a run fails or when a metric crosses a threshold (for example, loss goes to NaN or accuracy drops).
* **Registry CI/CD**: When a new model version is linked or an alias (such as `staging` or `production`) is added, trigger a webhook to run tests or deploy.
* **Project artifact workflows**: When a new artifact version is created or an alias is added in a project, run a downstream job or post to Slack.
* **Automated analysis**: When a run fails or a new model version is linked, send a prompt to [ARIA](/products/aria) to diagnose the failure or compare the new version against production.
For full event and scope details, see [Automation events and scopes](/products/wandb/automations/automation-events).
## Automation actions
When an event triggers an automation, it can perform one of these actions:
* **Slack notification**: Sends a message to a Slack channel with details about the triggering event. The message summarizes the event and includes a link to view more details in W\&B.
* **Webhook**: Calls a webhook URL with a JSON payload containing information about the triggering event. Use webhooks to integrate with external systems such as CI/CD pipelines, model deployment services, or custom workflows.
* **Trigger ARIA**: Sends a prompt to [ARIA](/products/aria), which starts a new conversation and works on the prompt on your behalf. Use ARIA actions to analyze a new model version, diagnose a failed run, or investigate a metric that crossed a threshold. Available on W\&B Multi-tenant Cloud only.
For implementation details, see [Create a Slack automation](/products/wandb/automations/create-automations/slack), [Create a webhook automation](/products/wandb/automations/create-automations/webhook), or [Create an ARIA automation](/products/wandb/automations/create-automations/aria).
## How automations work
To [create an automation](/products/wandb/automations/create-automations), you:
1. If required, configure [secrets](/products/wandb/platform/secrets) for sensitive strings the automation requires, such as access tokens, passwords, or sensitive configuration details. Secrets are defined in your **Team Settings**. Secrets are most commonly used in webhook automations to securely pass credentials or tokens to the webhook's external service without exposing it in plain text or hard-coding it in the webhook's payload.
2. Configure team-level webhook or Slack integrations to authorize Weights & Biases to post to Slack or run the webhook on your behalf. A single automation action (webhook or Slack notification) can be used by multiple automations. These actions are defined in your **Team Settings**. An ARIA action requires no integration.
3. In the project or registry, create the automation:
1. Define the [event](/products/wandb/automations/automation-events) to watch for, such as when a new artifact version is added.
2. Define the action to take when the event occurs, such as posting to a Slack channel, running a webhook, or sending a prompt to ARIA. For a webhook, specify a secret to use for the access token and/or a secret to send with the payload, if required.
## Recommendations
* **Start small**: Begin with one or two automations for high-value events (for example, run failures or production alias changes). Validate that they work as expected before adding more.
* **Test before production**: Create automations in a test project or with a test webhook or Slack channel first. Manually trigger the event, then confirm that the action runs and that the payload or message looks correct.
* **Avoid alert fatigue**: Use run filters, metric thresholds, or alias patterns to limit how often an automation fires. If you have multiple severities, route them to different channels.
## Limitations
* [Run metric automations](/products/wandb/automations/automation-events#run-metrics-change) and [run metrics z-score change automations](/products/wandb/automations/automation-events#run-metrics-z-score-change) are supported only in [Forge](/products/wandb/platform/hosting/hosting-options/multi_tenant_cloud) and [Dedicated Cloud](/products/wandb/platform/hosting/hosting-options/dedicated-cloud).
* At the **Team** and **Organization** scopes, the W\&B App supports only artifact and collection events. Run-based events aren't available in the App at these scopes. See [Automation events and scopes](/products/wandb/automations/automation-events).
## Next steps
* [Automations tutorial](/products/wandb/automations/tutorial): Guides you to create a project automation to alert on run failures and a Registry automation to run a webhook when an alias is added. The tutorial uses the W\&B App.
* [Create an automation](/products/wandb/automations/create-automations).
* [Automation events and scopes](/products/wandb/automations/automation-events).
* [Create a secret](/products/wandb/platform/secrets).
> [!NOTE]
>
> Looking for companion tutorials for automations?
>
> * [Learn to automatically trigger a Github Action for model evaluation and deployment](https://forge.coreweave.com/wandb/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw).
> * [Watch a video demonstrating automatically deploying a model to a Sagemaker endpoint](https://www.youtube.com/watch?v=s5CMj_w3DaQ).
> * [Watch a video series introducing automations](https://youtube.com/playlist?list=PLD80i8An1OEGECFPgY-HPCNjXgGu-qGO6\&feature=shared).
# Overview
Source: https://docs.coreweave.com/products/wandb/automations/create-automations
Create and manage W&B automations to streamline your ML workflows
This page shows how to create and manage [automations](/products/wandb/automations), which start in response to specific events in Weights & Biases. When a specific event occurs within an organization, team, project, registry, or collection, the automation notifies a Slack channel, runs a webhook, or sends a prompt to ARIA. Use this overview to understand the prerequisites and high-level workflow, then follow the detailed instructions in [Create a Slack automation](/products/wandb/automations/create-automations/slack), [Create a webhook automation](/products/wandb/automations/create-automations/webhook), or [Create an ARIA automation](/products/wandb/automations/create-automations/aria).
> [!NOTE]
>
> Looking for companion tutorials for automations?
>
> * [Learn to automatically trigger a GitHub Action for model evaluation and deployment](https://forge.coreweave.com/wandb/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw).
> * [Watch a video demonstrating automatically deploying a model to a SageMaker endpoint](https://www.youtube.com/watch?v=s5CMj_w3DaQ).
> * [Watch a video series introducing automations](https://youtube.com/playlist?list=PLD80i8An1OEGECFPgY-HPCNjXgGu-qGO6\&feature=shared).
## Requirements
When creating automations:
* A team admin can create and manage automations for the team's projects, as well as components of their automations, such as webhooks, secrets, and Slack integrations. Refer to [Team settings](/products/wandb/platform/app/settings-page/teams).
* To create a registry automation, you must have access to the registry. Refer to [Configure Registry access](/products/registry/configure_registry#registry-roles).
* To create a Slack automation, you must have permission to post to the Slack instance and channel you select.
## Create an automation
You can create an automation from the global **Automations** hub in the left navigation, from a project or registry's **Automations** tab, or directly from a line plot panel in the workspace. When you create an automation from the hub, you choose its scope (Organization, Team, Registry, Registry collection, or project). See [Automations hub](/products/wandb/automations#automations-hub).
### Create from the Automations tab
At a high level, to create an automation from the **Automations** tab, follow these steps:
1. If necessary, [create a Forge secret](/products/wandb/platform/secrets) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets let you reuse sensitive strings safely and keep them out of automation configurations. Define secrets in your **Team Settings**. Webhook automations most commonly use secrets.
2. Configure team-level webhook or Slack integrations to authorize Weights & Biases to post to Slack or run the webhook on your behalf. Multiple automations can use a single webhook or Slack integration. Define these actions in your **Team Settings**.
3. In the project or registry, create the automation, which specifies the event to watch for and the action to take (such as posting to Slack or running a webhook). When you create a webhook automation, you configure the payload it sends.
After you complete these steps, the automation runs whenever the specified event occurs in the project or registry.
### Create a run metric automation from a workspace panel
From a line plot in the workspace, you can also create a [run metric automation](/products/wandb/automations/automation-events#run-events) for the metric it shows:
1. Hover over the panel, then click the bell icon at the top of the panel.
2. Configure the automation using the basic or advanced configuration controls. For example, apply a run filter to limit the scope of the automation, or configure an absolute threshold.
For details, refer to:
* [Create a Slack automation](/products/wandb/automations/create-automations/slack)
* [Create a webhook automation](/products/wandb/automations/create-automations/webhook)
* [Create an ARIA automation](/products/wandb/automations/create-automations/aria)
## View and manage automations
View and manage automations from a project or registry's **Automations** tab:
* To view an automation's details, click its name.
* To view an automation's execution history (which actions ran and whether they succeeded), click its name and select the **History** tab. For more information, see [View an automation's history](/products/wandb/automations/view-automation-history).
* To edit an automation, click its **action ()** menu, then click **Edit automation**.
* To delete an automation, click its **action ()** menu, then click **Delete automation**.
## Go further
For more information, see the following resources:
* [Automations tutorials](/products/wandb/automations/tutorial): Build a [project run failure alert](/products/wandb/automations/project-automation-tutorial) or a [registry alias-to-webhook automation](/products/wandb/automations/registry-automation-tutorial) in the W\&B App.
* [Automation events and scopes](/products/wandb/automations/automation-events).
* [Create a Slack automation](/products/wandb/automations/create-automations/slack).
* [Create a webhook automation](/products/wandb/automations/create-automations/webhook).
* [Create an ARIA automation](/products/wandb/automations/create-automations/aria).
* [Create a secret](/products/wandb/platform/secrets).
# Use automations with ARIA
Source: https://docs.coreweave.com/products/wandb/automations/create-automations/aria
Create a W&B automation that sends a templated prompt to ARIA, starting a new conversation when a specific event occurs.
Use a W\&B automation to send a prompt to ARIA when a specific event occurs. Each time the event occurs, the automation starts a new ARIA conversation and sends it the configured prompt.
An automation that sends a prompt to ARIA consists of two main parts:
* The [event](/products/wandb/automations/automation-events) that triggers the automation.
* The prompt that ARIA receives when the event occurs.
Use this workflow to explore experiment results, summarize or report on runs, or investigate metrics that cross a threshold.
Sending a prompt to ARIA is one action that a W\&B automation can perform. To create automations that send notifications to other destinations, refer to [Create a Slack automation](/products/wandb/automations/create-automations/slack) or [Create a webhook automation](/products/wandb/automations/create-automations/webhook).
You can also ask ARIA to create and manage W\&B automations on your behalf. For details, see [Create and manage automations with ARIA](#create-and-manage-automations-with-aria).
> [!NOTE]
>
> Available only in [W\&B Multi-tenant Cloud](/products/wandb/platform/hosting#wb-multi-tenant-cloud).
>
> ARIA works in team projects, not in your personal entity. Your organization admin must also enable **Smart features**. See [Governance and security](/products/aria/governance).
## Requirements
Before you create an automation that sends a prompt to ARIA, verify the following:
* Your organization must have ARIA enabled. If **Trigger ARIA** doesn't appear in the **Action type** list, ARIA isn't available for your organization, scope, or deployment. The list omits the option without explanation. See [Governance and security](/products/aria/governance).
* You must have write access to the project, registry, or collection you scope the automation to.
* To scope a W\&B automation to a registry and send prompts to ARIA, you must belong to a team in the same organization. Each ARIA conversation is associated with a team, so you can't create a registry-scoped automation if you don't belong to a team in that organization.
Each time an ARIA automation runs, it starts an ARIA conversation that counts toward your organization's usage, the same as a conversation you start yourself.
## Create an automation
Select **Registry** or **Project** based on the scope you want the automation to apply to. Then follow these steps to create an automation that sends a prompt to ARIA.
> [!NOTE]
>
> To apply an automation more broadly, create it from the global **Automations** hub and select the **Team** or **Organization** scope. The steps are the same as those that follow. At these scopes, Weights & Biases supports artifact and collection events only. See [Automations hub](/products/wandb/automations#automations-hub).
**Registry**
A Registry admin can create automations in that registry.
1. Navigate to Registry.
2. Select a registry to view its details.
3. To create an automation scoped to the registry, click the **Automations** tab, then click **Create automation**. An automation that is scoped to a registry is automatically applied to all of its collections (including those created in the future).
4. Choose the [event](/products/wandb/automations/automation-events#registry-events) to watch for.
Fill in any additional fields that appear, which depend upon the event. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.
Click **Next step**.
5. Select the team that hosts the ARIA conversation. Registries belong to your organization, not a team, but ARIA conversations must belong to one. See [View conversations created by automations](#view-conversations-created-by-automations).
6. Set **Action type** to **Trigger ARIA**.
7. In the **Prompt** field, write the prompt to send to ARIA. See [Write the prompt](#write-the-prompt). Click **Next step**.
8. Provide a name for the automation. Optionally, provide a description.
9. Click **Create automation**.
The automation is now active and starts a new ARIA conversation whenever the chosen event occurs in the registry.
**Project**
An administrator can create automations in a project.
1. Navigate to Weights & Biases.
2. Go to the project page and click the **Automations** tab, then click **Create automation**.
3. Choose the [event](/products/wandb/automations/automation-events#project) to watch for.
1. Fill in any additional fields that appear. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.
1. For automations triggered by a run, optionally specify one or more run filters:
* **Filter to one user's runs**: Include only runs created by the specified user. Click the toggle to turn on the filter, then specify a username.
* **Filter on run name**: Include only runs whose names match the given regular expression. Click the toggle to turn on the filter, then specify a regular expression.
2. Click **Next step**.
4. Select the team that owns the project.
5. Set **Action type** to **Trigger ARIA**.
6. In the **Prompt** field, write the prompt to send to ARIA. See [Write the prompt](#write-the-prompt). Click **Next step**.
7. Provide a name for the automation. Optionally, provide a description.
8. Click **Create automation**.
The automation is now active and starts a new ARIA conversation whenever the chosen event occurs in the project.
## Create and manage automations with ARIA
ARIA can create, delete, enable, and disable automations on demand.
Ask ARIA to create an automation:
Create a new automation: When a run succeeds, trigger ARIA to write a report for your team that summarizes the run, metrics, and outcome.
Copy user prompt
I can create an enabled project-wide automation named **Summarize successful run**. It will fire whenever any run reaches `FINISHED` and ask ARIA to write a report to summarize the run's configuration, key metrics, outcome, and notable observations.
Ask ARIA about your existing automations:
What automations exist in this project?
Copy user prompt
You have 2 project-level automations, both enabled....
Ask ARIA to rename an automation:
Rename old-name to production-alert.
Copy user prompt
Proceed with the rename?
After you confirm, ARIA renames the automation.
### View your ARIA automations in chat
Open ARIA, click **Reveal chat history** in the chat header, then open the **Customize** section at the bottom of the chat history sidebar and select **My automations**.
**My automations** replaces the conversation view in ARIA's main panel rather than opening beside it. Selecting a conversation or starting a new chat returns you to the conversation. The pane isn't persisted across reloads, so refreshing the page also returns you to your conversation.
The pane lists only the automations you created that send prompts to ARIA. It omits automations that use other actions, and your teammates' ARIA automations.
The pane is read-only. You can see whether an automation is enabled or disabled, but to enable, disable, edit, or delete one, click **View all automations** to open the full Automations page in a new tab. See [View and manage automations](#view-and-manage-automations).
Search by name, or page through results 20 at a time with **Previous** and **Next**. The list shows each automation's name, **Event type**, and **Scope**.
Click an automation to open its detail view, which shows its **Description**, **Scope**, **Created by**, **Created**, **Status** (**Enabled** or **Disabled**), **Event type**, and the **Prompt ARIA** it sends.
The **Automation history** section lists each execution's **Triggered** time and **Status**, plus a **Conversation** column. Click **Open thread** to open the conversation an execution created. A **No thread** value means the execution didn't create one. Executions are shown for up to 6 months.
Use **Back to chat** or **Back to My automations** to return from the list or a detail view.
Click **Create automation** to start one. This doesn't open a form. Instead, it starts a new ARIA chat with the following prompt already sent on your behalf:
Guide me through creating an automation that prompts ARIA. Ask what I want to automate, help me choose its scope and trigger, and draft the prompt with me.
Copy user prompt
Happy to help. What event should trigger this, and should the automation live in a project or a registry? Once we settle that, I'll help you word the prompt.
**Create automation** is disabled if you have read-only billing access. **My automations** isn't available for personal accounts. Select an organization to view your automations. The pane requires no separate feature flag. It's available under the same access checks described in [Requirements](#requirements).
## Write the prompt
The prompt is the only thing you configure for an ARIA action. Write it the way you would write a message in the ARIA chat window, and include [template variables](#template-variables) to specify which run, artifact, or metric ARIA should examine.
Keep the following constraints in mind:
* A prompt is required and can be at most 4,000 characters. Weights & Biases checks the length again after it substitutes template variables, so a prompt that saves successfully can still fail when the automation runs if the substituted values are long. A prompt that fails this check is recorded as a failed execution in the automation's [execution history](/products/wandb/automations/view-automation-history).
* A template variable that doesn't apply to the event resolves to an empty string rather than to its own name. Use only the variables listed for your event.
* ARIA starts a new conversation each time the automation runs. It doesn't continue a previous conversation and doesn't remember earlier executions of the same automation.
To insert a variable, click the **** next to it in the list below the **Prompt** field. Weights & Biases appends the variable to the end of the prompt. The list shows only the variables that apply to the event you selected.
For example, this prompt investigates each newly linked model version:
```text theme={"system"}
A new version of ${artifact_collection_name} was just linked to ${project_name} by ${event_author}.
Compare ${artifact_version_string} against the current production alias. Summarize any regression in the evaluation metrics and recommend whether to promote it.
```
You can ask ARIA to help you word a prompt before you save the automation:
I want an automation that runs whenever a run fails in this project. Draft a prompt that asks you to diagnose the failure.
Copy user prompt
Here's a prompt to paste into the automation: 'Run $\{run_name\} in $\{project\_name} just failed. Look at its logs, config, and system metrics, compare it to the last few successful runs, and tell me the most likely cause plus one concrete fix to try.'
## Template variables
Template variables use the syntax `${variable_name}`. The variables available to a prompt depend on the event the automation watches for.
The following variables are available for every event:
| Variable | Description |
| - | - |
| `${automation_name}` | The name of the automation that triggered the action. |
| `${event_type}` | The type of event that triggered the action. |
| `${event_author}` | The user that triggered the action. |
| `${entity_name}` | The name of the entity owning the event that triggered the action. |
| `${project_name}` | The name of the project owning the event that triggered the action. |
The following variables are available in addition to the preceding ones, depending on the event:
| Event | Additional variables |
| - | - |
| An artifact version is created, linked, or unlinked | `${artifact_collection_name}`, `${artifact_version}`, `${artifact_version_string}`, `${artifact_version_index}`, `${artifact_metadata.KEY}` |
| An artifact alias is added or updated | The preceding artifact variables, plus `${alias}` |
| An artifact tag is added or removed | The preceding artifact variables, plus `${tag}` |
| A collection tag is added or removed | `${artifact_collection_name}`, `${tag}` |
| A run metric crosses a threshold, changes, or changes by z-score | `${run_name}`, `${metric_name}`, `${metric_values}`, `${metric_filter}`, `${metric_filter_short}`, `${metric_current_status}` |
| A run changes status | `${run_name}`, `${run_status}` |
| A Weave metric crosses a threshold | `${alert_name}`, `${metric_name}`, `${metric_value}`, `${threshold}`, `${comparison_op}`, `${message}` |
The following notes apply to individual variables:
* `${artifact_version}` resolves to an artifact instance, such as `wandb-artifact://_id/QXJ0aWZhY3Q6NTE3ODg5ODg3`. To refer to the version by name, use `${artifact_version_string}`.
* `${artifact_metadata.KEY}` reads a custom metadata value from the artifact version that triggered the event. Replace `KEY` with the metadata key you want. Only top-level metadata keys are supported.
For the events available at each scope, see [Automation events and scopes](/products/wandb/automations/automation-events).
## View conversations created by automations
The automation creates the ARIA conversation under the account of the user who created the automation. The conversation appears in that person's ARIA chat history sidebar, alongside conversations they started themselves. To identify the user that caused the event, include `${event_author}` in the prompt.
Where the conversation lives depends on the automation's scope:
* **Project-scoped and team-scoped automations**: The conversation is created in the project where the event occurred.
* **Registry-scoped automations**: A registry belongs to your organization rather than to a team, and ARIA conversations must belong to a team. The conversation is created in the team you selected when you created the automation, in a project named `wandb_agent_default_project`. Look there rather than in the registry.
`wandb_agent_default_project` is a placeholder project that Weights & Biases maintains for each team to hold conversations that aren't tied to a project you opened. You don't create it, and it's normally empty apart from these conversations.
To confirm that the automation ran, open its [execution history](/products/wandb/automations/view-automation-history). A successful ARIA execution shows the prompt that Weights & Biases sent and a response body containing the conversation's `thread_id`. To open that conversation, use the **Open thread** link in ARIA's [My automations](#view-your-aria-automations-in-chat) pane, which is built from the same `thread_id`. An automation-created conversation isn't marked differently in your chat history.
## View and manage automations
View and manage automations from the **Automations** tab in a project or registry, or from the global **Automations** hub. In the automations list, the action displays as **ARIA**.
Only the creator of an automation that sends prompts to ARIA can edit it because the ARIA conversation runs as that user. Team members with write access to the project or registry can view and delete the automation, but the **Edit** automation option is unavailable to them.
You can't transfer ownership of the automation. Instead, delete it and have the new owner create it again. Use the same process to recover an automation whose creator has left the team.
## Limitations
ARIA automations have the following limitations:
* ARIA automations are available only on [Forge](/products/wandb/platform/hosting/hosting-options/multi_tenant_cloud), in team projects, and only when your organization admin has enabled **Smart features**.
* Each automation can start at most three ARIA conversations per minute. Weights & Biases records executions above that limit in the automation's history with the status **Skipped**, meaning the automation matched the event but didn't run its action.
* If the person who created the automation loses write access to the project or their account is deactivated, every subsequent execution fails. The failure appears in the automation's execution history.
* Only the automation's creator can edit it.
* The `wandb` Python library doesn't support ARIA automations. See [Manage automations with the API](/products/wandb/automations/api).
* The in-chat [My automations](#view-your-aria-automations-in-chat) pane is read-only. Enable, disable, edit, or delete an automation from the **Automations** tab or hub instead.
## Next steps
Continue with the following resources:
* [Automation events and scopes](/products/wandb/automations/automation-events)
* [View an automation's history](/products/wandb/automations/view-automation-history)
* [Chat with ARIA](/products/aria/chat)
* [ARIA overview](/products/aria)
# Create a Slack automation
Source: https://docs.coreweave.com/products/wandb/automations/create-automations/slack
Set up a Slack integration and create a W&B Automation that sends notifications to a Slack channel on specific events.
This page shows how to create a Slack [automation](/products/wandb/automations) so that your team receives notifications in a Slack channel when specific events occur in Weights & Biases. Example events include a new model version or a run reaching a metric threshold. To create a webhook automation instead, refer to [Create a webhook automation](/products/wandb/automations/create-automations/webhook).
At a high level, to create a Slack automation, you take these steps:
1. [Add a Slack integration](#add-a-slack-integration), which authorizes Weights & Biases to post to the Slack instance and channel.
2. [Create the automation](#create-an-automation), which defines the [event](/products/wandb/automations/automation-events) to watch for and the channel to notify.
## Add a Slack integration
A team admin can add a Slack integration to the team.
1. Log in to CoreWeave Forge and go to **Team Settings**.
2. In the **Slack channel integrations** section, click **Connect Slack** to add a new Slack instance. To add a channel for an existing Slack instance, click **New integration**.
3. If necessary, sign in to Slack in your browser. When prompted, grant Weights & Biases permission to post to the Slack channel you select. Read the page, then click **Search for a channel** and begin typing the channel name. Select the channel from the list, then click **Allow**.
4. In Slack, go to the channel you selected. If you see a post like `[YOUR-SLACK-HANDLE] added an integration to this channel: Weights & Biases`, where `[YOUR-SLACK-HANDLE]` is your Slack username, the integration is configured correctly.
Now you can [create an automation](#create-an-automation) that notifies the Slack channel you configured.
## View and manage Slack integrations
A team admin can view and manage the team's Slack instances and channels.
1. Log in to CoreWeave Forge and go to **Team Settings**.
2. View each Slack destination in the **Slack channel integrations** section.
3. Delete a destination by clicking its trash icon.
## Create an automation
After you [add a Slack integration](#add-a-slack-integration), you can create an automation that uses it to send notifications. Select **Registry** or **Project** based on the scope you want the automation to apply to. Then follow these steps to create an automation that notifies the Slack channel.
> [!NOTE]
>
> To apply an automation more broadly, create it from the global **Automations** hub and select the **Team** or **Organization** scope. The steps are the same as those that follow. At these scopes, the W\&B App supports artifact and collection events only. See [Automations hub](/products/wandb/automations#automations-hub).
**Registry**
A Registry admin can create automations in that registry.
1. Navigate to Registry.
2. Click the name of a registry to view its details.
3. To create an automation scoped to the registry, click the **Automations** tab, then click **Create automation**. An automation that is scoped to a registry is automatically applied to all of its collections (including those created in the future).
4. Choose the [event](/products/wandb/automations/automation-events#registry-events) to watch for.
Fill in any additional fields that appear, which depend upon the event. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.
Click **Next step**.
5. Select the team that owns the [Slack integration](#add-a-slack-integration).
6. Set **Action type** to **Slack notification**. Select the Slack channel, then click **Next step**.
7. Provide a name for the automation. Optionally, provide a description.
8. Click **Create automation**.
The automation is now active and notifies the selected Slack channel whenever the chosen event occurs in the registry.
**Project**
A Weights & Biases admin can create automations in a project.
1. Log in to CoreWeave Forge.
2. Go to the project page and click the **Automations** tab, then click **Create automation**.
Or, directly from a line plot in the workspace, you can create a [run metric automation](/products/wandb/automations/automation-events#run-events) for the metric it shows. Hover over the panel, then click the bell icon at the top of the panel.
3. Choose the [event](/products/wandb/automations/automation-events#project) to watch for.
1. Fill in any additional fields that appear. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.
1. For automations triggered by a run, optionally specify one or more run filters:
* **Filter to one user's runs**: Include only runs created by the specified user. Click the toggle to turn on the filter, then specify a username.
* **Filter on run name**: Include only runs whose names match the given regular expression. Click the toggle to turn on the filter, then specify a regular expression.
2. Click **Next step**.
4. Select the team that owns the [Slack integration](#add-a-slack-integration).
5. Set **Action type** to **Slack notification**. Select the Slack channel, then click **Next step**.
6. Provide a name for the automation. Optionally, provide a description.
7. Click **Create automation**.
The automation is now active and notifies the selected Slack channel whenever the chosen event occurs in the project.
## View and manage automations
**Registry**
Manage the registry's automations from the registry's **Automations** tab:
* To view an automation's details, click its name.
* To edit an automation, click its **action ()** menu, then click **Edit automation**.
* To delete an automation, click its **action ()** menu, then click **Delete automation**. Confirmation is required.
**Project**
An administrator can view and manage a project's automations from the project's **Automations** tab.
* To view an automation's details, click its name.
* To edit an automation, click its **action ()** menu, then click **Edit automation**.
* To delete an automation, click its **action ()** menu, then click **Delete automation**. Confirmation is required.
# Create a webhook automation
Source: https://docs.coreweave.com/products/wandb/automations/create-automations/webhook
Create a webhook automation in Weights & Biases to send HTTP requests to external services when specific events occur.
This page shows how to create a webhook [automation](/products/wandb/automations), which sends an HTTP request to an external service when a specific event occurs in Weights & Biases. Use a webhook automation to integrate Weights & Biases with external systems such as CI/CD pipelines, notification services, or custom tooling. To create a Slack automation, refer to [Create a Slack automation](/products/wandb/automations/create-automations/slack) instead.
At a high level, to create a webhook automation, you take these steps:
1. If necessary, [create a Weights & Biases secret](/products/wandb/platform/secrets) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your **Team Settings**.
2. [Create a webhook](#create-a-webhook) to define the endpoint and authorization details and grant the integration access to any secrets it needs.
3. [Create the automation](#create-an-automation) to define the [event](/products/wandb/automations/automation-events) to watch for and the payload Weights & Biases sends. Grant the automation access to any secrets it needs for the payload.
## Create a webhook
A team admin can add a webhook for the team. The webhook defines the endpoint Weights & Biases sends requests to and any credentials required to authenticate with it.
> [!NOTE]
>
> If the webhook requires a Bearer token or its payload requires a sensitive string, [create a secret that contains it](/products/wandb/platform/secrets#add-a-secret) before creating the webhook. You can configure at most one access token and one other secret for a webhook. The webhook's service determines your webhook's authentication and authorization requirements.
1. Log in to CoreWeave Forge and go to the **Team Settings** page.
2. In the **Webhooks** section, click **New webhook**.
3. Provide a name for the webhook.
4. Provide the endpoint URL of the webhook.
5. If the webhook requires a Bearer token, set **Access token** to the [secret](/products/wandb/platform/secrets) that contains it. When using the webhook automation, Weights & Biases sets the `Authorization: Bearer` HTTP header to the access token, and you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). For more information about the structure of the `POST` request Weights & Biases sends to the webhook service, see [Troubleshoot your webhook](#troubleshoot-your-webhook).
6. If the webhook requires a password or other sensitive string in its payload, set **Secret** to the secret that contains it. When you configure the automation that uses the webhook, you can access the secret as a [payload variable](#payload-variables) by prefixing its name with `$`.
If the webhook's access token is stored in a secret, you must *also* complete the next step to specify the secret as the access token.
7. To verify that Weights & Biases can connect and authenticate to the endpoint:
1. Optionally, provide a payload to test. To refer to a secret the webhook has access to in the payload, prefix its name with `$`. Weights & Biases uses this payload only for testing and doesn't save it. You configure an automation's payload when you [create the automation](#create-an-automation). For more information about where the secret and access token appear in the `POST` request, see [Troubleshoot your webhook](#troubleshoot-your-webhook).
2. Click **Test**. Weights & Biases attempts to connect to the webhook's endpoint using the credentials you configured. If you provided a payload, Weights & Biases sends it.
If the test doesn't succeed, verify the webhook's configuration and try again. If necessary, refer to [Troubleshoot your webhook](#troubleshoot-your-webhook).
Now you can [create an automation](#create-an-automation) that uses the webhook.
## Create an automation
After you [configure a webhook](#create-a-webhook), create an automation that defines which Weights & Biases event triggers the webhook and what payload to send. Select **Registry** or **Project** depending on the scope you want, then follow these steps to create an automation that triggers the webhook.
> [!NOTE]
>
> To apply an automation more broadly, create it from the global **Automations** hub and select the **Team** or **Organization** scope. The steps are the same as those that follow. At these scopes, the W\&B App supports artifact and collection events only. See [Automations hub](/products/wandb/automations#automations-hub).
**Registry**
A Registry admin can create automations in that registry. Registry automations apply to all collections in the registry, including those added in the future.
1. Navigate to Registry.
2. Click the name of a registry to view its details.
3. To create an automation scoped to the registry, click the **Automations** tab, then click **Create automation**.
4. Choose the [event](/products/wandb/automations/automation-events#registry-events) to watch for.
Fill in any additional fields that appear. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.
Click **Next step**.
5. Select the team that owns the [webhook](#create-a-webhook).
6. Set **Action type** to **Webhooks**. Then select the [webhook](#create-a-webhook) to use.
7. If you configured an access token for the webhook, you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). If you configured a secret for the webhook, you can access it in the payload by prefixing its name with `$`. The webhook's service determines your webhook's requirements.
8. Click **Next step**.
9. Provide a name for the automation. Optionally, provide a description. Click **Create automation**.
**Project**
An administrator can create automations in a project.
1. Navigate to Weights & Biases and select a project.
2. In the project sidebar, click **Automations**, then click **Create automation**.
Or, from a line plot in the workspace, you can quickly create a [run metric automation](/products/wandb/automations/automation-events#run-events) for the metric it shows. Hover over the panel, then click the bell icon at the top of the panel.
3. Choose the [event](/products/wandb/automations/automation-events#project) to watch for, such as when an artifact alias is added or when a run metric meets a given threshold.
1. Fill in any additional fields that appear, which depend upon the event. For example, if you select **An artifact alias is added**, you must specify the **Alias regex**.
1. For automations triggered by a run, optionally specify one or more run filters.
* **Filter to one user's runs**: Include only runs created by the specified user. Click the toggle to turn on the filter, then specify a username.
* **Filter on run name**: Include only runs whose names match the given regular expression. Click the toggle to turn on the filter, then specify a regular expression.
The automation applies to all collections in the project, including those added in the future.
2. Click **Next step**.
4. Select the team that owns the [webhook](#create-a-webhook).
5. Set **Action type** to **Webhooks**. Then select the [webhook](#create-a-webhook) to use.
6. If your webhook requires a payload, construct it and paste it into the **Payload** field. If you configured an access token for the webhook, you can access the token in the `${ACCESS_TOKEN}` [payload variable](#payload-variables). If you configured a secret for the webhook, you can access it in the payload by prefixing its name with `$`. The webhook's service determines your webhook's requirements.
7. Click **Next step**.
8. Provide a name for the automation. Optionally, provide a description. Click **Create automation**.
After you complete these steps, the automation is active and runs the webhook whenever the specified event occurs in the registry or project.
## View and manage automations
Review, edit, or delete an automation from the **Automations** tab.
**Registry**
Manage a registry's automations from the registry's **Automations** tab.
* To view an automation's details, click its name.
* To edit an automation, click its **action ()** menu, then click **Edit automation**.
* To delete an automation, click its **action ()** menu, then click **Delete automation**. Weights & Biases prompts you to confirm.
**Project**
An administrator can view and manage a project's automations from the project's **Automations** tab.
* To view an automation's details, click its name.
* To edit an automation, click its **action ()** menu, then click **Edit automation**.
* To delete an automation, click its **action ()** menu, then click **Delete automation**. Weights & Biases prompts you to confirm.
## Payload reference
Use these sections to construct your webhook's payload. The following sections describe the variables available to your payload and show example payloads for common services. For details about testing your webhook and its payload, refer to [Troubleshoot your webhook](#troubleshoot-your-webhook).
### Payload variables
The following table describes the variables you can use to construct your webhook's payload.
| Variable | Details |
| - | - |
| `${project_name}` | The name of the project that owns the mutation that triggered the action. |
| `${entity_name}` | The name of the entity or team that owns the mutation that triggered the action. |
| `${event_type}` | The type of event that triggered the action. |
| `${event_author}` | The user that triggered the action. |
| `${alias}` | Contains an artifact's alias if the automation is triggered by the **An artifact alias is added** event. For other automations, this variable is blank. |
| `${tag}` | Contains an artifact's tags if the automation is triggered by the **An artifact tag is added** event. For other automations, this variable is blank. |
| `${artifact_collection_name}` | The name of the artifact collection that the artifact version is linked to. |
| `${artifact_metadata.}` | The value of an arbitrary top-level metadata key from the artifact version that triggered the action. Replace `` with the name of a top-level metadata key. Only top-level metadata keys are available in the webhook's payload. |
| `${artifact_version}` | The [`Wandb.Artifact`](/products/wandb/ref/python/experiments/artifact) representation of the artifact version that triggered the action. |
| `${artifact_version_string}` | The `string` representation of the artifact version that triggered the action. |
| `${ACCESS_TOKEN}` | The value of the access token configured in the [webhook](#create-a-webhook), if you configured an access token. Weights & Biases automatically passes it in the `Authorization: Bearer` HTTP header. |
| `${SECRET_NAME}` | If configured, the value of a secret configured in the [webhook](#create-a-webhook). Replace `SECRET_NAME` with the name of the secret. |
### Payload examples
The following examples show webhook payloads for common use cases. The examples demonstrate how to use [payload variables](#payload-variables).
**GitHub repository dispatch**
> [!NOTE]
>
> Verify that your access tokens have the required set of permissions to trigger your GitHub Actions workflow. For more information, see the [GitHub repository dispatch event documentation](https://docs.github.com/en/rest/repos/repos?#create-a-repository-dispatch-event).
Send a repository dispatch from Weights & Biases to trigger a GitHub action. For example, suppose you have a GitHub workflow file that accepts a repository dispatch as a trigger for the `on` key:
```yaml theme={"system"}
on:
repository_dispatch:
types: BUILD_AND_DEPLOY
```
The payload for the repository might look something like:
```json theme={"system"}
{
"event_type": "BUILD_AND_DEPLOY",
"client_payload":
{
"event_author": "${event_author}",
"artifact_version": "${artifact_version}",
"artifact_version_string": "${artifact_version_string}",
"artifact_collection_name": "${artifact_collection_name}",
"project_name": "${project_name}",
"entity_name": "${entity_name}"
}
}
```
> [!NOTE]
>
> The `event_type` key in the webhook payload must match the `types` field in the GitHub workflow YAML file.
The contents and positioning of rendered template strings depend on the event or model version you configured the automation for. `${event_type}` renders as either `LINK_ARTIFACT` or `ADD_ARTIFACT_ALIAS`. The following shows an example mapping:
```text theme={"system"}
${event_type} --> "LINK_ARTIFACT" or "ADD_ARTIFACT_ALIAS"
${event_author} --> ""
${artifact_version} --> "wandb-artifact://_id/QXJ0aWZhY3Q6NTE3ODg5ODg3"
${artifact_version_string} --> "/model-registry/:"
${artifact_collection_name} --> ""
${project_name} --> "model-registry"
${entity_name} --> ""
```
Use template strings to dynamically pass context from Weights & Biases to GitHub Actions and other tools. If those tools can call Python scripts, they can consume the registered model artifacts through the [W\&B API](/products/wandb/artifacts/download-and-use-an-artifact).
For more information, see the following resources:
* For more information about repository dispatch, see the [official documentation on the GitHub Marketplace](https://github.com/marketplace/actions/repository-dispatch).
* Watch the videos [Webhook Automations for Model Evaluation](https://www.youtube.com/watch?v=7j-Mtbo-E74\&ab_channel=Weights%26Biases) and [Webhook Automations for Model Deployment](https://www.youtube.com/watch?v=g5UiAFjM2nA\&ab_channel=Weights%26Biases), which guide you to create automations for model evaluation and deployment.
* Review the W\&B report [Model CI/CD with Weights & Biases](https://forge.coreweave.com/wandb/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw), which illustrates how to use a GitHub Actions webhook automation for model CI.
* For an example of model CI with a Modal Labs webhook, see the [wandb-modal-webhook GitHub repository](https://github.com/hamelsmu/wandb-modal-webhook).
**Microsoft Teams notification**
This example payload shows how to notify your Teams channel using a webhook:
```json theme={"system"}
{
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"summary": "New Notification",
"sections": [
{
"activityTitle": "Notification from WANDB",
"text": "This is an example message sent via Teams webhook.",
"facts": [
{
"name": "Author",
"value": "${event_author}"
},
{
"name": "Event Type",
"value": "${event_type}"
}
],
"markdown": true
}
]
}
```
You can use template strings to inject Weights & Biases data into your payload at the time of execution. See the [Teams example](#microsoft-teams-notification).
**Slack notifications**
> [!NOTE]
>
> This section is provided for historical purposes. If you use a webhook to integrate with Slack, Weights & Biases recommends that you update your configuration to use the [Slack integration](/products/wandb/automations/create-automations/slack) instead.
Set up your Slack app and add an incoming webhook integration by following the instructions in the [Slack API documentation](https://api.slack.com/messaging/webhooks). Ensure that you have the secret specified under `Bot User OAuth Token` as your Weights & Biases webhook's access token.
The following is an example payload:
```json theme={"system"}
{
"text": "New alert from WANDB!",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Registry event: ${event_type}"
}
},
{
"type":"section",
"text": {
"type": "mrkdwn",
"text": "New version: ${artifact_version_string}"
}
},
{
"type": "divider"
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Author: ${event_author}"
}
}
]
}
```
## Troubleshoot your webhook
If a webhook isn't working as expected, you can troubleshoot it interactively with the W\&B App UI or programmatically with a shell script. You can troubleshoot a webhook during creation or afterward.
For details about the format Weights & Biases uses for the `POST` request, refer to the **Shell script** tab.
**W\&B App UI**
A team admin can test a webhook interactively with the W\&B App UI.
1. Go to your team page, then click **Settings**.
2. Scroll to the **Webhooks** section.
3. Click the **action ()** menu next to the name of your webhook.
4. Select **Test**.
5. From the UI panel that appears, paste your `POST` request into the field that appears.
6. Click **Test webhook**. Within the W\&B App UI, Weights & Biases posts the response from your endpoint.
Watch the video [Testing Webhooks in Weights & Biases](https://www.youtube.com/watch?v=bl44fDpMGJw\&ab_channel=Weights%26Biases) for a demonstration.
**Shell script**
This shell script shows one method to generate a `POST` request similar to the request Weights & Biases sends to your webhook automation when it's triggered.
Copy and paste the following code into a shell script to troubleshoot your webhook. Specify your own values for:
* `ACCESS_TOKEN`
* `SECRET`
* `PAYLOAD`
* `API_ENDPOINT`
```bash webhook_test.sh theme={"system"}
#!/bin/bash
# Your access token and secret
ACCESS_TOKEN="your_api_key"
SECRET="your_api_secret"
# URL of your webhook endpoint
API_ENDPOINT="https://your.webhook.endpoint/path"
# The data you want to send (for example, in JSON format)
PAYLOAD='{"key1": "value1", "key2": "value2"}'
# Generate the HMAC signature. For security, W&B includes the
# X-Wandb-Signature header computed from the payload and the shared secret
# associated with the webhook using HMAC with SHA-256.
SIGNATURE=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $2}')
# Make the cURL request
curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ACCESS_TOKEN" \
-H "X-Wandb-Signature: $SIGNATURE" \
-d "$PAYLOAD" "$API_ENDPOINT"
```
# Tutorial: Project run-failure alert automation
Source: https://docs.coreweave.com/products/wandb/automations/project-automation-tutorial
Build a run-failure alert that sends a Slack notification when a run in your project fails.
This tutorial walks you through building a **project** automation triggered by run status: when a run in your project transitions to **Failed**, Weights & Biases sends a Slack notification. This automation lets your team learn about failed runs in real time, so you can investigate and remediate them instead of discovering failures later.
```mermaid theme={"system"}
%%{init: {'flowchart': {'rankSpacing': 200;}}}%%
flowchart LR
Event[Run state change to Failed]
Action[Slack notification]
Event --> Action
```
> [!TIP]
>
> For guidance creating a Registry automation, see [Tutorial: Registry artifact alias automation](/products/wandb/automations/registry-automation-tutorial).
## Prerequisites
* A Weights & Biases project.
* A [Slack integration](/products/wandb/automations/create-automations/slack#add-a-slack-integration) configured in **Team Settings**.
## Create a project automation
Set up a project-scoped automation so that when a run in the project transitions to **Failed**, Weights & Biases sends a Slack notification.
1. Open the project and click the **Automations** tab in the sidebar, then click **Create automation**.
2. Choose the event **Run state change**. Set the state to **Failed**. Optionally add a run name or user filter to limit which runs trigger the automation.
3. Click **Next step**. Set **Action type** to **Slack notification** and select the Slack channel.
4. Click **Next step**. Give the automation a name (for example, "Run failure alert") and an optional description, then click **Create automation**.
Your project now has an active automation that posts to the Slack channel you selected when a run fails. For more detail, see [Create a Slack automation](/products/wandb/automations/create-automations/slack#create-an-automation) (Project tab).
## Test the automation
To confirm the automation is configured correctly, trigger it with a deliberately failed run. Create a run and log it to the project, explicitly marking it as failed:
```python theme={"system"}
import wandb
with wandb.init(project="my-project") as run:
run.log({"loss": 1.23})
run.finish(exit_code=1)
```
Shortly after, you see a Slack message with the run link and status.
## Go further
To learn more about W\&B automations beyond what this tutorial covers, see the following resources:
* [Automation events and scopes](/products/wandb/automations/automation-events) for all project and registry event types.
* [Create a Slack automation](/products/wandb/automations/create-automations/slack) and [Create a webhook automation](/products/wandb/automations/create-automations/webhook) for full UI and payload details.
# Tutorial: Registry artifact alias automation
Source: https://docs.coreweave.com/products/wandb/automations/registry-automation-tutorial
Build an automation that runs a webhook when a Registry artifact gets a specific alias like "production".
This tutorial walks you through building a **registry** automation triggered by artifact metadata: when an artifact in your registry gets a specific alias (for example, **production**), Weights & Biases sends a `POST` request to your webhook. Use this pattern to notify downstream systems like deployment pipelines, paging services, or notification channels, whenever you promote a model to a known stage. This tutorial is intended for ML engineers and MLOps practitioners who manage model lifecycles in Registry.
```mermaid theme={"system"}
flowchart LR
Event[Artifact alias added]
Action[Webhook]
Event --> Action
```
> [!TIP]
>
> For guidance creating a project automation, see [Tutorial: Project run-failure alert automation](/products/wandb/automations/project-automation-tutorial).
## Prerequisites
* A [webhook](/products/wandb/automations/create-automations/webhook#create-a-webhook) configured in **Team Settings**.
* A [Registry](/products/registry/create_registry) with at least one collection, or reuse an existing registry.
## Create a registry automation
Set up a registry-scoped automation so that when an artifact in any collection in the registry gets a specific alias (for example, **production**), Weights & Biases sends a `POST` request to your webhook.
1. Open the registry and click the **Automations** tab, then click **Create automation**.
2. Choose the event **An artifact alias is added**. Enter an **Alias regex** that matches the alias you care about (for example, **production** or **staging**).
3. Click **Next step**. Set **Action type** to **Webhooks** and select your webhook. If the webhook expects a payload, paste a JSON body and use [payload variables](/products/wandb/automations/create-automations/webhook#payload-variables) such as `${artifact_collection_name}` and `${artifact_version_string}`.
4. Click **Next step**. Give the automation a name and optional description, then click **Create automation**.
For more information, see [Create a webhook automation](/products/wandb/automations/create-automations/webhook#create-an-automation) (Registry tab).
## Test the automation
To confirm that the automation fires end-to-end, trigger the configured event by adding the alias to an artifact version. Add the alias (for example, **production**) to an artifact version in the registry, using the W\&B App or the public API. For example:
```python theme={"system"}
import wandb
with wandb.init(project="my-project") as run:
artifact = wandb.Artifact("my-model", type="model")
# ... log files or metadata to artifact as needed ...
run.log_artifact(artifact)
run.wait() # Ensure the artifact is logged before proceeding
# Add an alias to the latest version in the collection
api = wandb.Api()
collection = api.artifact_collection(name="my-model", type_name="model")
version = next(collection.versions()) # Get the latest version
version.aliases.append("production")
version.save()
print("Added alias 'production' to", version.name)
```
Within a short time, your webhook endpoint receives a `POST` with the payload you configured. You now have a working registry automation that fires whenever a matching alias is applied to an artifact in this registry.
## Go further
To learn more about W\&B automations beyond what this tutorial covers, see the following resources:
* [Automation events and scopes](/products/wandb/automations/automation-events) for all project and registry event types.
* [Create a Slack automation](/products/wandb/automations/create-automations/slack) and [Create a webhook automation](/products/wandb/automations/create-automations/webhook) for full UI and payload details.
# Automation tutorial overview
Source: https://docs.coreweave.com/products/wandb/automations/tutorial
Learn to build a project run-failure alert or a registry alias automation.
W\&B Automations follow this pattern: when an **event** occurs and optional **conditions** are met, an **action** runs automatically. For example:
* When a run fails (event), notify a Slack channel (action).
* When the `production` alias is added to an artifact (event), call a webhook to trigger deployment (action).
* When a new model version is linked (event), send a prompt to ARIA to compare it against production (action).
Events and available conditions differ for automations scoped to a [project](/products/wandb/automations/automation-events#project) or a [registry](/products/wandb/automations/automation-events#registry). See [Automation events and scopes](/products/wandb/automations/automation-events).
This page links to two end-to-end tutorials that show you how to create an automation in Weights & Biases. Each tutorial walks through a different scenario, so you can follow the one that matches what you want to build. For detailed guidance, select a tutorial.
* **[Project automation tutorial](/products/wandb/automations/project-automation-tutorial)**: Alert when a run fails (Slack notification).
* **[Registry automation tutorial](/products/wandb/automations/registry-automation-tutorial)**: Trigger a webhook when an alias (for example, `production`) is added to an artifact.
# Overview
Source: https://docs.coreweave.com/products/wandb/evaltables
Learn how to create, compare, and visualize Eval Tables in Weights & Biases.
Eval Tables help you compare aggregate scores and example-level model inputs, outputs, and scores across multiple runs. Use an Eval Table to compare model versions or training steps, review aggregate score changes, and investigate the examples behind changes in model performance.
The following image shows an Eval Table called `"validation_prediction_eval"` that compares two runs `"summer-butterfly-9"` and `"gentle-flower-8"`:
An **Eval Table** panel contains three sections:
1. Run comparison selector: Select the [runs that you want to compare](/products/wandb/evaltables/compare-runs).
2. Aggregate scores: Review aggregate scores for the selected runs and compare the differences between them. For more information, see [View aggregate scores](/products/wandb/evaltables/compare-runs#aggregate-scores).
3. Examples: Compare the inputs, outputs, and scores for each example across the selected runs.
The following image highlights each section of the panel:
[Create an Eval Table with the `EvalTable` class from the W\&B Python SDK](/products/wandb/evaltables/create-an-evaluation-table).
Existing [W\&B Tables](/products/wandb/tables) can also be converted to Eval Tables with ARIA. For more information, see [Convert a W\&B Table to an Eval Table](/products/wandb/evaltables/create-an-evaluation-table#convert-a-wb-table-to-an-eval-table).
> [!TIP]
>
> Convert existing W\&B Tables to Eval Tables to improve rendering performance and access additional comparison features.
To create your first Eval Table, see [Create an Eval Table](/products/wandb/evaltables/create-an-evaluation-table).
# Compare runs with Eval Tables
Source: https://docs.coreweave.com/products/wandb/evaltables/compare-runs
Use Eval Tables to compare inputs, outputs, and scores across multiple runs. Weights & Biases aligns corresponding examples and calculates aggregate scores and score differences for the selected runs.
> [!NOTE]
>
> To compare runs with workspace-level summary metrics, line plots, pinned runs, or a baseline run, see [Pin and compare runs](/products/wandb/runs/compare-runs). Eval Tables compare aligned examples and Eval Table-specific scores within a logged table.
To compare runs:
1. Navigate to your project workspace.
2. Scroll to the Eval Tables panel.
3. Select the **+** button.
4. Select the run that you want to add.
Weights & Biases groups columns that have the same role and name across the selected runs. For example, if two runs use the same `input_columns`, Weights & Biases displays those columns together in the **Inputs** section. Weights & Biases similarly groups shared `output_columns` and `score_columns` in the **Outputs** and **Scores** sections.
The following screenshot shows an Eval Table that compares the `summer-butterfly-9` and `gentle-flower-8` runs:
## Set the reference run
When you compare multiple runs, Weights & Biases defaults to designating the left-most run as the reference run. The reference run is the baseline to calculate score deltas.
To choose a different reference run, hover over the run, open its menu, and select **Set as reference**.
## View aggregate scores
Weights & Biases calculates an aggregate value for each score column in each selected run. The calculation depends on the score's data type:
| Type of score | Aggregate value |
| - | - |
| Boolean | Count and fraction of `true` values |
| Numeric | Mean |
| String or categorical | No scalar aggregate |
| Null | Excluded from calculation |
The following screenshot highlights the aggregate values for the correct and confidence scores:
## Compare score deltas
For each score column, Weights & Biases calculates the difference between the reference run and every other selected run.
Weights & Biases calculates each delta as:
```text theme={"system"}
comparison run value - reference run value
```
For example, suppose `summer-butterfly-9` is the reference run and `gentle-flower-8` is the comparison run. Weights & Biases calculates the confidence delta as follows:
| `summer-butterfly-9` | `gentle-flower-8` | delta |
| - | - | - |
| 0.43 | 0.62 | +0.19 |
| 0.92 | 0.86 | -0.06 |
| 0.97 | 0.85 | -0.12 |
A positive delta means that the comparison run has a higher value than the reference run. A negative delta means that it has a lower value.
The following screenshot highlights the score deltas:
# Create an Eval Table
Source: https://docs.coreweave.com/products/wandb/evaltables/create-an-evaluation-table
Learn how to create an Eval Table in Weights & Biases.
Create an Eval Table using the W\&B Python SDK or convert an existing [W\&B Table](/products/wandb/tables) with [ARIA](/products/aria).
## Create an Eval Table
Use the `wandb.EvalTable` class to create an Eval Table.
For example, suppose you have the following pandas DataFrame:
```python theme={"system"}
import pandas as pd
import wandb
df = pd.DataFrame(
[
{
"image_id": "img_001",
"true_label": "cat",
"predicted_label": "cat",
"confidence": 0.97,
"correct": True,
},
{
"image_id": "img_002",
"true_label": "dog",
"predicted_label": "cat",
"confidence": 0.72,
"correct": False,
},
{
"image_id": "img_003",
"true_label": "car",
"predicted_label": "car",
"confidence": 0.89,
"correct": True,
},
]
)
```
Create an Eval Table by passing the DataFrame to `wandb.EvalTable`. Use the `input_columns`, `output_columns`, and `score_columns` arguments to identify the role of each column.
The following example uses:
* `image_id` as the input
* `predicted_label` as the output
* `correct` and `confidence` as scores
```python theme={"system"}
with wandb.init(project="classifier-eval-table-demo") as run:
eval_table = wandb.EvalTable(
dataframe=df,
input_columns=["image_id"],
output_columns=["predicted_label"],
score_columns=["correct", "confidence"],
)
run.log({"validation_predictions_eval": eval_table})
```
## Convert a W\&B Table to an Eval Table
Use the conversion banner to convert an existing W\&B Table to an Eval Table:
1. Navigate to the project that contains the W\&B Table.
2. Select the **Try now** button in the conversion banner. The button opens [ARIA](/products/aria) with information about which table to convert.
3. Follow the prompts in ARIA's conversation window.
If you closed the banner or do not see it, open ARIA manually:
1. Select the blue circle in the upper-right corner of the page to open ARIA.
2. Enter `/convert-eval-table` in the ARIA conversation.
3. Select Send, represented by the upward-pointing arrow () in the lower-right corner of the conversation.
Converting a W\&B Table does not modify the original table.
# View Eval Tables
Source: https://docs.coreweave.com/products/wandb/evaltables/visualize-evaluation-tables
When you create an Eval Table, specify which columns contain inputs, outputs, and scores. Weights & Biases groups the columns into corresponding sections in the Eval Table.
The following example shows the basic structure of an EvalTable object:
```python theme={"system"}
import wandb
wandb.EvalTable(
input_columns=["column1", "column2"],
output_columns=["output_column1", "output_column2"],
score_columns=["score_column1", "score_column2"],
)
```
Pass one or more column names to each of the following arguments:
* `input_columns`: Data provided to the model, such as an image, prompt, or ground-truth label.
* `output_columns`: Values produced by the model, such as a prediction or generated response.
* `score_columns`: Metrics or other values used to evaluate the output.
Weights & Biases displays these columns in the Inputs, Outputs, and Scores sections of the Eval Table.
> [!NOTE]
>
> The examples on this page use code from [Create an Eval Table](/products/wandb/evaltables/create-an-evaluation-table). Expand **View code** to view or copy the complete example.
>
> > **View code**
> >
> > ```python theme={"system"}
> > import pandas as pd
> > import wandb
> >
> > df = pd.DataFrame(
> > [
> > {
> > "image_id": "img_001",
> > "true_label": "cat",
> > "predicted_label": "cat",
> > "confidence": 0.97,
> > "correct": True,
> > },
> > {
> > "image_id": "img_002",
> > "true_label": "dog",
> > "predicted_label": "cat",
> > "confidence": 0.72,
> > "correct": False,
> > },
> > {
> > "image_id": "img_003",
> > "true_label": "car",
> > "predicted_label": "car",
> > "confidence": 0.89,
> > "correct": True,
> > },
> > ]
> > )
> >
> > df["correct"] = df["correct"].astype(object) # preserves native Python bools
> >
> > with wandb.init(project="classifier-eval-table-demo") as run:
> > eval_table = wandb.EvalTable(
> > dataframe=df,
> > input_columns=["image_id", "true_label"],
> > output_columns=["predicted_label"],
> > score_columns=["correct", "confidence"],
> > )
> > run.log({"validation_predictions_eval": eval_table})
> > ```
For example, consider the code example from [Create an Eval Table](/products/wandb/evaltables/create-an-evaluation-table):
```python theme={"system"}
import wandb
with wandb.init(project="classifier-eval-table-demo") as run:
eval_table = wandb.EvalTable(
dataframe=df,
input_columns=["image_id", "true_label"],
output_columns=["predicted_label"],
score_columns=["correct", "confidence"],
)
run.log({"validation_predictions_eval": eval_table})
```
In that example:
* The **Inputs** section contains the `image_id` and `true_label` columns.
* The **Outputs** section contains the `predicted_label` column.
* The **Scores** section contains the `correct` and `confidence` columns.
The following image highlights these sections:
## Detail view
Select a row to open a detailed view of that example. The detail view shows the example's inputs, outputs, and scores.
The following image shows the detail view for the first row in the Eval Table:
Use the detail view to inspect examples when the table contains many columns or when you compare results across multiple runs.
Select the up or down arrow above the **Inputs** section to move between examples in the current table view.
## Filter data
Use filters to display only rows that match specific conditions.
To add a filter:
1. Select the **Filter** button above the table.
2. From the dropdown, select the column to filter.
3. Select an operator.
4. Specify a value.
Weights & Biases automatically applies the filter to the table and displays only rows that match the condition.
You can apply multiple filters. To remove a filter, select the `X` next to it.
## Show or hide columns
Use the **Columns** menu to control which columns appear in the Eval Table. The available columns depend on the data logged to the table.
To show or hide a column:
1. Select **Columns** above the table.
2. Select a column to show it, or clear the column to hide it.
The following image shows a Columns menu in which all available columns are selected:
# Reports overview
Source: https://docs.coreweave.com/products/wandb/reports
Project management and collaboration tools for machine learning projects
[Try in Colab](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/intro/Report_API_Quickstart.ipynb)
[Try in W\&B](https://forge.coreweave.com/wandb/stacey/deep-drive/reports/The-View-from-the-Driver-s-Seat--Vmlldzo1MTg5NQ?utm_source=fully_connected\&utm_medium=blog\&utm_campaign=view+from+the+drivers+seat)
Use W\&B Reports to:
* Organize Runs.
* Embed and automate visualizations.
* Describe your findings.
* Share updates with collaborators, either as a LaTeX zip file or a PDF.
The report below was created from metrics logged to Weights & Biases over the course of training. Scroll within it to explore the benchmark runs, panels, and commentary, or [open the full report](https://forge.coreweave.com/wandb/stacey/saferlife/reports/SafeLife-Benchmark-Experiments--Vmlldzo0NjE4MzM) in the Weights & Biases app.
**SafeLife benchmark experiments report**
## How it works
Create a collaborative report with a few clicks.
1. Navigate to your Weights & Biases project workspace in the W\&B App.
2. Click the **Create report** button in the upper right corner of your workspace.
3. A modal titled **Create Report** will appear. Select the charts and panels you want to add to your report. (You can add or remove charts and panels later).
4. Click **Create report**.
5. Edit the report to your desired state.
6. Click **Publish to project**.
7. Click the **Share** button to share your report with collaborators.
See the [Create a report](/products/wandb/reports/create-a-report) page for more information on how to create reports interactively and programmatically with the W\&B Python SDK.
## How to get started
Depending on your use case, explore the following resources to get started with W\&B Reports:
* Check out our [video demonstration](https://www.youtube.com/watch?v=2xeJIv_K_eI) to get an overview of W\&B Reports.
* Explore the [Reports gallery](/products/wandb/reports/reports-gallery) for examples of live reports.
* Try the [Programmatic Workspaces](https://colab.research.google.com/github/wandb/wandb-workspaces/blob/Update-wandb-workspaces-tuturial/Workspace_tutorial.ipynb) notebook to learn how to create and customize your workspace.
* Read curated Reports in [W\&B Fully Connected](https://wandb.me/fc).
## Recommended practices and tips
For best practices and tips for Experiments and logging, see [Best Practices: Reports](https://forge.coreweave.com/wandb/wandb/pytorch-lightning-e2e/reports/W-B-Best-Practices-Guide--VmlldzozNTU1ODY1#reports).
# Clone and export reports
Source: https://docs.coreweave.com/products/wandb/reports/clone-and-export-reports
Export W&B Reports as PDF or LaTeX files, and clone reports using the App UI or the Report and Workspace API.
> [!NOTE]
>
> W\&B Report and Workspace API is in Public Preview.
This page describes how to export a W\&B Report to a portable file format. It also describes how to clone an existing report to reuse its structure as a starting point for new work.
## Export a report
Export a report as a PDF or LaTeX file to share its contents outside of Weights & Biases or to archive a static version of your analysis. In your report, select the **action ()** menu. Choose **Download** and select either PDF or LaTeX output format.
## Clone a report
Clone a report to reuse an existing project's template and formatting as the basis for a new report. You can clone reports in the W\&B App or programmatically with the Report and Workspace API.
**W\&B App**
In your report:
1. Select the **action ()** menu, then choose **Make a copy**.
2. In the modal, select a destination for your cloned report.
3. Choose **Clone report**.
The destination dropdown defaults to the current project and lists projects across the teams you belong to. To copy the report into a different project or team, open the dropdown and select it.
When you clone a report, you specify the destination. If you clone it to a team, all team members can view it. If you clone it to your personal account, only you can view it by default.
**Report and Workspace API**
Use the Report and Workspace API to load an existing report from its URL and reuse it as a template for a new report.
Load a report from a URL to use it as a template. Replace `PROJECT` with the name of your Weights & Biases project.
```python theme={"system"}
report = wr.Report(
project=PROJECT, title="Quickstart Report", description="That was easy!"
) # Create
report.save() # Save
new_report = wr.Report.from_url(report.url) # Load
```
After you load the report, edit the content in `new_report.blocks` to customize the cloned report, then save it. Replace `ENTITY` with your Weights & Biases entity name.
```python theme={"system"}
pg = wr.PanelGrid(
runsets=[
wr.Runset(ENTITY, PROJECT, "First Run Set"),
wr.Runset(ENTITY, PROJECT, "Elephants Only!", query="elephant"),
],
panels=[
wr.LinePlot(x="Step", y=["val_acc"], smoothing_factor=0.8),
wr.BarPlot(metrics=["acc"]),
wr.MediaBrowser(media_keys="img", num_columns=1),
wr.RunComparer(diff_only="split", layout={"w": 24, "h": 9}),
],
)
new_report.blocks = (
report.blocks[:1] + [wr.H1("Panel Grid Example"), pg] + report.blocks[1:]
)
new_report.save()
```
After you save, your new report appears in your Weights & Biases project.
# Collaborate on reports
Source: https://docs.coreweave.com/products/wandb/reports/collaborate-on-reports
Collaborate and share W&B Reports with peers, coworkers, and your team.
This page describes ways to collaborate on W\&B Reports with your team so you can share results, gather feedback, and keep important reports accessible. You can share a report, edit it collaboratively, add comments, or star it for quick access.
## Share a report
Share a report when you want to give peers, coworkers, or teammates access to view or edit its contents. When viewing a report, click **Share**. You can share a report in one of the following ways:
* To share a link to the report with an email address or a username, click **Invite**. Enter an email address or username, select **Can view** or **Can edit**, then click **Invite**. If you share by email, the email address doesn't need to be a member of your organization or team.
* To generate a sharing link instead, click **Share**. Adjust the permissions for the link, then click **Copy report link**. Share the link with the member.
When viewing the report, click a panel to open it in full-screen mode. If you copy the URL from the browser and share it with another user, the panel opens in full-screen mode when they access the link.
## Edit a report
Reports support collaborative editing so that team members can contribute updates without overwriting each other's work. Multiple team members can edit the same report and publish changes when they're ready. When any team member clicks **Edit** to begin editing the report, Weights & Biases automatically saves a draft. Select **Save to report** to publish your changes.
If an edit conflict occurs, such as when two team members edit the report at once, a warning notification helps you resolve any conflicts.
## Comment on a report
Use comments to give feedback or ask questions about a report or one of its panels without modifying its contents. To leave a comment on a report, click **Comment**.
To comment directly on a panel, hover over the panel, then click the comment button ().
## Star a report
Star a report to mark it as a favorite and make it easier to find later. Click the open star icon () at the top of a report to add it to your favorites. To remove the star, click the filled star icon (). When viewing your team's list of reports, click the open star in a report's row to add it to your favorites. Starred reports appear at the top of the list.
From the list of reports, you can see how many members have starred each report to gauge its popularity.
# Create a report
Source: https://docs.coreweave.com/products/wandb/reports/create-a-report
Create a W&B Report with the W&B App or programmatically.
> [!NOTE]
>
> W\&B Report and Workspace API is in Public Preview.
W\&B Reports let you share findings, results, and visualizations from your runs with collaborators. Create a report from a project workspace, from the **Reports** tab, or programmatically with the Report and Workspace API. Select a tab to get started.
For a programmatic example, see the [Report API Quickstart Google Colab](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/intro/Report_API_Quickstart.ipynb).
> [!TIP]
>
> To have [ARIA](/products/aria) draft a report for you, open the project's **Reports** tab, click **Draft report with ARIA**, then describe the report you want.
**W\&B App**
1. Navigate to your project workspace in the W\&B App.
2. In the upper-right corner of your workspace, click **Create report**.
3. A modal appears. Select the charts you want to start with. You can add or remove charts later from the report interface.
4. To prevent Weights & Biases from adding more runs to your report, select **Filter run sets**. Weights & Biases saves a draft report automatically. Access it in the **Reports** tab.
**Report tab**
1. Navigate to your project workspace in the W\&B App.
2. Select the **Reports** tab in your project.
3. Click **Create report**.
**Report and Workspace API**
Create a report programmatically:
1. Install the W\&B SDK (`wandb`) and the Report and Workspace API (`wandb-workspaces`):
```bash theme={"system"}
pip install wandb wandb-workspaces
```
2. Import the W\&B Python SDK and the Report and Workspace API:
```python theme={"system"}
import wandb
import wandb_workspaces.reports.v2 as wr
```
3. Create a report instance with `wandb_workspaces.reports.v2.Report` or with the Report Class Public API ([`wandb.apis.reports`](/products/wandb/ref/python/public-api/api#reports)). Specify the project to associate with the report:
```python theme={"system"}
report = wr.Report(project="report_standard")
```
4. Save the report. Weights & Biases doesn't upload reports until you call the `.save()` method:
```python theme={"system"}
report.save()
```
For more information, see [Edit a report](/products/wandb/reports/edit-a-report).
# Compare runs across projects
Source: https://docs.coreweave.com/products/wandb/reports/cross-project-reports
Compare runs from two different projects with cross-project reports.
> [!NOTE]
>
> Watch a [video demonstrating comparing runs across projects](https://www.youtube.com/watch?v=uD4if_nGrs4) (2 min).
Cross-project reports let you compare runs from two different projects side by side. This is useful when you want to evaluate experiments tracked under separate Weights & Biases projects without duplicating data. This page describes how to pull runs from another project into a report and how to share that report with collaborators who don't have Weights & Biases accounts.
To compare runs across projects, use the project selector in the run set table to pick a project.
The visualizations in the section pull columns from the first active run set. If you don't see the metric you're looking for in the line plot, make sure that the first run set checked in the section has that column available.
This feature supports history data on time series lines, but doesn't support pulling different summary metrics from different projects. In other words, you can't create a scatter plot from columns that are only logged in another project.
If you need to compare runs from two projects and the columns aren't working, add a tag to the runs in one project and then move those runs to the other project. You can still filter only the runs from each project, but the report includes all the columns for both sets of runs.
## View-only report links
When you need to share a report with someone outside your project (such as a stakeholder who doesn't have a Weights & Biases account), use a view-only link. Share a view-only link to a report that is in a private project or team project.
View-only report links add a secret access token to the URL, so anyone who opens the link can view the page. Anyone can use the magic link to view the report without logging in first. For customers on [W\&B Local](/products/wandb/platform/hosting) private cloud installations, these links remain behind your firewall. Only team members who have both access to your private instance and access to the view-only link can view the report.
In view-only mode, someone who isn't logged in can see the charts and hover over them to see tooltips of values, zoom in and out on charts, and scroll through columns in the table. They can't create new charts or new table queries to explore the data. View-only visitors can't click a run to get to the run page, and they can't see the share modal. Instead, they see a tooltip on hover that says `Sharing not available for view only access`.
> [!NOTE]
>
> The magic links are only available for Private and Team projects. For Public projects (anyone can view) or Open projects (anyone can view and contribute runs), you can't turn the links on or off. The project is already public, so the report is already available to anyone with the link.
## Send a graph to a report
You can also populate a report with charts you've already built in a workspace, rather than recreating them. To preserve a chart from a workspace alongside related analysis, send it directly to a report. Click the dropdown menu on the chart or panel you want to copy, then click **Add to report** to select the destination report.
# Edit a report
Source: https://docs.coreweave.com/products/wandb/reports/edit-a-report
Edit a report interactively with the App UI or programmatically with the W&B SDK.
> [!NOTE]
>
> W\&B Report and Workspace API is in Public Preview.
This page describes how to edit a report interactively with the App UI or programmatically with the W\&B SDK. It covers adding plots, run sets, code blocks, markdown, HTML elements, and rich media. It also covers filtering and grouping run sets, organizing report layout, and visualizing multi-dimensional relationships.
A report's body consists of *blocks*. Blocks contain text, images, embedded visualizations, plots from experiments and runs, and panel grids.
*Panel grids* are a specific type of block that hold panels and *run sets*. Run sets are a collection of runs logged to a project in Weights & Biases. Panels are visualizations of run set data.
> [!TIP]
>
> To have [ARIA](/products/aria) revise a report you own, open the report and click **Edit with ARIA**, then describe the changes you want. ARIA saves its changes to the existing report.
> [!NOTE]
>
> Run the [Programmatic workspaces notebook](https://colab.research.google.com/github/wandb/wandb-workspaces/blob/Update-wandb-workspaces-tuturial/Workspace_tutorial.ipynb) for an end-to-end tutorial of creating a customized saved workspace view.
> [!NOTE]
>
> To programmatically edit a report, you must have the W\&B Report and Workspace API (`wandb-workspaces`) installed along with the W\&B Python SDK:
>
> ```bash theme={"system"}
> pip install wandb wandb-workspaces
> ```
## Add plots
Plots let you visualize run data inside a report. Each panel grid has a set of run sets and a set of panels. The run sets in the panel grid control what data appears on the panels in the grid. Create a new panel grid if you want to add charts that pull data from a different set of runs.
**App UI**
Enter a forward slash (`/`) in the report to display a dropdown menu. Select **Add panel** to add a panel. You can add any panel that Weights & Biases supports, including a line plot, scatter plot, or parallel coordinates chart.
**Report and Workspace API**
Add plots to a report programmatically with the SDK. Pass a list of one or more plot or chart objects to the `panels` parameter in the `PanelGrid` Public API class. Create a plot or chart object with its associated Python class.
The following examples demonstrate how to create a line plot and scatter plot.
```python theme={"system"}
import wandb
import wandb_workspaces.reports.v2 as wr
report = wr.Report(
project="report-editing",
title="Report title",
description="Report description.",
)
blocks = [
wr.PanelGrid(
panels=[
wr.LinePlot(x="time", y="velocity"),
wr.ScatterPlot(x="time", y="acceleration"),
]
)
]
report.blocks = blocks
report.save()
```
For more information about available plots and charts you can add to a report programmatically, see `wr.panels`.
## Add run sets
Add run sets from projects interactively with the App UI or the W\&B SDK.
**App UI**
Enter a forward slash (`/`) in the report to display a dropdown menu. From the dropdown, choose **Panel Grid**. Weights & Biases automatically imports the run set from the project the report was created from.
If you import a panel into a report, the report inherits the run names from the project. In the report, you can optionally [rename a run](/products/wandb/runs#rename-a-run) to give the reader more context. The rename applies only to the individual panel. If you clone the panel in the same report, the cloned panel also reflects the new name.
1. In the report, click the pencil icon to open the report editor.
2. In the run set, find the run to rename. Hover over the run name and click the **action ()** menu. Select one of the following choices, then submit the form.
* **Rename run for project**: Rename the run across the entire project. To generate a new random name, leave the field blank.
* **Rename run for panel grid**: Rename the run only in the report, preserving the existing name in other contexts. You can't generate a new random name with this option.
3. Click **Publish report**.
**Report and Workspace API**
Add run sets from projects with the `wr.Runset()` and `wr.PanelGrid` classes. To add a run set, follow these steps:
1. Create a `wr.Runset()` object instance. Provide the name of the project that contains the run sets for the project parameter and the entity that owns the project for the entity parameter.
2. Create a `wr.PanelGrid()` object instance. Pass a list of one or more run set objects to the `runsets` parameter.
3. Store one or more `wr.PanelGrid()` object instances in a list.
4. Update the report instance blocks attribute with the list of panel grid instances.
Replace `[PROJECT-NAME]` with your Weights & Biases project name and `[ENTITY-NAME]` with your Weights & Biases entity name in the following example:
```python theme={"system"}
import wandb
import wandb_workspaces.reports.v2 as wr
report = wr.Report(
project="report-editing",
title="Report title",
description="Report description.",
)
panel_grids = wr.PanelGrid(
runsets=[wr.RunSet(project="[PROJECT-NAME]", entity="[ENTITY-NAME]")]
)
report.blocks = [panel_grids]
report.save()
```
You can optionally add runsets and panels with one call to the SDK:
```python theme={"system"}
import wandb
report = wr.Report(
project="report-editing",
title="Report title",
description="Report description.",
)
panel_grids = wr.PanelGrid(
panels=[
wr.LinePlot(
title="line title",
x="x",
y=["y"],
range_x=[0, 100],
range_y=[0, 100],
log_x=True,
log_y=True,
title_x="x axis title",
title_y="y axis title",
ignore_outliers=True,
groupby="hyperparam1",
groupby_aggfunc="mean",
groupby_rangefunc="minmax",
smoothing_factor=0.5,
smoothing_type="gaussian",
smoothing_show_original=True,
max_runs_to_show=10,
plot_type="stacked-area",
font_size="large",
legend_position="west",
),
wr.ScatterPlot(
title="scatter title",
x="y",
y="y",
# z='x',
range_x=[0, 0.0005],
range_y=[0, 0.0005],
# range_z=[0,1],
log_x=False,
log_y=False,
# log_z=True,
running_ymin=True,
running_ymean=True,
running_ymax=True,
font_size="small",
regression=True,
),
],
runsets=[wr.RunSet(project="[PROJECT-NAME]", entity="[ENTITY-NAME]")],
)
report.blocks = [panel_grids]
report.save()
```
## Freeze a run set
A report automatically updates run sets to show the latest data from the project. Freeze a run set to preserve its state in a report at a specific time, so that later runs don't change what the report displays.
To freeze a run set when viewing a report, click the snowflake icon in its panel grid near the **Filter** button.
## Group a run set programmatically
Group runs in a run set programmatically with the [Workspace and Reports API](/products/wandb/ref/wandb_workspaces/reports). Grouping organizes related runs together in panels, which makes comparing configurations and results easier.
You can group runs in a run set by config values, run metadata, or summary metrics. The following table lists the available grouping methods along with the available keys for each grouping method:
| Grouping method | Description | Available keys |
| - | - | - |
| Config values | Group runs by config values | Values specified in config parameter in `wandb.init(config=)` |
| Run metadata | Group runs by run metadata | `State`, `Name`, `JobType` |
| Summary metrics | Group runs by summary metrics | Values you log to a run with `wandb.Run.log()` |
### Group runs by config values
Group runs by config values to compare runs with similar configurations. Config values are parameters you specify in your run configuration `(wandb.init(config=))`. To group runs by config values, use the `config.[KEY]` syntax, where `[KEY]` is the name of the config value you want to group by.
For example, the following code snippet first initializes a run with a config value for `group`, then groups runs in a report based on the `group` config value. Replace `[ENTITY]` and `[PROJECT]` with your Weights & Biases entity and project names.
```python theme={"system"}
import wandb
import wandb_workspaces.reports.v2 as wr
entity = "[ENTITY]"
project = "[PROJECT]"
for group in ["control", "experiment_a", "experiment_b"]:
for i in range(3):
with wandb.init(entity=entity, project=project, group=group, config={"group": group, "run": i}, name=f"{group}_run_{i}") as run:
# Simulate some training
for step in range(100):
run.log({
"acc": 0.5 + (step / 100) * 0.3 + (i * 0.05),
"loss": 1.0 - (step / 100) * 0.5
})
```
Within your Python script or notebook, you can then group runs by the `config.group` value:
```python theme={"system"}
runset = wr.Runset(
project=project,
entity=entity,
groupby=["config.group"] # Group by the "group" config value
)
```
Continuing from the previous example, you can create a report with the grouped run set:
```python theme={"system"}
report = wr.Report(
entity=entity,
project=project,
title="Grouped Runs Example",
)
report.blocks = [
wr.PanelGrid(
runsets=[runset],
)
]
report.save()
```
### Group runs by run metadata
Group runs by metadata to organize them based on intrinsic run properties rather than logged values. You can group by a run's name (`Name`), state (`State`), or job type (`JobType`).
Continuing from the previous example, you can group your runs by their name with the following code snippet:
```python theme={"system"}
runset = wr.Runset(
project=project,
entity=entity,
groupby=["Name"] # Group by run names
)
```
> [!NOTE]
>
> The name of the run is the name you specify in the `wandb.init(name=)` parameter. If you don't specify a name, Weights & Biases generates a random name for the run.
>
> You can find the name of the run in the **Overview** page of a run in the W\&B App or programmatically with `Api.runs().run.name`.
### Group runs by summary metrics
The following examples demonstrate how to group runs by summary metrics. Summary metrics are the values you log to a run with `wandb.Run.log()`. After you log a run, you can find the names of your summary metrics in the W\&B App under the **Summary** section of a run's **Overview** page.
The syntax for grouping runs by summary metrics is `summary.[KEY]`, where `[KEY]` is the name of the summary metric you want to group by.
For example, suppose you log a summary metric called `acc`. Replace `[ENTITY]` and `[PROJECT]` with your Weights & Biases entity and project names:
```python theme={"system"}
import wandb
import wandb_workspaces.reports.v2 as wr
entity = "[ENTITY]"
project = "[PROJECT]"
for group in ["control", "experiment_a", "experiment_b"]:
for i in range(3):
with wandb.init(entity=entity, project=project, group=group, config={"group": group, "run": i}, name=f"{group}_run_{i}") as run:
# Simulate some training
for step in range(100):
run.log({
"acc": 0.5 + (step / 100) * 0.3 + (i * 0.05),
"loss": 1.0 - (step / 100) * 0.5
})
```
You can then group runs by the `summary.acc` summary metric:
```python theme={"system"}
runset = wr.Runset(
project=project,
entity=entity,
groupby=["summary.acc"] # Group by summary values
)
```
## Filter a run set programmatically
Programmatically filter run sets and add them to a report with the [Workspace and Reports API](/products/wandb/ref/wandb_workspaces/reports). Filtering narrows a run set to the specific runs you want to display, based on config values, metrics, tags, or run properties.
The general syntax for a filter expression is:
```text theme={"system"}
Filter('[KEY]') operation [VALUE]
```
In this expression, `[KEY]` is the name of the filter, `operation` is a comparison operator (for example, `>`, `<`, `==`, `in`, `not in`, `or`, and `and`), and `[VALUE]` is the value to compare against. `Filter` is a placeholder for the type of filter you want to apply. The following table lists the available filters and their descriptions:
| Filter | Description | Available keys |
| - | - | - |
| `Config('[KEY]')` | Filter by config values | Values specified in `config` parameter in `wandb.init(config=)`. |
| `SummaryMetric('[KEY]')` | Filter by summary metrics | Values you log to a run with `wandb.Run.log()`. |
| `Tags('[KEY]')` | Filter by tags | Tag values that you add to your run (programmatically or with the W\&B App). |
| `Metric('[KEY]')` | Filter by run properties | `tags`, `state`, `displayName`, `jobType` |
After you define your filters, you can create a report and pass the filtered run sets to `wr.PanelGrid(runsets=)`. See the **Report and Workspace API** tabs throughout this page for more information about how to add different elements to a report programmatically.
The following examples demonstrate how to filter run sets in a report. Replace values enclosed in brackets (for example, `[ENTITY]` and `[PROJECT]`) with your own values.
### Config filters
Filter a run set by one or more config values. Config values are parameters you specify in your run configuration (`wandb.init(config=)`).
For example, the following code snippet first initializes a run with a config value for `learning_rate` and `batch_size`, then filters runs in a report based on the `learning_rate` config value.
```python theme={"system"}
import wandb
config = {
"learning_rate": 0.01,
"batch_size": 32,
}
with wandb.init(project="[PROJECT]", entity="[ENTITY]", config=config) as run:
# Your training code here
pass
```
Within your Python script or notebook, you can then programmatically filter runs that have a learning rate greater than `0.01`.
```python theme={"system"}
import wandb_workspaces.reports.v2 as wr
runset = wr.Runset(
entity="[ENTITY]",
project="[PROJECT]",
filters="Config('learning_rate') > 0.01"
)
```
You can also filter by multiple config values with the `and` operator:
```python theme={"system"}
runset = wr.Runset(
entity="[ENTITY]",
project="[PROJECT]",
filters="Config('learning_rate') > 0.01 and Config('batch_size') == 32"
)
```
Continuing from the previous example, you can create a report with the filtered run set as follows:
```python theme={"system"}
report = wr.Report(
entity="[ENTITY]",
project="[PROJECT]",
title="My Report"
)
report.blocks = [
wr.PanelGrid(
runsets=[runset],
panels=[
wr.LinePlot(
x="Step",
y=["accuracy"],
)
]
)
]
report.save()
```
### Metric filters
Filter run sets based on a run's tag (`tags`), run state (`state`), run name (`displayName`), or job type (`jobType`).
> [!NOTE]
>
> `Metric` filters use a different syntax than other filters. You must pass values as a list.
>
> ```text theme={"system"}
> Metric('[KEY]') operation [VALUE]
> ```
For example, consider the following Python snippet that creates three runs and assigns each of them a name:
```python theme={"system"}
import wandb
with wandb.init(project="[PROJECT]", entity="[ENTITY]") as run:
for i in range(3):
run.name = f"run{i+1}"
# Your training code here
pass
```
When you create your report, you can filter runs by their display name. For example, to filter runs with names `run1`, `run2`, and `run3`, you can use the following code:
```python theme={"system"}
runset = wr.Runset(
entity="[ENTITY]",
project="[PROJECT]",
filters="Metric('displayName') in ['run1', 'run2', 'run3']"
)
```
> [!NOTE]
>
> You can find the name of the run in the **Overview** page of a run in the W\&B App or programmatically with `Api.runs().run.name`.
The following examples demonstrate how to filter a run set by the run's state (`finished`, `crashed`, or `running`):
```python theme={"system"}
runset = wr.Runset(
entity="[ENTITY]",
project="[PROJECT]",
filters="Metric('state') in ['finished']"
)
```
```python theme={"system"}
runset = wr.Runset(
entity="[ENTITY]",
project="[PROJECT]",
filters="Metric('state') not in ['crashed']"
)
```
### SummaryMetric filters
Filter a run set by summary metrics to focus on runs that meet specific outcome thresholds, such as accuracy or loss values. Summary metrics are the values you log to a run with `wandb.Run.log()`. After you log a run, you can find the names of your summary metrics in the W\&B App under the **Summary** section of a run's **Overview** page.
```python theme={"system"}
runset = wr.Runset(
entity="[ENTITY]",
project="[PROJECT]",
filters="SummaryMetric('accuracy') > 0.9"
)
```
```python theme={"system"}
runset = wr.Runset(
entity="[ENTITY]",
project="[PROJECT]",
filters="Metric('state') in ['finished'] and SummaryMetric('train/train_loss') < 0.5"
)
```
### Tags filters
Filter a run set by tags to show only runs that you've labeled with a particular tag. Tags are values you add to a run (programmatically or with the W\&B App).
```python theme={"system"}
runset = wr.Runset(
entity="[ENTITY]",
project="[PROJECT]",
filters="Tags('training') == 'training'"
)
```
## Add code blocks
Add code blocks to share configuration, examples, or snippets alongside your results. You can add them interactively with the App UI or programmatically with the W\&B SDK.
**App UI**
Enter a forward slash (`/`) in the report to display a dropdown menu. From the dropdown, choose **Code**.
Select the name of the programming language in the code block to expand a dropdown. From the dropdown, select your programming language syntax. You can choose from JavaScript, Python, CSS, JSON, HTML, Markdown, and YAML.
**Report and Workspace API**
Use the `wr.CodeBlock` class to create a code block programmatically. Provide the name of the language and the code you want to display for the language and code parameters, respectively.
The following example demonstrates a list in a YAML file:
```python theme={"system"}
import wandb
import wandb_workspaces.reports.v2 as wr
report = wr.Report(project="report-editing")
report.blocks = [
wr.CodeBlock(
code=["this:", "- is", "- a", "cool:", "- yaml", "- file"], language="yaml"
)
]
report.save()
```
This renders a code block similar to:
```yaml theme={"system"}
this:
- is
- a
cool:
- yaml
- file
```
The following example demonstrates a Python code block:
```python theme={"system"}
report = wr.Report(project="report-editing")
report.blocks = [wr.CodeBlock(code=["Hello, World!"], language="python")]
report.save()
```
This renders a code block similar to:
```md theme={"system"}
Hello, World!
```
## Add markdown
Add markdown to format narrative text, headings, and inline emphasis in your report. You can add markdown interactively with the App UI or programmatically with the W\&B SDK.
**App UI**
Enter a forward slash (`/`) in the report to display a dropdown menu. From the dropdown, choose **Markdown**.
**Report and Workspace API**
Use the `wr.MarkdownBlock` class to create a markdown block programmatically. Pass a string to the `text` parameter:
```python theme={"system"}
import wandb
import wandb_workspaces.reports.v2 as wr
report = wr.Report(project="report-editing")
report.blocks = [
wr.MarkdownBlock(text="Markdown cell with *italics* and **bold** and $e=mc^2$")
]
```
This renders a markdown block similar to:
## Add HTML elements
Add HTML elements such as headings and lists to structure your report. You can add them interactively with the App UI or programmatically with the W\&B SDK.
> [!NOTE]
>
> When you log an image with `wandb.Image`, Weights & Biases saves the file under a generated name that includes a content hash, such as `media/images/[key]_[step]_[hash].png`, instead of the original filename. As a result, an `![]()
` tag or markdown image that references the original filename doesn't resolve. To embed a logged image in a report, log the image first, then use the generated file path as the image's `src` value.
**App UI**
Enter a forward slash (`/`) in the report to display a dropdown menu. From the dropdown, select a type of text block. For example, to create an H2 heading block, select the `Heading 2` option.
**Report and Workspace API**
Pass a list of one or more HTML elements to the report's `blocks` attribute. The following example demonstrates how to create an H1, H2, and an unordered list:
```python theme={"system"}
import wandb
import wandb_workspaces.reports.v2 as wr
report = wr.Report(project="report-editing")
report.blocks = [
wr.H1(text="How Programmatic Reports work"),
wr.H2(text="Heading 2"),
wr.UnorderedList(items=["Bullet 1", "Bullet 2"]),
]
report.save()
```
This renders the HTML elements to the following:
## Embed rich media links
Embed rich media such as tweets, videos, and audio to enrich your report with external context. You can embed rich media with the App UI or with the W\&B SDK.
**App UI**
Copy and paste URLs into reports to embed rich media within the report. The following animations demonstrate how to copy and paste URLs from Twitter, YouTube, and SoundCloud.
### Twitter
Copy and paste a Tweet link URL into a report to view the Tweet within the report.
### YouTube
Copy and paste a YouTube video URL link to embed a video in the report.
### SoundCloud
Copy and paste a SoundCloud link to embed an audio file into a report.
**Report and Workspace API**
Pass a list of one or more embedded media objects to the report's `blocks` attribute. The following example demonstrates how to embed video and Twitter media into a report:
```python theme={"system"}
import wandb
import wandb_workspaces.reports.v2 as wr
report = wr.Report(project="report-editing")
report.blocks = [
wr.Video(url="https://www.youtube.com/embed/6riDJMI-Y8U"),
wr.Twitter(
embed_html='
'Soft Delete'] SDKDelete --> SoftDelete TTLDelete --> SoftDelete SoftDelete --> GCWait[(Wait for
best-effort
Garbage Collection)] GCWait --> GCRun[Garbage Collection
Process Runs
- Reviews all soft-deleted artifacts
- Checks file dependencies] GCRun --> CheckUsage{Are files used by
other artifact versions?} CheckUsage -->|Yes| KeepFiles[Files Kept in Storage
- Artifact marked deleted
- Files remain for other versions] CheckUsage -->|No| DeleteFiles[Files Deleted from Storage
- Artifact fully removed
- Storage space reclaimed] KeepFiles --> End([End]) DeleteFiles --> End style Start fill:#e1f5fe,stroke:#333,stroke-width:2px,color:#000 style SoftDelete fill:#fff3e0,stroke:#333,stroke-width:2px,color:#000 style GCRun fill:#f3e5f5,stroke:#333,stroke-width:2px,color:#000 style KeepFiles fill:#e8f5e9,stroke:#333,stroke-width:2px,color:#000 style DeleteFiles fill:#ffebee,stroke:#333,stroke-width:2px,color:#000 style End fill:#e0e0e0,stroke:#333,stroke-width:2px,color:#000 ``` You can schedule when artifacts are deleted from Weights & Biases with TTL policies. For more information, see [Manage data retention with Artifact TTL policy](/products/wandb/artifacts/ttl). > [!NOTE] > > Artifacts deleted by a TTL policy, the W\&B Python SDK, or the W\&B App are first soft-deleted. Soft-deleted artifacts are then garbage-collected before they are permanently deleted. > [!NOTE] > > Deleting an entity, project, or artifact collection triggers the artifact deletion process described on this page. When you delete a run and choose to delete its associated artifacts, those artifacts follow the same soft-delete and garbage collection workflow. ## Delete an artifact version Delete an artifact version interactively with the W\&B App or programmatically with the W\&B Python SDK. **W\&B App** To delete an artifact version: 1. Navigate to the project that contains the artifact version you want to delete. 2. Select the **Artifacts** tab. 3. From the list of artifact types, select the type of artifact that contains the version you want to delete. 4. Click the **action ()** menu next to the artifact version you want to delete. 5. From the dropdown, choose **Delete Version**. **W\&B Python SDK** Delete an artifact version programmatically with the [wandb.Artifact.delete()](/products/wandb/ref/python/experiments/artifact#delete) method. Provide the full name of the artifact. The full name consists of `
\n' ), ] report.save() ``` ## Duplicate panel grids Duplicate a panel grid to reuse its layout in the same report or in a different report. Select a panel grid and copy-paste it to duplicate it in the same report or paste it into a different report. Highlight a whole panel grid section by selecting the drag handle in the upper right corner. Click and drag to highlight and select a region in a report such as panel grids, text, and headings.  ## Delete panel grids Delete a panel grid when you no longer need its layout or content in the report. Select a panel grid and press `delete` on your keyboard to delete a panel grid.  ## Collapse headers to organize reports Collapse headers in a report to hide content within a text block. When the report loads, only expanded headers show content. Collapsing headers in reports helps organize your content and prevents loading more data than you need. The following animation demonstrates the process.  ## Configure report and chart display Configure how the report and its charts appear. The chart display and layout options are located in the upper-left corner of the report interface. ### Configure chart display Use the chart display options to change how charts appear. | Icon | Option | Description | | - | - | - | | `x →` | **X-axis option** | Choose the x-axis metric for supported line charts, such as step or elapsed time. | | Iron icon | **Smoothing settings** | Smooth chart curves to make trends easier to see. Smoothing changes only the visualization, not the logged data. | | Points with an `×` | **Exclude extreme outliers when scaling charts** | Adjust chart scaling to reduce the effect of extreme values. This option doesn't delete logged data. The visual effect depends on the chart's sampling mode. | ### Configure layout Select **Narrow**, **Wide**, or **Full width** to change the report width. ## Publish report changes Your edits are saved automatically as a draft. To publish the draft, select **Publish to project** for a new report or **Publish changes** for an existing published report. ## Visualize relationships across multiple dimensions When you want to surface relationships between three or more variables in a single chart, a standard 2D plot can be limiting. To compare more variables than a 2D plot can show, you can use a color gradient as an additional dimension. Using a color gradient to represent one of the variables can make patterns easier to interpret. 1. Choose a variable to represent with a color gradient, like penalty scores or learning rates. This provides a clearer understanding of how penalty (color) interacts with reward or side effects (y-axis) over training time (x-axis). 2. Highlight key trends. Hover over a specific group of runs to highlight them in the visualization. # Embed a report Source: https://docs.coreweave.com/products/wandb/reports/embed-reports Embed W&B reports directly into Notion or with an HTML `iframe` element. Embed a W\&B report in an external page or tool so that your team can view live experiment results alongside other documentation. This page describes how to generate an embed code from a report and use it in HTML, Confluence, Notion, or Gradio. ## HTML `iframe` element Use an HTML `iframe` element to embed a report on any web page that accepts custom HTML. Weights & Biases provides a prebuilt embed code that you can copy directly from the report. In the upper right of a report, select the **Share** button to open a dialog, then select **Copy embed code**. The copied snippet is an `iframe` HTML element that you can paste into any page that accepts custom HTML. > [!NOTE] > > You can only embed and render reports from public projects or reports that you've shared using [view-only links](/products/wandb/reports/cross-project-reports#view-only-report-links) (the link's access token is part of the URL).  ## Confluence Confluence supports embedded reports through its built-in `iframe` macro. The following animation demonstrates how to insert the direct link to the report within an `iframe` cell in Confluence.  ## Notion Notion's **Embed** block accepts the same embed code that Weights & Biases provides. The following animation demonstrates how to insert a report into a Notion document using an **Embed** block in Notion and the report's embed code.  ## Gradio To embed a W\&B report in a Gradio app, including an app hosted on Hugging Face Spaces, use Gradio's `gr.HTML` element to render an `iframe` that points to the report URL. ```python theme={"system"} import gradio as gr def wandb_report(url): iframe = f'The voice of an angel, truly. #MassEffect pic.twitter.com/nMev97Uw7F
— Mass Effect (@masseffect) August 20, 2021