Telemetry Forwarding

Forwarding metrics and logs from your CKS clusters

CoreWeave supports forwarding telemetry data from CKS clusters to external systems. Compared to direct querying, telemetry forwarding provides a more efficient and reliable method to export metrics and logs for storage, monitoring, and analysis purposes.

Forwarding metrics

All CKS clusters come with Prometheus Operator pre-installed. Using Prometheus Operator, you can deploy Prometheus Agent inside a CKS cluster, and configure it to use remote write for writing to an external system.

Prerequisites

To forward metrics to an external system, you will need:

• kubectl installed and configured to your target CKS cluster
• The address and credentials for the external system for which to use remote write

Warning

Always provide credentials as Secrets. To do this, you will first need to create the appropriate Secrets objects in your namespace. Never include credentials as plaintext in your YAML file.

Choosing a namespace

You may deploy Prometheus Agent to an existing namespace, or create a new namespace with kubectl.

If desired, create a new namespace with the following command, replacing my-remote-write-namespace with your chosen namespace:

Example

$
kubectl create namespace my-remote-write-namespace

Deploying Prometheus Agent

The Prometheus Operator pre-installed in your CKS cluster allows you to deploy Prometheus Agent as follows:

First, prepare a Prometheus Agent CustomResource YAML file with the appropriate configuration for your remote-write destination, as in the following example:

sample-prometheus-agent.yaml

apiVersion: monitoring.coreos.com/v1alpha1
kind: PrometheusAgent
metadata:
  name: my-prom-agent
  namespace: my-remote-write-namespace
spec:
  remoteWrite:
    - url: ...
      ...

Note

For more details on specifying the remoteWrite configuration(s), see Prometheus Operator's API reference.

Next, use kubectl to deploy the Prometheus Agent to your CKS cluster with the following command, replacing sample-prometheus-agent.yaml with the appropriate filename:

Example

$
kubectl apply -f sample-prometheus-agent.yaml

Metrics generated within your CKS cluster will now remote-write to your specified destination.

Known limitations

This approach will only forward metrics that are generated directly from within your CKS cluster. CoreWeave exposes additional metrics, such as those used for billing, through our central metrics service. These metrics are computed in our metrics service downstream, and are thus not available upstream within your cluster and will not be forwarded to you with this approach.

To forward metrics that are only available in our central metrics service, you may consider writing and running your own service that queries directly from our central metrics service and forwards results to your external systems.

Forwarding logs

To enable log forwarding, please contact CoreWeave support to assist with setup and configuration. CKS supports the following endpoint types for log forwarding:

Endpoint type	Supported versions
Loki	2.4.x and above
Fluentbit (via TCP socket with JSON codec)	2.0.x and above
Kafka	3.0.x and above
AWS S3	N/A