Skip to main content

Query Logs and Metrics

Learn how to query metrics and logs

CoreWeave allows for direct querying of metrics and logs collected within a CKS environment for those who wish to use a solution other than CoreWeave Grafana or self-hosted Grafana.

To efficiently export metrics and logs at scale, consider setting up telemetry forwarding with CoreWeave Telecaster™ to an external system.

For more information about metric and log data sources, see Data sources.

Query metrics​

You can query metrics through the API.

Base URLobserve.coreweave.com
Supported endpoints
/api/v1/query
/api/v1/labels
/api/v1/query_range
/api/v1/series
/api/v1/label/<label_name>/values
/api/v1/query_exemplars
/api/v1/metadata
/api/v1/rules

Metric query format​

To query metrics using curl, see the following example which returns the total time the GPU was in idle mode (in seconds):

Example
$
 curl -G "https://observe.coreweave.com/api/v1/query" \

  -H 'Authorization: Bearer <YOUR_COREWEAVE_ACCESS_TOKEN>' \

  --data-urlencode 'query=node_gpu_seconds_total{mode="idle"}'

For more information about querying Prometheus metrics, see the Prometheus HTTP API documentation

Note

CoreWeave does not support the Prometheus Federation API.

Query logs​

You can query logs through the API.

Base URLobserve.coreweave.com
Supported endpoints
/loki/api/v1/query
/loki/api/v1/query_range
/loki/api/v1/labels
/loki/api/v1/label/<name>/values
/loki/api/v1/series
/loki/api/v1/index/stats
/loki/api/v1/index/volume
/loki/api/v1/index/volume_range
/loki/api/v1/format_query

Log query format​

To query logs using curl, see the following example which return Kubernetes audit logs for a cluster named llmd:

Example
$
curl -G "https://observe.coreweave.com/loki/api/v1/query_range" \

  -H 'Authorization: Bearer <YOUR_COREWEAVE_ACCESS_TOKEN>' \

  --data-urlencode 'query={logging_component="unicaster", app="kube-apiserver", stream="", cluster="llmd"} | json | apiVersion="audit.k8s.io/v1" | json sourceIPs="sourceIPs" | json user_groups="user.groups"'

Grafana Explore​

Grafana Explore is now available on in CoreWeave Grafana. Explore enables customers to use the full expressivity of PromQL and LogQL for ad hoc queries on metrics and logs and to view metadata such as which metrics are available, or which labels are available to use for filtering PromQL queries.

While CoreWeave's default dashboards offer curated views of specific logs and metrics, the Explore tab allows users to more deeply investigate and learn about metrics.

Learn more

Learn more about Explore from Grafana's official documentation.

Last updated on