Response caching - CoreWeave Docs

Exact-response caching lets a project-routed request reuse the response the proxy already produced for an identical request. When a request opts in and a matching response exists, the proxy returns that response without contacting the provider. This saves provider spend and latency for workloads that send the same request more than once, such as evaluation suites, regression tests, retries, and development loops. Caching is off by default and applies per request. A cached response is replayed byte for byte, so a hit returns the same answer even when the request uses sampling parameters such as a nonzero temperature.

Enable caching for a request

Set the wandb-cache-mode request header to one of the following modes:

Value	Read an existing response	Store a successful response
`readWrite`	Yes	Yes
`readOnly`	Yes	No
`writeOnly`	No	Yes

To bypass the cache, omit the header. For compatibility with OpenPipe clients, the proxy also accepts the deprecated op-cache header. It takes the same modes, plus true as an alias for readWrite and false to bypass the cache. If a request sends both headers, they must select the same mode. Otherwise, the proxy returns 400 Bad Request.

Python
JavaScript
cURL

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://proxy.training.wandb.ai/v1",
    api_key=os.environ["WANDB_API_KEY"],
)

raw = client.chat.completions.with_raw_response.create(
    model="ticket-classifier",
    messages=[
        {"role": "user", "content": "My package arrived damaged. What should I do?"}
    ],
    extra_body={"metadata": {"wandb.entity": "your-team"}},
    extra_headers={"wandb-cache-mode": "readWrite"},
)

print(raw.headers.get("wandb-cache-status"))  # "hit" or "miss"
response = raw.parse()
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://proxy.training.wandb.ai/v1",
  apiKey: process.env.WANDB_API_KEY,
});

const { data: response, response: raw } = await client.chat.completions
  .create(
    {
      model: "ticket-classifier",
      messages: [
        { role: "user", content: "My package arrived damaged. What should I do?" },
      ],
      metadata: { "wandb.entity": "your-team" },
    },
    { headers: { "wandb-cache-mode": "readWrite" } },
  )
  .withResponse();

console.log(raw.headers.get("wandb-cache-status")); // "hit" or "miss"
console.log(response.choices[0].message.content);

curl -i https://proxy.training.wandb.ai/v1/chat/completions \
  -H "Authorization: Bearer $WANDB_API_KEY" \
  -H "Content-Type: application/json" \
  -H "wandb-cache-mode: readWrite" \
  -d '{
    "model": "ticket-classifier",
    "messages": [
      {"role": "user", "content": "My package arrived damaged. What should I do?"}
    ],
    "metadata": {"wandb.entity": "your-team"}
  }'

When the request enables cache reads (readWrite or readOnly), the response includes a wandb-cache-status header with the value hit or miss. The proxy sends the same value in the x-wandb-cache compatibility header. The response to a writeOnly request doesn’t include either header.

What makes two requests identical

The proxy looks up a cached response after it resolves the project version and routing revision, and before it selects a provider. The cache key combines the following values:

The W&B entity, project, and project version that the proxy resolves from the model value.
The routing revision that applies to the version when the request arrives.
The request path and query string.
A SHA-256 hash of the request body, serialized with object keys in sorted order and with the following values excluded.

The proxy removes the following values from the body before hashing, so they don’t change the key:

The raw model string. Its resolved project, version, and routing revision are already in the key, so only equivalent references share entries: ticket-classifier and ticket-classifier@v1 both resolve to version 1. Different versions never share entries.
Metadata keys that start with wandb., including wandb.entity and wandb.thread_id.
stream: false, which the proxy treats the same as omitting stream.
An empty metadata object, which the proxy treats the same as omitting metadata.

The proxy hostname and the order of query parameters don’t change the key either. Each time you save a version’s routing, Model Distillation publishes a new routing revision, which changes the key. Existing entries stop matching but remain until they expire. The proxy doesn’t hash parameter overrides on a routing target. Changing an override is a routing change like any other, so it also changes the key. Every other field in the body is part of the hash, regardless of whether the provider uses it. These fields include messages, tools, tool_choice, response_format, temperature, top_p, max_tokens, seed, n, stop, user, any provider-specific fields, and metadata keys that don’t start with wandb., such as gen_ai.conversation.id or user.id. The order of keys within an object doesn’t matter. The order of items in an array, such as messages, does. Entries are scoped to your W&B entity and project. An identical request from another entity, or for another project in your entity, never matches your entries. A cached response records the routing target that produced it. The proxy serves a hit from that target without weighted or sticky routing. Direct provider/model requests can also set a cache header. The proxy keys their entries separately by entity, provider, model reference, and the caller’s W&B identity. These entries never match project entries.

Limits

The cache has the following limits:

Non-streaming requests only. Requests with stream: true that set a cache header return 400 Bad Request.
Successful responses only. The proxy stores only 2xx responses up to 8 MiB.
Retention is 7 days. Entries are deleted 7 days after they’re written. Hits don’t refresh them.
Invalid modes are rejected. Any wandb-cache-mode value other than those listed, or conflicting modes in wandb-cache-mode and op-cache, returns 400 Bad Request.

Cache failures fail open. If the cache can’t be read or written, the proxy forwards the request to the provider as usual. The proxy doesn’t coalesce concurrent requests. If identical requests arrive before the proxy stores the first response, each one reaches the provider, and the proxy stores the last response written.

Traces and Analytics

The proxy still records a cache hit as a project trace. The trace keeps the original response and its token usage, is marked with cache_hit=true, and has no provider latency. A request that enables cache reads but is forwarded to the provider is marked with cache_hit=false. Analytics counts the provider spend of a cache hit as zero, because no inference ran, while token totals still include the replayed usage. Dataset creation deduplicates identical inputs, so replayed responses don’t add duplicate training rows.

Browser clients

The proxy allows the wandb-cache-mode and op-cache request headers in cross-origin requests, and it exposes wandb-cache-status, x-wandb-cache, and x-proxy-request-id to browser code.

Chat Completions

See how the proxy builds the provider request, handles streaming, and forwards tools and structured output.

​Enable caching for a request

​What makes two requests identical

​Limits

​Traces and Analytics

​Browser clients