Response caching - CoreWeave Docs

Exact-response caching lets a project-routed request reuse the response the proxy already produced for an identical request. When a request opts in and a matching response exists, the proxy returns that response without contacting the provider. This saves provider spend and latency for workloads that send the same request more than once, such as evaluation suites, regression tests, retries, and development loops. Caching is off by default and applies per request. A cached response is replayed byte for byte, so a hit returns the same answer even when the request uses sampling parameters such as a nonzero temperature.

Enable caching for a request

Set the wandb-cache-mode request header to one of the following modes:

Value	Read an existing response	Store a successful response
`readWrite`	Yes	Yes
`readOnly`	Yes	No
`writeOnly`	No	Yes

To bypass the cache, omit the header. For compatibility with OpenPipe clients, the proxy also accepts the deprecated op-cache header. It takes the same modes, plus true as an alias for readWrite and false to bypass the cache. If a request sends both headers, they must select the same mode. Otherwise, the proxy returns 400 Bad Request. The following examples send a request in readWrite mode and read the cache result from the response headers.

Python
JavaScript
cURL

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://proxy.training.wandb.ai/v1",
    api_key=os.environ["WANDB_API_KEY"],
)

raw = client.chat.completions.with_raw_response.create(
    model="ticket-classifier",
    messages=[
        {"role": "user", "content": "My package arrived damaged. What should I do?"}
    ],
    extra_body={"metadata": {"wandb.entity": "your-team"}},
    extra_headers={"wandb-cache-mode": "readWrite"},
)

print(raw.headers.get("wandb-cache-status"))  # "hit" or "miss"
response = raw.parse()
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://proxy.training.wandb.ai/v1",
  apiKey: process.env.WANDB_API_KEY,
});

const { data: response, response: raw } = await client.chat.completions
  .create(
    {
      model: "ticket-classifier",
      messages: [
        { role: "user", content: "My package arrived damaged. What should I do?" },
      ],
      metadata: { "wandb.entity": "your-team" },
    },
    { headers: { "wandb-cache-mode": "readWrite" } },
  )
  .withResponse();

console.log(raw.headers.get("wandb-cache-status")); // "hit" or "miss"
console.log(response.choices[0].message.content);

curl -i https://proxy.training.wandb.ai/v1/chat/completions \
  -H "Authorization: Bearer $WANDB_API_KEY" \
  -H "Content-Type: application/json" \
  -H "wandb-cache-mode: readWrite" \
  -d '{
    "model": "ticket-classifier",
    "messages": [
      {"role": "user", "content": "My package arrived damaged. What should I do?"}
    ],
    "metadata": {"wandb.entity": "your-team"}
  }'

When the request enables cache reads (readWrite or readOnly), the response includes a wandb-cache-status header with the value hit or miss. The proxy sends the same value in the x-wandb-cache compatibility header. The response to a writeOnly request doesn’t include either header.

What makes two requests identical

The proxy looks up a cached response after it resolves the project version and routing revision, and before it selects or decrypts a provider. The cache key combines the following values:

The W&B entity, project, and project version.
The routing revision in effect for the request.
The request path and query string.
A canonical hash of the request body fields that affect the output.

The following values don’t change the key:

The model value, so ticket-classifier and ticket-classifier@v1 share entries.
Metadata keys that start with wandb., including wandb.entity and wandb.thread_id.
stream: false, which the proxy treats the same as omitting stream.
The proxy hostname and the order of query parameters.

Everything else in the body does change the key, including messages, tools, response format, sampling parameters, and metadata keys that don’t start with wandb., such as gen_ai.conversation.id or user.id. Publishing a new routing revision changes the key, so existing entries stop matching without being deleted. A cached response also records which routing target produced it. If that target is no longer in the routing configuration, the proxy treats the entry as a miss.

The proxy serves a cache hit from the target that produced the stored response. It doesn’t participate in weighted or sticky routing for that request.

Limits

The following limits apply to which requests can use the cache and which responses the proxy stores.

Project-routed requests only. Direct provider/model requests that set a cache header return 400 Bad Request.
Non-streaming requests only. Requests with stream: true that set a cache header return 400 Bad Request.
Successful responses only. The proxy stores only 2xx responses up to 8 MiB.
Retention is 7 days. Stored responses are deleted 7 days after they’re written. Within that window, a hit doesn’t depend on the entry’s age.
Invalid modes are rejected. Any wandb-cache-mode value other than those listed, or conflicting modes in wandb-cache-mode and op-cache, returns 400 Bad Request.

Cache failures fail open. If the cache can’t be read or written, the proxy forwards the request to the provider as usual.

Traces and Analytics

The proxy still records a cache hit as a project trace. The trace keeps the original response and its token usage, is marked with cache_hit=true, and has no provider latency. A request that enables cache reads but is forwarded to the provider is marked with cache_hit=false. Analytics counts the provider spend of a cache hit as zero, because no inference ran, while token totals still include the replayed usage. Dataset creation deduplicates identical inputs, so replayed responses don’t add duplicate training rows.

Browser clients

The proxy allows the wandb-cache-mode and op-cache request headers in cross-origin requests, and it exposes wandb-cache-status, x-wandb-cache, and x-proxy-request-id to browser code.

Chat Completions

See how the proxy builds the provider request, handles streaming, and forwards tools and structured output.

​Enable caching for a request

​What makes two requests identical

​Limits

​Traces and Analytics

​Browser clients

Chat Completions

Enable caching for a request

What makes two requests identical

Limits

Traces and Analytics

Browser clients