> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Response caching

> Reuse the exact response for a repeated non-streaming request instead of calling the provider again.

Exact-response caching lets a project-routed request reuse the response the proxy already produced for an identical request. When a request opts in and a matching response exists, the proxy returns that response without contacting the provider. This saves provider spend and latency for workloads that send the same request more than once, such as evaluation suites, regression tests, retries, and development loops.

Caching is off by default and applies per request. A cached response is replayed byte for byte, so a hit returns the same answer even when the request uses sampling parameters such as a nonzero temperature.

## Enable caching for a request

Set the `wandb-cache-mode` request header to one of the following modes:

| Value       | Read an existing response | Store a successful response |
| ----------- | ------------------------- | --------------------------- |
| `readWrite` | Yes                       | Yes                         |
| `readOnly`  | Yes                       | No                          |
| `writeOnly` | No                        | Yes                         |

To bypass the cache, omit the header.

For compatibility with OpenPipe clients, the proxy also accepts the deprecated `op-cache` header. It takes the same modes, plus `true` as an alias for `readWrite` and `false` to bypass the cache. If a request sends both headers, they must select the same mode. Otherwise, the proxy returns `400 Bad Request`.

The following examples send a request in `readWrite` mode and read the cache result from the response headers.

<Tabs>
  <Tab title="Python">
    ```python theme={"system"}
    import os

    from openai import OpenAI

    client = OpenAI(
        base_url="https://proxy.training.wandb.ai/v1",
        api_key=os.environ["WANDB_API_KEY"],
    )

    raw = client.chat.completions.with_raw_response.create(
        model="ticket-classifier",
        messages=[
            {"role": "user", "content": "My package arrived damaged. What should I do?"}
        ],
        extra_body={"metadata": {"wandb.entity": "your-team"}},
        extra_headers={"wandb-cache-mode": "readWrite"},
    )

    print(raw.headers.get("wandb-cache-status"))  # "hit" or "miss"
    response = raw.parse()
    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="JavaScript">
    ```javascript theme={"system"}
    import OpenAI from "openai";

    const client = new OpenAI({
      baseURL: "https://proxy.training.wandb.ai/v1",
      apiKey: process.env.WANDB_API_KEY,
    });

    const { data: response, response: raw } = await client.chat.completions
      .create(
        {
          model: "ticket-classifier",
          messages: [
            { role: "user", content: "My package arrived damaged. What should I do?" },
          ],
          metadata: { "wandb.entity": "your-team" },
        },
        { headers: { "wandb-cache-mode": "readWrite" } },
      )
      .withResponse();

    console.log(raw.headers.get("wandb-cache-status")); // "hit" or "miss"
    console.log(response.choices[0].message.content);
    ```
  </Tab>

  <Tab title="cURL">
    ```bash theme={"system"}
    curl -i https://proxy.training.wandb.ai/v1/chat/completions \
      -H "Authorization: Bearer $WANDB_API_KEY" \
      -H "Content-Type: application/json" \
      -H "wandb-cache-mode: readWrite" \
      -d '{
        "model": "ticket-classifier",
        "messages": [
          {"role": "user", "content": "My package arrived damaged. What should I do?"}
        ],
        "metadata": {"wandb.entity": "your-team"}
      }'
    ```
  </Tab>
</Tabs>

When the request enables cache reads (`readWrite` or `readOnly`), the response includes a `wandb-cache-status` header with the value `hit` or `miss`. The proxy sends the same value in the `x-wandb-cache` compatibility header. The response to a `writeOnly` request doesn't include either header.

## What makes two requests identical

The proxy looks up a cached response after it resolves the project version and routing revision, and before it selects or decrypts a provider. The cache key combines the following values:

* The W\&B entity, project, and project version.
* The routing revision in effect for the request.
* The request path and query string.
* A canonical hash of the request body fields that affect the output.

The following values don't change the key:

* The `model` value, so `ticket-classifier` and `ticket-classifier@v1` share entries.
* Metadata keys that start with `wandb.`, including `wandb.entity` and `wandb.thread_id`.
* `stream: false`, which the proxy treats the same as omitting `stream`.
* The proxy hostname and the order of query parameters.

Everything else in the body does change the key, including messages, tools, response format, sampling parameters, and metadata keys that don't start with `wandb.`, such as `gen_ai.conversation.id` or `user.id`.

Publishing a new routing revision changes the key, so existing entries stop matching without being deleted. A cached response also records which routing target produced it. If that target is no longer in the routing configuration, the proxy treats the entry as a miss.

<Note>
  The proxy serves a cache hit from the target that produced the stored response. It doesn't participate in weighted or sticky routing for that request.
</Note>

## Limits

The following limits apply to which requests can use the cache and which responses the proxy stores.

* **Project-routed requests only.** Direct `provider/model` requests that set a cache header return `400 Bad Request`.
* **Non-streaming requests only.** Requests with `stream: true` that set a cache header return `400 Bad Request`.
* **Successful responses only.** The proxy stores only 2xx responses up to 8 MiB.
* **Retention is 7 days.** Stored responses are deleted 7 days after they're written. Within that window, a hit doesn't depend on the entry's age.
* **Invalid modes are rejected.** Any `wandb-cache-mode` value other than those listed, or conflicting modes in `wandb-cache-mode` and `op-cache`, returns `400 Bad Request`.

Cache failures fail open. If the cache can't be read or written, the proxy forwards the request to the provider as usual.

## Traces and Analytics

The proxy still records a cache hit as a project trace. The trace keeps the original response and its token usage, is marked with `cache_hit=true`, and has no provider latency. A request that enables cache reads but is forwarded to the provider is marked with `cache_hit=false`.

Analytics counts the provider spend of a cache hit as zero, because no inference ran, while token totals still include the replayed usage. Dataset creation deduplicates identical inputs, so replayed responses don't add duplicate training rows.

## Browser clients

The proxy allows the `wandb-cache-mode` and `op-cache` request headers in cross-origin requests, and it exposes `wandb-cache-status`, `x-wandb-cache`, and `x-proxy-request-id` to browser code.

<Card title="Chat Completions" href="/model-distillation/proxy/chat-completions" arrow="true">
  See how the proxy builds the provider request, handles streaming, and forwards tools and structured output.
</Card>