Response caching - CoreWeave Docs

Exact-response caching lets a project-routed request reuse the response the proxy already produced for an identical request. When a request opts in and a matching response exists, the proxy returns that response without contacting the provider. This saves provider spend and latency for workloads that send the same request more than once, such as evaluation suites, regression tests, retries, and development loops. Caching is off by default and applies per request. A cached response is replayed byte for byte, so a hit returns the same answer even when the request uses sampling parameters such as a nonzero temperature.

Enable caching for a request

Set the Forge-Cache-Mode request header to one of the following modes:

Value	Read an existing response	Store a successful response
`readWrite`	Yes	Yes
`readOnly`	Yes	No
`writeOnly`	No	Yes

To bypass the cache, omit the header.

Python
JavaScript
cURL

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://proxy.training.coreweave.com/v1",
    api_key=os.environ["WANDB_API_KEY"],
    project="your-team/ticket-classifier",  # sent as OpenAI-Project
)

raw = client.chat.completions.with_raw_response.create(
    model="router",
    messages=[
        {"role": "user", "content": "My package arrived damaged. What should I do?"}
    ],
    extra_headers={"Forge-Cache-Mode": "readWrite"},
)

print(raw.headers.get("forge-cache-status"))  # "hit" or "miss"
response = raw.parse()
print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://proxy.training.coreweave.com/v1",
  apiKey: process.env.WANDB_API_KEY,
  project: "your-team/ticket-classifier", // sent as OpenAI-Project
});

const { data: response, response: raw } = await client.chat.completions
  .create(
    {
      model: "router",
      messages: [
        { role: "user", content: "My package arrived damaged. What should I do?" },
      ],
    },
    { headers: { "Forge-Cache-Mode": "readWrite" } },
  )
  .withResponse();

console.log(raw.headers.get("forge-cache-status")); // "hit" or "miss"
console.log(response.choices[0].message.content);

curl -i https://proxy.training.coreweave.com/v1/chat/completions \
  -H "Authorization: Bearer $WANDB_API_KEY" \
  -H "Content-Type: application/json" \
  -H "OpenAI-Project: your-team/ticket-classifier" \
  -H "Forge-Cache-Mode: readWrite" \
  -d '{
    "model": "router",
    "messages": [
      {"role": "user", "content": "My package arrived damaged. What should I do?"}
    ]
  }'

When the request enables cache reads (readWrite or readOnly), the response includes a Forge-Cache-Status header with the value hit or miss. The response to a writeOnly request doesn’t include it.

What makes two requests identical

The proxy looks up a cached response after it resolves the project version and routing revision, and before it selects a provider. For a model override, the lookup happens after the proxy resolves the override’s provider. The cache key combines the following values:

The Weights & Biases entity, project, and project version that the proxy resolves from the OpenAI-Project header and forge.project_version metadata.
The routing revision that applies to the version when the request arrives.
The request path and query string.
A SHA-256 hash of the request body, serialized with object keys in sorted order and with the following values excluded.

The proxy removes the following values from the body before hashing, so they don’t change the key:

The raw model string of a routed request. Its resolved project, version, and routing revision are already in the key, so "router", a blank model, and an omitted model share entries. Different versions never share entries.
Metadata keys that start with forge. or wandb., including forge.project_version.
The conversation identifier in gen_ai.conversation.id.
stream: false, which the proxy treats the same as omitting stream.
An empty metadata object, which the proxy treats the same as omitting metadata.

The proxy hostname and the order of query parameters don’t change the key either. Each time you save a version’s routing, Model Distillation publishes a new routing revision, which changes the key. Existing entries stop matching but remain until they expire. The proxy doesn’t hash parameter overrides on a routing target. Changing an override is a routing change like any other, so it also changes the key. Every other field in the body is part of the hash, regardless of whether the provider uses it. These fields include messages, tools, tool_choice, response_format, temperature, top_p, max_tokens, seed, n, stop, user, any provider-specific fields, and metadata keys that don’t start with forge. or wandb., such as user.id. The order of keys within an object doesn’t matter. The order of items in an array, such as messages, does. Entries are scoped to your Weights & Biases entity and project. An identical request from another entity, or for another project in your entity, never matches your entries. A cached response records the routing target that produced it. The proxy serves a hit from that target without weighted or sticky routing. A model override sends a provider/model value in model together with the OpenAI-Project header. The proxy caches it in the selected project version like routed traffic. Its key also includes the provider, the model reference, and the provider’s configuration, so an override never matches routed entries or entries for another model.

Limits

The cache has the following limits:

Non-streaming requests only. Requests with stream: true that set a cache header return 400 Bad Request.
Successful responses only. The proxy stores only 2xx responses up to 8 MiB.
Retention is 7 days. Entries are deleted 7 days after they’re written. Hits don’t refresh them.
Invalid modes are rejected. Any Forge-Cache-Mode value other than those listed returns 400 Bad Request.

Cache failures fail open. If the cache can’t be read or written, the proxy forwards the request to the provider as usual. The proxy doesn’t coalesce concurrent requests. If identical requests arrive before the proxy stores the first response, each one reaches the provider, and the proxy stores the last response written.

Traces and Analytics

The proxy still records a cache hit as a project trace. The trace keeps the original response and its token usage, is marked with cache_hit=true, and has no provider latency. A request that enables cache reads but is forwarded to the provider is marked with cache_hit=false. Analytics counts the provider spend of a cache hit as zero, because no inference ran, while token totals still include the replayed usage. Dataset creation deduplicates identical inputs, so replayed responses don’t add duplicate training rows.

Browser clients

The proxy allows the OpenAI-Project and Forge-Cache-Mode request headers in cross-origin requests, and it exposes Forge-Cache-Status and x-proxy-request-id to browser code.

Chat Completions

See how the proxy builds the provider request, handles streaming, and forwards tools and structured output.

​Enable caching for a request

​What makes two requests identical

​Limits

​Traces and Analytics

​Browser clients