Prefix caching

Serverless Inference uses prefix caching on supported hosted models to speed up repeated requests with identical prompt prefixes. When a request shares the same prompt prefix as an earlier request on the same backend, the model reuses the previously computed key-value (KV) cache instead of recomputing the entire prefix. This reduces latency for repeated prompts, long system prompts, and workloads with a stable shared prefix. Prefix caching is automatic on supported models, so you don’t need to enable it in your request. This page describes when prefix caching is most effective and how to control cache isolation with cache_salt.

When prefix caching helps

Prefix caching is most useful when you repeatedly send requests that share a long common prefix, such as:

A large system prompt reused across many requests.
A long shared document followed by different user questions.
Repeated evaluation prompts with only small per-request changes.
Multi-turn workloads where much of the conversation history stays the same.

Cache isolation

In some environments, you might need to prevent cache reuse across different users or applications. The cache_salt parameter provides this control. By default, requests with identical prompt prefixes may reuse cache on shared infrastructure when the backend allows it. To isolate cache reuse to a specific trust boundary, set the cache_salt request parameter. Requests only reuse prefix cache when both the prompt prefix and the cache_salt match. Use cache_salt when you want cache reuse within a single user, tenant, session, or application boundary, but don’t want reuse across other callers.

How it works

The presence and value of cache_salt affect cache reuse as follows:

Same prompt prefix, no cache_salt: cache may be reused across matching requests.
Same prompt prefix, same cache_salt: cache can be reused.
Same prompt prefix, different cache_salt: cache is isolated and not reused across salts.

cache_salt must be a non-empty string when provided.

Examples

The following examples show how to send a chat completion request with cache_salt to isolate cache reuse.

Python
Bash

import openai

client = openai.OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key="<your-api-key>",
)

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.5",
    messages=[
        {
            "role": "system",
            "content": "You are a careful assistant that answers concisely."
        },
        {
            "role": "user",
            "content": "Summarize this document in one sentence: <long shared prefix here>"
        },
    ],
    extra_body={
        "cache_salt": "tenant-a-user-123-secret",
    },
)

print(response.choices[0].message.content)

curl https://api.inference.wandb.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-api-key>" \
  -d '{
    "model": "moonshotai/Kimi-K2.5",
    "messages": [
      { "role": "system", "content": "You are a careful assistant that answers concisely." },
      { "role": "user", "content": "Summarize this document in one sentence: <long shared prefix here>" }
    ],
    "cache_salt": "tenant-a-user-123-secret"
  }'

Response behavior

To confirm that prefix caching is active for a request, check the response usage details. On some models, usage details may include cached token counts in usage.prompt_tokens_details.cached_tokens when prefix cache is reused. The availability of that field varies by model and backend. The following pages cover related topics:

Response Settings

Tutorials

API Reference

When prefix caching helps

Cache isolation

How it works

Examples

Response behavior

​When prefix caching helps

​Cache isolation

​How it works

​Examples

​Response behavior

​Related pages

When prefix caching helps

Cache isolation

How it works

Examples

Response behavior

Related pages