Skip to main content
The proxy accepts a JSON Chat Completions body. messages is required. With the OpenAI-Project header, model is router, blank, or omitted for weighted routing, or provider/model-id for a model override. The proxy forwards other fields when the selected provider supports them.

Streaming

Set stream: true as usual. The provider’s server-sent events are forwarded as they arrive, including [DONE]. For traced streams, the proxy may request stream usage internally so token counts are available; this does not change the client-visible message content. Streaming requests can’t use response caching.

Tools and structured output

Tool definitions, tool choices, response-format JSON schemas, chat-template controls, log probabilities, multimodal content, provider extensions, and reasoning fields are preserved in the raw trace contract. Unknown request fields are forwarded unless routing replaces or removes them.

Precedence

For a weighted project route, the proxy builds the effective provider request in the following order. When the request enables response caching, the proxy checks the cache after it resolves the project version and before it selects a target. On a cache hit, it skips these steps.
  1. Start with the client body.
  2. Replace model with the selected target’s provider model.
  3. Merge the target’s params_override values, which win on conflict.
  4. Remove forge.* and wandb.* metadata before provider dispatch.
  5. Add trace-supporting stream options when needed.
A model override skips target selection and target parameters. The proxy replaces model with the requested provider’s model ID and keeps the caller’s parameters and the provider’s inference defaults. It doesn’t apply a fine-tuned model’s inference settings, such as its default temperature, or the targets’ params_override values. To use that temperature with an override, send temperature in the request. The proxy doesn’t forward client headers other than those it constructs itself. Cookies, provider API keys, organization headers, and unrelated authorization headers can’t leak to the selected provider. The one exception is the wandb-inference provider, which receives the validated OpenAI-Project value for attribution.

Vapi requests

The proxy recognizes Vapi’s request wrapper, removes Vapi-only fields before forwarding, and uses call.id as a conversation fallback. An explicit gen_ai.conversation.id takes precedence.
Last modified on October 26, 2026