Skip to main content
Routing defines which model serves a project version. Every target contains a model reference, non-negative weight, and optional parameter overrides.

Weights are relative

Weights don’t need to add to 100. A configuration with weights 3 and 1 sends approximately 75% and 25%. A zero-weight target stays available in the configuration but receives no new weighted traffic. Use zero weight to preserve the previous model for visibility and rollback after a full promotion.

Sticky routing

When a request supplies gen_ai.conversation.id in metadata, the proxy deterministically keeps that conversation on one target for the routing revision. Without a conversation ID, the proxy selects a target independently for each request.

Target parameters

To add values such as temperature, reasoning effort, response format, or provider-specific controls, expand Parameters under a model. Numbers, booleans, arrays, and objects are parsed from JSON. Other text remains a string. The stored override wins over the same field supplied by the caller. A model override, which sends a provider/model value in model instead of "router", bypasses the targets. It uses the caller’s parameters and the provider’s defaults, not the targets’ parameters. The proxy traces, samples, and caches it in the selected version like routed traffic.

Save to an existing or new version

Open the project’s Routing page. Add or edit targets, choose the destination in the version bar, and select Save routing. The version bar is part of the same save action:
  • Choose an existing version to replace its desired routing.
  • Choose the new-version option to create a new version. The version is created only when you select Save routing. To call the new version, set forge.project_version in the request metadata to its number, such as "2". Requests without forge.project_version use version 1.
Each save publishes a new routing revision: a numbered configuration for that version that includes its targets, weights, parameter overrides, and trace-sampling rate. Publishing is asynchronous, so a saved configuration serves traffic only after its deployment status becomes Applied at that revision.
If you delete a version, callers pinned to it receive 404 responses. You can’t delete version 1.
Routing updates replace the whole configuration, so the request must include trace_sampling_rate as well as every target. To keep the existing sampling rate, pass it in the request. If you want the previous model available for rollback, keep it at weight 0:
Poll the returned routing state until deployment is applied at the current revision. See Replace routing for a project version.
Last modified on October 26, 2026