RL training

This guide shows how to use serverless sandboxes for reinforcement learning (RL) training workflows where models execute tool calls in isolated environments. It’s intended for ML engineers and researchers building agent training pipelines who need safe, ephemeral environments to evaluate model-generated actions and compute rewards.

Why sandboxes for agent tool execution

Training code agents with RL requires executing tool calls (bash commands, file operations) in isolated environments. Untrusted model-generated code can modify the host filesystem, hit the network, or produce non-deterministic results. Sandboxes give you isolated, ephemeral environments where tool calls run without affecting the host or other rollouts. In a training loop, the model generates actions (tool calls), the sandbox executes them, and observations flow back to the model. The sandbox persists across tool calls within an episode, so file changes and installed packages carry over between steps. Reward comes from the final sandbox state (for example, tests passing) or trajectory quality. The tagging and listing APIs make it practical to clean up and monitor training runs with thousands of sandboxes.

Prerequisites

Before you begin, complete the following setup steps. Set CWSANDBOX_API_KEY to a CoreWeave API access token:

export CWSANDBOX_API_KEY="[API-ACCESS-TOKEN]"

Install the Python SDK:

uv pip install cwsandbox

The examples on this page create sandboxes from SandboxDefaults and use CoreWeave authentication by default. To use a Forge API key, follow the explicit authentication setup in Get started and pass the resulting auth value to each SandboxDefaults(...).

Core pattern

The core setup is an agent loop that runs on your training infrastructure, with tool calls that execute in a sandbox.

import cwsandbox
from cwsandbox import Sandbox

def run_agent_episode(model, task: dict, sandbox: Sandbox) -> tuple[list, float]:
    """Run one agent episode, returning trajectory and reward."""
    messages = [{"role": "user", "content": task["prompt"]}]

    for step in range(task.get("max_steps", 10)):
        # Model generates next action
        response = model.generate(messages)
        messages.append({"role": "assistant", "content": response})

        tool_calls = parse_tool_calls(response)
        if not tool_calls:
            break

        # Execute tool calls in sandbox
        for tool in tool_calls:
            if tool.name == "bash":
                result = sandbox.exec(
                    ["bash", "-c", tool.command],
                    timeout_seconds=30.0,
                ).result()
                observation = f"exit={result.returncode}\n{result.stdout}{result.stderr}"
            elif tool.name == "read_file":
                content = sandbox.read_file(tool.path).result()
                observation = content.decode()
            elif tool.name == "write_file":
                sandbox.write_file(tool.path, tool.content.encode()).result()
                observation = "File written successfully"

            messages.append({"role": "tool", "name": tool.name, "content": observation})

    # Compute reward from final sandbox state
    test_result = sandbox.exec(task["test_command"]).result()
    reward = 1.0 if test_result.returncode == 0 else 0.0

    return messages, reward

The sandbox persists across tool calls within an episode, so file changes accumulate as the agent works.

Training step with parallel episodes

Process a batch of tasks with one sandbox per episode:

def training_step(model, batch: list[dict], session) -> list[float]:
    """Run agent episodes for a batch of tasks."""

    # Create and pre-start all sandboxes in parallel
    sandboxes = [session.sandbox() for _ in batch]
    refs = [sb.start() for sb in sandboxes]
    [r.result() for r in refs]  # Wait for all backends to accept

    trajectories = []
    rewards = []

    for task, sandbox in zip(batch, sandboxes):
        trajectory, reward = run_agent_episode(model, task, sandbox)
        trajectories.append(trajectory)
        rewards.append(reward)
        sandbox.stop()  # Non-blocking cleanup

    # trajectories and rewards go to policy update
    return rewards

Tagging for job metadata

Tags let you filter and find sandboxes created by your training jobs, which becomes essential when a single run can spawn thousands of sandboxes. Include metadata that helps identify sandboxes when debugging or cleaning up:

import os
from cwsandbox import SandboxDefaults

def make_defaults(model_name: str) -> SandboxDefaults:
    return SandboxDefaults(
        container_image="python:3.11",
        tags=(
            f"training-run:{os.environ.get('TRAINING_RUN_ID', 'local')}",
            f"slurm-job:{os.environ.get('SLURM_JOB_ID', 'interactive')}",
            f"model:{model_name}",
            "rl-training",
        ),
    )

Useful metadata to include in tags:

Tag pattern	Purpose
`training-run:{id}`	Training run ID (from `TRAINING_RUN_ID`) for filtering by training run
`slurm-job:{id}`	Slurm job ID (from `SLURM_JOB_ID` env var) for cluster job tracking
`model:{name}`	Model name or checkpoint for multi-model experiments
`env:{name}`	Environment (dev, staging, prod) for resource management

Sandbox tags become Kubernetes pod labels, which the CoreWeave observability platform uses for filtering and dashboards.

Try the reward function example

A minimal integration that computes code execution rewards with parallel sandbox execution. What it does:

Executes a set of toy code completions (arithmetic, string operations, syntax errors, runtime errors).
Creates one sandbox per completion for isolation.
Computes binary rewards: 1.0 for successful execution, 0.0 for failure.
Shows progress as results arrive (faster executions complete first).

How it uses CoreWeave sandboxes: The example uses cwsandbox.wait() to process results as they complete:

# Create sandboxes and execute all completions in parallel
processes = [
    session.sandbox().exec(
        ["python", "-c", code],
        timeout_seconds=EXECUTION_TIMEOUT_SECONDS,
    )
    for code in completions
]

# Collect results as they complete
while pending:
    [process], pending = cwsandbox.wait(pending, num_returns=1)
    result = process.result()
    reward = 1.0 if result.returncode == 0 else 0.0

Run it:

uv run examples/rl_training/reward_function.py

No additional dependencies required. No GPU needed. Expected output: Results arrive as executions complete, so faster problems finish first:

RL Training Reward Function Example (job: cc92116b)
============================================================

Evaluating 5 completions...

Progress (results arrive as executions complete):
------------------------------------------------------------
  [1/5] Problem 1 (string-ops): PASS
  [2/5] Problem 3 (syntax-error): FAIL
  [3/5] Problem 2 (delayed-error): FAIL
  [4/5] Problem 0 (slow-sum): PASS
  [5/5] Problem 4 (slow-list): PASS
------------------------------------------------------------

Final summary (original order):
------------------------------------------------------------
  Problem 0 (slow-sum): reward=1.0 [PASS] OK
  Problem 1 (string-ops): reward=1.0 [PASS] OK
  Problem 2 (delayed-error): reward=0.0 [FAIL] OK
  Problem 3 (syntax-error): reward=0.0 [FAIL] OK
  Problem 4 (slow-list): reward=1.0 [PASS] OK
------------------------------------------------------------
Total reward: 3.0/5
Pass rate: 3/5 (60%)

TRL GRPOTrainer integration

TRL uses a reward function interface where completions map directly to rewards. The agent generates a completion, and the reward function executes it in a sandbox. The standard pattern uses <answer> XML tags for code extraction (matching the format used in GRPO math examples with \boxed{}):

import cwsandbox
from cwsandbox import SandboxDefaults

session = cwsandbox.Session(defaults=SandboxDefaults(
    container_image="python:3.11",
    tags=("trl-grpo",),
))

def extract_xml_answer(text: str) -> str:
    if "<answer>" not in text:
        return ""
    return text.split("<answer>")[-1].split("</answer>")[0].strip()

def reward_fn(completions, **kwargs) -> list[float]:
    # Conversational datasets pass message dicts, standard datasets pass strings
    texts = [c[0]["content"] if isinstance(c, list) else c for c in completions]
    codes = [extract_xml_answer(t) for t in texts]
    code_indices = [(i, code) for i, code in enumerate(codes) if code]

    processes = [
        (i, session.sandbox().exec(
            ["python", "-c", code],
            timeout_seconds=30.0,
        ))
        for i, code in code_indices
    ]

    rewards = [0.0] * len(codes)
    for i, process in processes:
        try:
            rewards[i] = 1.0 if process.result().returncode == 0 else 0.0
        except Exception:
            pass

    return rewards

This pattern works for training models to generate correct code in a single turn.

Try the TRL GRPO integration example

Uses CoreWeave sandboxes with TRL’s GRPOTrainer for code execution rewards. What it does:

Loads a small model (Qwen/Qwen2.5-0.5B-Instruct).
Creates a toy dataset of simple coding problems.
Trains the model using GRPO with sandbox-based reward computation.
Runs 10 training steps to demonstrate the integration.

How it uses CoreWeave sandboxes: The reward function extracts code from <answer> tags (the standard GRPO pattern), creates sandboxes in parallel through a Session, executes each completion, and returns binary rewards:

def extract_xml_answer(text: str) -> str:
    """Extract answer from XML-style <answer> tags."""
    if "<answer>" not in text:
        return ""
    answer = text.split("<answer>")[-1]
    answer = answer.split("</answer>")[0]
    return answer.strip()

def code_execution_reward(completions, **kwargs) -> list[float]:
    # Conversational datasets pass message dicts, standard datasets pass strings
    texts = [c[0]["content"] if isinstance(c, list) else c for c in completions]
    codes = [extract_xml_answer(t) for t in texts]

    # Create sandboxes and execute non-empty code in parallel
    processes = [
        (i, session.sandbox().exec(
            ["python", "-c", code],
            timeout_seconds=EXECUTION_TIMEOUT_SECONDS,
        ))
        for i, code in enumerate(codes) if code
    ]

    # Collect rewards, defaulting to 0.0
    rewards = [0.0] * len(codes)
    for i, process in processes:
        try:
            result = process.result()
            rewards[i] = 1.0 if result.returncode == 0 else 0.0
        except Exception:
            rewards[i] = 0.0

    return rewards

The prompts use a system message instructing the model to format code with <answer> tags:

SYSTEM_PROMPT = """You solve coding problems by writing Python code.
Put your code inside <answer> tags like this: <answer>print("hello")</answer>
Only include the code, no explanations."""

The Session tracks sandboxes and cleans them up when it closes. Run it:

uv pip install trl==0.27.1 transformers==5.0.0 datasets==4.5.0 torch==2.10.0
uv run examples/rl_training/trl_grpo_integration.py

GPU is recommended for reasonable performance. Without one, training works but is slow. Expected output:

TRL GRPO Integration Example (job: def67890)
============================================================

Loading model: Qwen/Qwen2.5-0.5B-Instruct
Creating toy dataset...
Dataset size: 5 problems

Setting up GRPOTrainer...

Starting training (10 steps)...
------------------------------------------------------------
  [CWSandbox] Reward call 1: 2 sandboxes, 0/2 passed
  [CWSandbox] Reward call 2: 1 sandboxes, 0/1 passed, 1 skipped (no code)
  [CWSandbox] Reward call 3: 0 sandboxes, 0/0 passed, 2 skipped (no code)
  [CWSandbox] Reward call 4: 2 sandboxes, 0/2 passed
  ...
  [CWSandbox] Reward call 8: 2 sandboxes, 1/2 passed
  [CWSandbox] Reward call 9: 2 sandboxes, 1/2 passed
  [CWSandbox] Reward call 10: 1 sandboxes, 0/1 passed, 1 skipped (no code)
[training logs]
------------------------------------------------------------

Training completed successfully!

Understanding the output: The number of sandboxes varies per step because the script only creates sandboxes when extract_xml_answer() finds extractable code in the model’s completion. When the model generates text without the expected <answer>...</answer> tags, that completion is skipped and receives a reward of 0.0.

2 sandboxes, 0/2 passed: Model generated 2 code blocks, both failed execution.
1 sandboxes, 0/1 passed, 1 skipped (no code): Model generated 1 code block (failed) and 1 text-only completion.
0 sandboxes, 0/0 passed, 2 skipped (no code): Model generated no extractable code in either completion.

Expected with a small, untrained model. As training progresses, you should see fewer skipped completions and more passes.

Error handling in agent episodes

Sandbox operations can fail (timeouts, missing files, sandbox termination), and how you surface those failures shapes what the agent learns. Return observations that describe what went wrong:

from cwsandbox import SandboxTimeoutError, SandboxFileError

def execute_tool(sandbox, tool) -> str:
    """Execute a tool call, returning an observation string."""
    try:
        if tool.name == "bash":
            result = sandbox.exec(
                ["bash", "-c", tool.command],
                timeout_seconds=30.0,
            ).result()
            return f"exit={result.returncode}\n{result.stdout}{result.stderr}"
        elif tool.name == "read_file":
            content = sandbox.read_file(tool.path).result()
            return content.decode()
        elif tool.name == "write_file":
            sandbox.write_file(tool.path, tool.content.encode()).result()
            return "File written successfully"
    except SandboxTimeoutError:
        return "Error: command timed out after 30 seconds"
    except SandboxFileError as e:
        return f"Error: {e}"
    except Exception as e:
        return f"Error: {type(e).__name__}: {e}"

For reward computation, catch exceptions and return a fallback reward instead of propagating to the training loop.

Weights & Biases metrics integration

When you use Weights & Biases for training, cwsandbox Sessions log sandbox usage metrics to your active wandb run automatically. This lets you correlate sandbox health (startup time, execution failures, tool calls per rollout) with your training metrics in the same dashboard. The following sections describe how auto-detection works, how to control reporting explicitly, and which metrics are tracked.

Auto-detection

If the wandb package is installed and a Weights & Biases run is active (wandb.run exists), metrics logging is enabled automatically. Install the optional package and set a Forge API key for experiment tracking:

uv pip install wandb
export FORGE_API_KEY="[FORGE-API-KEY]"

Pass the key explicitly to wandb.login(). FORGE_API_KEY is read by this example; it isn’t automatically detected by the SDK. Sandbox authentication continues to use CWSANDBOX_API_KEY:

import os

import wandb
from cwsandbox import Session, SandboxDefaults

wandb.login(key=os.environ["FORGE_API_KEY"])
wandb.init(project="my-rl-training")
defaults = SandboxDefaults(container_image="python:3.11")
code = "print('Training tool call')"

# Metrics logging enabled automatically
with Session(defaults) as session:
    for step in range(3):
        sandbox = session.sandbox()
        # Exec results are automatically tracked - no manual calls needed
        result = sandbox.exec(["python", "-c", code]).result()
        # Log metrics at this training step for correlation
        session.log_metrics(step=step)
# Final metrics logged on session close

Explicit control

Control metrics reporting with the report_to parameter:

# Explicit opt-in (creates reporter; metrics logged when wandb.run exists)
session = Session(defaults, report_to=["wandb"])

# Disable reporting (even if wandb run exists)
session = Session(defaults, report_to=[])

# Auto-detect (default behavior)
session = Session(defaults, report_to=None)

Metrics

Execution metrics are tracked automatically when exec() completes:

Metric	Description
`cwsandbox/sandboxes_created`	Total sandboxes created through session
`cwsandbox/executions`	Total exec() calls
`cwsandbox/exec_completed_ok`	Completed executions (returncode=0)
`cwsandbox/exec_completed_nonzero`	Completed executions (returncode!=0)
`cwsandbox/exec_failures`	Failed executions (timeouts, transport failures)
`cwsandbox/exec_completion_rate`	Fraction of exec() that completed with returncode=0
`cwsandbox/exec_failure_rate`	Fraction of exec() that failed to complete
`cwsandbox/startup_count`	Number of sandbox startup times recorded
`cwsandbox/avg_startup_seconds`	Average sandbox startup time
`cwsandbox/min_startup_seconds`	Minimum sandbox startup time
`cwsandbox/max_startup_seconds`	Maximum sandbox startup time

Tracking is automatic. Call exec() on any sandbox associated with a session. Call session.log_metrics(step=N) to log at specific training steps:

def training_step(session, model, batch, step: int) -> list[float]:
    rewards = []
    for task in batch:
        sandbox = session.sandbox()
        # Metrics tracked automatically on exec() completion
        result = sandbox.exec(["python", "-c", task["code"]]).result()
        reward = 1.0 if result.returncode == 0 else 0.0
        rewards.append(reward)
        sandbox.stop()

    # Log metrics at this training step for correlation
    session.log_metrics(step=step)
    return rewards

You can also access per-sandbox statistics through the exec_stats property:

sandbox = session.sandbox()
result = sandbox.exec(["echo", "hello"]).result()
print(sandbox.exec_stats)  # {"exec_count": 1, "completed_ok": 1, "completed_nonzero": 0, "failures": 0}

Per-sandbox exec metrics

Sessions with Weights & Biases integration also track per-sandbox metrics:

Metric	Description
`cwsandbox/avg_execs_per_sandbox`	Average exec() calls per sandbox (useful for “tool calls per rollout”)
`cwsandbox/min_execs_per_sandbox`	Minimum exec() calls in any sandbox
`cwsandbox/max_execs_per_sandbox`	Maximum exec() calls in any sandbox

What these tell you about agent behavior:

High avg_execs_per_sandbox may indicate verbose agents that make many tool calls per episode.
Large variance (max-min) may indicate inconsistent rollout behavior across episodes.
Trends over training steps show how agent behavior evolves as the policy improves.

Example dashboard usage:

Plot avg_execs_per_sandbox versus training step to see tool usage trends over training.
Alert if max_execs_per_sandbox exceeds a threshold (runaway agent making excessive tool calls).
Compare min/max spread to detect episodes where agents get stuck in loops versus complete quickly.

By default, log_metrics() resets the counters after logging. Set reset=False to keep accumulating:

session.log_metrics(step=step, reset=False)  # Keep accumulating

Metrics are also logged automatically when the session closes, so you get final summary metrics even without explicit logging.

Monitoring and debugging

The following patterns help you observe what your training run is doing in real time and capture details that make failures easier to diagnose.

Counting active sandboxes

Monitor sandbox usage during training:

from cwsandbox import Sandbox, SandboxStatus

def count_active_sandboxes(run_id: str) -> dict:
    sandboxes = Sandbox.list(
        tags=[f"training-run:{run_id}"],
        status=[SandboxStatus.RUNNING, SandboxStatus.PENDING],
    ).result()

    return {
        "running": sum(1 for s in sandboxes if s.status == SandboxStatus.RUNNING),
        "pending": sum(1 for s in sandboxes if s.status == SandboxStatus.PENDING),
        "total": len(sandboxes),
    }

Logging execution details

Capture execution details for debugging reward computation:

import logging

logger = logging.getLogger(__name__)

# Assumes `session` is created at module level

def logged_reward(completion: str, step: int) -> float:
    code = extract_xml_answer(completion)  # Extract code from <answer> tags

    sandbox = session.sandbox()
    sandbox.wait()
    result = sandbox.exec(
        ["python", "-c", code],
        timeout_seconds=30.0,
    ).result()

    logger.debug(
        "Reward computation",
        extra={
            "step": step,
            "sandbox_id": sandbox.sandbox_id,
            "returncode": result.returncode,
            "stdout_len": len(result.stdout),
            "stderr_len": len(result.stderr),
        },
    )

    sandbox.stop()
    return 1.0 if result.returncode == 0 else 0.0

Multi-step rollouts with ART

The preceding TRL example uses sandboxes for single-shot execution: one sandbox per completion, execute once, return a reward. This works for training models to generate correct code in one attempt. Stateful multi-step rollouts are different: the agent takes multiple actions within a single sandbox, and the sandbox maintains state between actions. The agent can write a file, run it, observe the error, edit the file, and try again, all within the same sandbox. The examples/rl_training/art/ directory demonstrates this pattern on the MBPP benchmark. When a solution fails, the agent receives error feedback and can iterate on its approach.

Overview

The remainder of this section walks through what ART is, the training approach the example uses, prerequisites, installation, how to run the example, and how the components fit together. ART (Agent Reinforcement Trainer) is an open source RL framework by OpenPipe for training multi-step agents using GRPO. This example integrates CoreWeave sandboxes with ART:

Uses the art package (openpipe-art) for trajectory collection and training.
Supports two backends: LocalBackend (requires GPU) or TinkerBackend (no GPU).
Executes code through tool calling in CoreWeave sandbox.
Computes binary rewards based on MBPP test case results.

Training approach with GRPO and distillation

This example uses distillation with reinforcement learning: a stronger model generates demonstrations, and a smaller model learns to replicate the successful ones. Two models are involved:

Inference model (--model, default: gpt-5.1-codex-mini): Generates trajectories during rollouts. This model makes tool calls, sees sandbox results, iterates on errors, and submits solutions. It does not get trained.
Base model (--base-model, default: Qwen/Qwen3-8B): The model being trained. It receives the trajectories generated by the inference model and learns from them through GRPO (Group Relative Policy Optimization).

How it works:

The inference model generates multiple trajectories per problem, each with tool calls executed in a CoreWeave sandbox.
Each trajectory receives a binary reward: 1.0 if tests pass, 0.0 otherwise.
Trajectories for the same problem form a group, and GRPO compares trajectories within each group.
The base model (Qwen3-8B) is trained to prefer higher-reward trajectories over lower-reward ones.

After training, you deploy Qwen3-8B with the same tool definitions. It has learned to make similar tool calls by imitating the successful trajectories from the inference model.

Prerequisites

Mode	Requirements
Dry run	CPU only
TinkerBackend	CPU only (training through API)
LocalBackend	GPU required

Environment variables:

export OPENAI_API_KEY="your-openai-key"
export ART_TINKER_API_KEY="your-tinker-key"  # required for --backend=tinker
export CWSANDBOX_API_KEY="[API-ACCESS-TOKEN]"  # sandbox authentication

For optional Weights & Biases logging, set a Forge API key:

export FORGE_API_KEY="[FORGE-API-KEY]"

ART 0.5.7 checks the legacy WANDB_API_KEY variable before enabling its logging integration. Add this compatibility mapping to train.py before it creates the backend. The key is a Forge API key:

import os

if api_key := os.environ.get("FORGE_API_KEY"):
    os.environ["WANDB_API_KEY"] = api_key

Installation

uv pip install -r examples/rl_training/art/requirements.txt

This installs:

openpipe-art==0.5.7       # ART framework
openai==2.15.0            # LLM inference
datasets==4.5.0           # MBPP loading
wandb==0.24.0             # Optional logging

For LocalBackend with GPU support, also install:

uv pip install "openpipe-art[backend]==0.5.7"

Running the example

# Dry run - validate setup without training
uv run examples/rl_training/art/train.py --dry-run

# Train with TinkerBackend (no local GPU required)
uv run examples/rl_training/art/train.py --backend tinker --num-problems 10

# Train with LocalBackend (requires GPU)
uv run examples/rl_training/art/train.py --backend local --num-problems 10

Expected output:

ART Training with CWSandbox
========================================
Backend: tinker
Model: gpt-5.1-codex-mini
Base model: Qwen/Qwen3-8B
Problems: 10
Steps: 5
Trajectories per problem: 2
Project: cwsandbox-mbpp
Run name: train-001

Loading MBPP problems...
Loaded 10 problems

Creating tinker backend...
Creating trainable model...
Registering model with backend...

Starting training...

=== Step 1 ===
Collecting trajectories for 10 problems...
step 1: 100%|██████████| 10/10 [00:45<00:00]
Collected 20 trajectories, avg reward: 0.35
Training...
Training complete: step=1, metrics={'loss': 0.42}
...

Configuration options

Flag	Default	Description
`--backend`	`local`	Training backend: `local` (GPU) or `tinker` (no GPU)
`--model`	`gpt-5.1-codex-mini`	Model for inference
`--base-model`	`Qwen/Qwen3-8B`	Base model for training
`--num-problems`	`10`	Number of MBPP problems
`--num-steps`	`5`	Training steps
`--trajectories-per-problem`	`2`	Trajectories collected per problem per step
`--base-url`	`None`	OpenAI-compatible API base URL
`--project`	`cwsandbox-mbpp`	Weights & Biases project name
`--run-name`	`train-001`	Training run name
`--learning-rate`	`1e-5`	Learning rate
`--dry-run`	`false`	Validate setup without training

Architecture

art/
├── train.py         # Training loop, CLI, and ART backend setup
├── rollout.py       # Multi-step sandbox execution, builds Trajectory
├── tools.py         # Tool schemas for execute_code and submit_solution
└── __init__.py

Key ART imports:

import art
from art.local import LocalBackend
from art.tinker import TinkerBackend

# Create trainable model
model = art.TrainableModel(
    name="train-001",
    project="cwsandbox-mbpp",
    base_model="Qwen/Qwen3-8B",
)

# Collect trajectories
groups = await art.gather_trajectory_groups(
    (collect_trajectories(problem) for problem in problems),
    pbar_desc="collecting",
)

# Train
result = await backend.train(model, groups, learning_rate=1e-5)

Rollout returns art.Trajectory:

from openai.types.chat import ChatCompletionToolParam

trajectory = art.Trajectory(
    messages_and_choices=messages_and_choices,  # Conversation history
    tools=ROLLOUT_TOOLS,                        # Tool definitions
    reward=1.0 if passed else 0.0,              # Binary reward
    metadata={"task_id": problem.task_id},
)
return trajectory.finish()

Tool-calling pattern: The rollout uses OpenAI-compatible tool calling with two tools:

execute_code: Test code in sandbox, returns stdout/stderr.
submit_solution: Final submission, runs all test cases.

ROLLOUT_TOOLS: list[ChatCompletionToolParam] = [
    {
        "type": "function",
        "function": {
            "name": "execute_code",
            "description": "Execute Python code in an isolated sandbox...",
            "parameters": {
                "type": "object",
                "properties": {"code": {"type": "string"}},
                "required": ["code"],
            },
        },
    },
    # submit_solution tool...
]

Sandboxes are tagged for tracking:

sandbox_defaults = SandboxDefaults(
    container_image="python:3.11",
    tags=("art-training", args.project, args.run_name),
)

Data flow

The training pipeline has several components:

Local machine: The training script (train.py) runs on your machine or training server. It loads problems from the MBPP dataset and orchestrates the training loop.
OpenAI API (inference): During trajectory collection, the rollout code calls the OpenAI API (or compatible endpoint) to generate model responses. The model receives tool definitions and returns tool calls that the rollout executes.
CoreWeave sandbox (code execution): Each rollout uses a single CoreWeave sandbox that persists across all tool calls. When the model calls execute_code or submit_solution, the code runs in that sandbox. This means file changes and state accumulate as the agent iterates: it can write a file, run it, observe an error, and fix it. The sandbox provides isolation so untrusted model-generated code can’t affect the host. Results (stdout, stderr, exit code) flow back to the rollout.
Trajectory collection: The rollout accumulates the conversation history (messages and tool results) along with the final reward into an art.Trajectory object. Multiple trajectories for the same problem form an art.TrajectoryGroup.
Training backend: The collected trajectory groups are sent to the training backend. With LocalBackend, training happens on your local GPU. With TinkerBackend, trajectories are uploaded to Thinking Machines’s Tinker service, which handles training remotely. No local GPU is required.

Sandboxes

Use sandboxes

Manage sandboxes

Reference

Why sandboxes for agent tool execution

Prerequisites

Core pattern

Training step with parallel episodes

Tagging for job metadata

Try the reward function example

TRL GRPOTrainer integration

Try the TRL GRPO integration example

Error handling in agent episodes

Weights & Biases metrics integration

Auto-detection

Explicit control

Metrics

Per-sandbox exec metrics

Monitoring and debugging

Counting active sandboxes

Logging execution details

Multi-step rollouts with ART

Overview

Training approach with GRPO and distillation

Prerequisites

Installation

Running the example

Configuration options

Architecture

Data flow

​Why sandboxes for agent tool execution

​Prerequisites

​Core pattern

​Training step with parallel episodes

​Tagging for job metadata

​Try the reward function example

​TRL GRPOTrainer integration

​Try the TRL GRPO integration example

​Error handling in agent episodes

​Weights & Biases metrics integration

​Auto-detection

​Explicit control

​Metrics

​Per-sandbox exec metrics

​Monitoring and debugging

​Counting active sandboxes

​Logging execution details

​Multi-step rollouts with ART

​Overview

​Training approach with GRPO and distillation

​Prerequisites

​Installation

​Running the example

​Configuration options

​Architecture

​Data flow

Why sandboxes for agent tool execution

Prerequisites

Core pattern

Training step with parallel episodes

Tagging for job metadata

Try the reward function example

TRL GRPOTrainer integration

Try the TRL GRPO integration example

Error handling in agent episodes

Weights & Biases metrics integration

Auto-detection

Explicit control

Metrics

Per-sandbox exec metrics

Monitoring and debugging

Counting active sandboxes

Logging execution details

Multi-step rollouts with ART

Overview

Training approach with GRPO and distillation

Prerequisites

Installation

Running the example

Configuration options

Architecture

Data flow