Tutorial: Define, initialize, and run a sweep

This tutorial shows how to define, initialize, and run a sweep so you can automate hyperparameter search and find the configuration that produces the best model. It’s intended for users who are already familiar with logging runs to Weights & Biases and want to start tuning hyperparameters at scale. The tutorial has four main steps:

Set up your training code
Define the search space with a sweep configuration
Initialize the sweep
Start the sweep agent

To get started, copy and paste the following code into a Jupyter Notebook or Python script. The sections that follow break down each part of this example.

# Import the W&B Python Library and log into W&B
import wandb

# 1: Define objective/training function
def objective(config):
    score = config.x**3 + config.y
    return score

def main():
    with wandb.init(project="my-first-sweep") as run:
        score = objective(run.config)
        run.log({"score": score})

# 2: Define the search space
sweep_configuration = {
    "method": "random",
    "metric": {"goal": "minimize", "name": "score"},
    "parameters": {
        "x": {"max": 0.1, "min": 0.01},
        "y": {"values": [1, 3, 7]},
    },
}

# 3: Start the sweep
sweep_id = wandb.sweep(sweep=sweep_configuration, project="my-first-sweep")

wandb.agent(sweep_id, function=main, count=10)

Set up your training code

The sweep agent calls your training function with each combination of hyperparameter values to try, so the first step is to write a function that accepts those values and reports a metric back to Weights & Biases. Define a training function that takes in hyperparameter values from wandb.Run.config and uses them to train a model and return metrics. Optionally provide the name of the project where you want to store the output of the run (project parameter in wandb.init()). If you don’t specify a project, Weights & Biases puts the run in an “Uncategorized” project.

Both the sweep and the run must be in the same project. Therefore, the name you provide when you initialize Weights & Biases must match the name of the project you provide when you initialize a sweep.

# 1: Define objective/training function
def objective(config):
    score = config.x**3 + config.y
    return score


def main():
    with wandb.init(project="my-first-sweep") as run:
        score = objective(run.config)
        run.log({"score": score})

Define the search space with a sweep configuration

With the training function in place, the next step is to tell Weights & Biases which hyperparameters to vary and how to search over them. Specify the hyperparameters to sweep in a dictionary. For configuration options, see Define sweep configuration. The following example shows a sweep configuration that uses a random search ('method':'random'). The sweep randomly selects a set of values listed in the configuration for the x and y parameters. Weights & Biases minimizes the metric specified in the metric key when "goal": "minimize" is associated with it. In this case, Weights & Biases optimizes for minimizing the metric score ("name": "score").

# 2: Define the search space
sweep_configuration = {
    "method": "random",
    "metric": {"goal": "minimize", "name": "score"},
    "parameters": {
        "x": {"max": 0.1, "min": 0.01},
        "y": {"values": [1, 3, 7]},
    },
}

Initialize the sweep

Initializing the sweep registers your search space with Weights & Biases and returns an identifier that the agent uses to request hyperparameter combinations. Weights & Biases uses a Sweep Controller to manage sweeps in the cloud (standard) or locally (local) across one or more machines. For more information about Sweep Controllers, see Search and stop algorithms locally. Initializing a sweep returns a sweep identification number:

sweep_id = wandb.sweep(sweep=sweep_configuration, project="my-first-sweep")

For more information, see Initialize sweeps.

Start the sweep

With the sweep registered, start an agent to execute the runs and explore the search space. To start a sweep, use the wandb.agent() API call.

wandb.agent(sweep_id, function=main, count=10)

After the agent starts, it requests hyperparameter combinations from Weights & Biases, calls your training function for each one, and logs the resulting metrics back to your project.

MultiprocessingIf you use the Python standard library’s multiprocessing package or PyTorch’s pytorch.multiprocessing package, you must wrap your wandb.agent() and wandb.sweep() calls with if __name__ == '__main__':. For example:

if __name__ == '__main__':
    wandb.agent(sweep_id="[SWEEP-ID]", function="[FUNCTION]", count="[COUNT]")

This convention ensures the code runs only when the script runs directly, not when imported as a module in a worker process.For more information about multiprocessing, see Python standard library multiprocessing or PyTorch multiprocessing. For more information about the if __name__ == '__main__': convention, see Real Python’s guide to __main__.

Optional: Visualize results

Once the sweep is running, you can explore how different hyperparameter combinations affect your metric in the W&B App. Open your project to see your live results in the W&B App dashboard. In a few clicks, build interactive charts such as parallel coordinates plots, parameter importance analyses, and other chart types.

For more information, see Visualize sweep results. For an example dashboard, see this sample Sweeps Project.

Optional: Stop the agent

In the terminal, press Ctrl+C to stop the current run. Press it again to end the agent.

​Set up your training code

​Define the search space with a sweep configuration

​Initialize the sweep

​Start the sweep

​Optional: Visualize results

​Optional: Stop the agent

Set up your training code

Define the search space with a sweep configuration

Initialize the sweep

Start the sweep

Optional: Visualize results

Optional: Stop the agent