Hugging Face Accelerate - CoreWeave Docs

Hugging Face Accelerate is a library that enables the same PyTorch code to run across any distributed configuration, to simplify model training and inference at scale. Accelerate includes a W&B Tracker, which this page shows how to use to log metrics, configuration, and artifacts from distributed training runs to Weights & Biases. For more information, see Accelerate Trackers in Hugging Face.

Start logging with Accelerate

This section shows how to configure Accelerate to log experiment data to Weights & Biases during training. To get started with Accelerate and Weights & Biases, follow this pseudocode:

from accelerate import Accelerator

# Tell the Accelerator object to log with wandb
accelerator = Accelerator(log_with="wandb")

# Initialise your wandb run, passing wandb parameters and any config information
accelerator.init_trackers(
    project_name="my_project", 
    config={"dropout": 0.1, "learning_rate": 1e-2},
    init_kwargs={"wandb": {"entity": "my-wandb-team"}}
    )

...

# Log to wandb by calling accelerator.log(); step is optional
accelerator.log({"train_loss": 1.12, "valid_loss": 0.8}, step=global_step)


# Make sure that the wandb tracker finishes correctly
accelerator.end_training()

In more detail:

Pass log_with="wandb" when you initialize the Accelerator class.
Call the init_trackers method and pass it:
- A project name through project_name.
- Any parameters you want to pass to wandb.init() through a nested dict to init_kwargs.
- Any other experiment config information you want to log to your wandb run, through config.
Use the wandb.Run.log() method to log to Weights & Biases. The step argument is optional.
Call .end_training() when training finishes.

Access the W&B tracker

Once Accelerate logs to Weights & Biases, you may want direct access to the underlying W&B run object to log artifacts, custom charts, or other data that the tracker doesn’t expose. To access the W&B tracker, use the Accelerator.get_tracker() method. Pass in the string corresponding to a tracker’s .name attribute, which returns the tracker on the main process.

wandb_tracker = accelerator.get_tracker("wandb")

From there, you can interact with the wandb run object as usual:

wandb_tracker.log_artifact(some_artifact_to_log)

Trackers built in Accelerate automatically execute on the correct process, so if a tracker only needs to run on the main process it does so automatically.To remove Accelerate’s wrapping entirely, you can achieve the same outcome with:

wandb_tracker = accelerator.get_tracker("wandb", unwrap=True)
with accelerator.on_main_process:
    wandb_tracker.log_artifact(some_artifact_to_log)

Accelerate articles

For a deeper walkthrough of using Accelerate with Weights & Biases, see the following article.

​Start logging with Accelerate

​Access the W&B tracker

​Accelerate articles

Start logging with Accelerate

Access the W&B tracker

Accelerate articles