Shell for Machine Learning on Determined AI

Use a shell for freeform commands with Determined AI

Determined AI supports the use of commands and shells in order to engage with Determined clusters and cluster GPUs without needing to use the trial APIs or for running existing code in batch mode.

On Determined AI, shells start SSH servers, which in turn provide access to the cluster and its resources in the form of interactive SSH sessions.

Additional Resources

For more information, refer to the official Determined AI documentation regarding the CLI tool det and shells on Determined AI.

Setup

Before continuing with the rest of this guide, first install and configure the Determined AI application via the Cloud UI into your namespace.

Next, create a Python virtual environment using a tool such as venv or virtualenv on your local environment. Then, install the Determined AI CLI tool using pip:

$ pip install determined==0.19.8

Create a shell

Using the Determined CLI tool (det), create a shell by invoking:

$ det shell start

Connect to Visual Studio Code Remote

It is possible to integrate Determined shells with Visual Studio. To do so, first ensure the extension Visual Studio Code Remote - SSH is installed in Visual Studio.

Then, either acquire the SSH connection command by starting a new shell with the --show-ssh-command option:

$ det shell start --show-ssh-command

Or, get the SSH command for an existing shell by using show_ssh_command:

$ det shell show_ssh_command <SHELL UUID>

Copy the output SSH command.

From Visual Studio, navigate to the Command Palette. Select Remote-SSH: Add new SSH Host....

When prompted, paste the copied SSH command into the dialogue box. The final prompt will be to select a configuration file; the default file should work for most users.

The remote host will now be available in the Visual Studio Code Remote Explorer.

Note

For more information regarding IDE integration with Determined AI, refer to the Determined AI documentation's article on IDE integration or the official Visual Studio Code Remote SSH extension documentation.

Commands

Determined AI CLI commands start with det command, abbreviated as det cmd. The main sub-command is det cmd run, which runs a command in the cluster and streams its output.

For example, the following CLI command uses nvidia-smi to display information about the GPUs available to tasks in the container:

$ det cmd run nvidia-smi

More complex commands, including shell constructs, can also be run. These must be quoted to prevent interpretation by the local shell:

$ det cmd run 'for x in a b c; do echo $x; done'

Important

The det cmd run command streams its output until it exits, but the command will continue executing - and occupying cluster resources - even if the CLI is interrupted (Ctrl-C) or killed.

To stop the command, or to view additional output, first acquire the command UUID. Command UUIDs are output with the oriignal output of the first det cmd run, or can be found by running det cmd list. Once you have the command's UUID, you may perform additional operations, such as:

CommandOperation

det cmd logs <UUID>

View a snapshot of logs

det cmd logs -f <UUID>

View the current logs, and continue streaming future output

det cmd kill <UUID>

Stop the command

Additional Resources

Refer to the Determined AI documentation on shell commands for more information on det shell commands.

Last updated