Tutorial: Create and Run an Experiment on local with NNI API

In this tutorial, we will use the example in [nni/examples/trials/mnist-pytorch] to explain how to create and run an experiment on local with NNI API.

Before starts

You have an implementation for MNIST classifer using convolutional layers, the Python code is similar to mnist.py.

Step 1 - Update model codes

To enable NNI API, make the following changes:

1.1 Declare NNI API: include import nni in your trial code to use NNI APIs.

1.2 Get predefined parameters

Use the following code snippet:

tuner_params = nni.get_next_parameter()

to get hyper-parameters’ values assigned by tuner. tuner_params is an object, for example:

{"batch_size": 32, "hidden_size": 128, "lr": 0.01, "momentum": 0.2029}

1.3 Report NNI results: Use the API: nni.report_intermediate_result(accuracy) to send accuracy to assessor. Use the API: nni.report_final_result(accuracy) to send accuracy to tuner.

NOTE:

accuracy - The `accuracy` could be any python object, but  if you use NNI built-in tuner/assessor, `accuracy` should be a numerical variable (e.g. float, int).
tuner    - The tuner will generate next parameters/architecture based on the explore history (final result of all trials).
assessor - The assessor will decide which trial should early stop based on the history performance of trial (intermediate result of one trial).

Step 2 - Define SearchSpace

The hyper-parameters used in Step 1.2 - Get predefined parameters is defined in a search_space.json file like below:

{
    "batch_size": {"_type":"choice", "_value": [16, 32, 64, 128]},
    "hidden_size":{"_type":"choice","_value":[128, 256, 512, 1024]},
    "lr":{"_type":"choice","_value":[0.0001, 0.001, 0.01, 0.1]},
    "momentum":{"_type":"uniform","_value":[0, 1]}
}

Refer to define search space to learn more about search space.

Step 3 - Define Experiment

To run an experiment in NNI, you only needed:

  • Provide a runnable trial

  • Provide or choose a tuner

  • Provide a YAML experiment configure file

  • (optional) Provide or choose an assessor

Prepare trial:

You can download nni source code and a set of examples can be found in nni/examples, run ls nni/examples/trials to see all the trial examples.

Let’s use a simple trial example, e.g. mnist, provided by NNI. After you cloned NNI source, NNI examples have been put in ~/nni/examples, run ls ~/nni/examples/trials to see all the trial examples. You can simply execute the following command to run the NNI mnist example:

python ~/nni/examples/trials/mnist-pytorch/mnist.py

This command will be filled in the YAML configure file below. Please refer to here for how to write your own trial.

Prepare tuner: NNI supports several popular automl algorithms, including Random Search, Tree of Parzen Estimators (TPE), Evolution algorithm etc. Users can write their own tuner (refer to here), but for simplicity, here we choose a tuner provided by NNI as below:

tuner:
  name: TPE
  classArgs:
    optimize_mode: maximize

name is used to specify a tuner in NNI, classArgs are the arguments pass to the tuner (the spec of builtin tuners can be found here), optimization_mode is to indicate whether you want to maximize or minimize your trial’s result.

Prepare configure file: Since you have already known which trial code you are going to run and which tuner you are going to use, it is time to prepare the YAML configure file. NNI provides a demo configure file for each trial example, cat ~/nni/examples/trials/mnist-pytorch/config.yml to see it. Its content is basically shown below:

experimentName: local training service example

searchSpaceFile ~/nni/examples/trials/mnist-pytorch/search_space.json
trailCommand: python3 mnist.py
trialCodeDirectory: ~/nni/examples/trials/mnist-pytorch

trialGpuNumber: 0
trialConcurrency: 1
maxExperimentDuration: 3h
maxTrialNumber: 10

trainingService:
  platform: local

tuner:
  name: TPE
  classArgs:
    optimize_mode: maximize

With all these steps done, we can run the experiment with the following command:

nnictl create --config ~/nni/examples/trials/mnist-pytorch/config.yml

You can refer to here for more usage guide of nnictl command line tool.

View experiment results

The experiment has been running now. Other than nnictl, NNI also provides WebUI for you to view experiment progress, to control your experiment, and some other appealing features.