nnictl

Introduction

nnictl is a command line tool, which can be used to control experiments, such as start/stop/resume an experiment, start/stop NNIBoard, etc.

Commands

nnictl support commands:

Manage an experiment

nnictl create

  • Description

    You can use this command to create a new experiment, using the configuration specified in config file.

    After this command is successfully done, the context will be set as this experiment, which means the following command you issued is associated with this experiment, unless you explicitly changes the context(not supported yet).

  • Usage

    nnictl create [OPTIONS]
    
  • Options

Name, shorthand

Required

Default

Description

–config, -c

True

YAML configure file of the experiment

–port, -p

False

the port of restful server

–debug, -d

False

set debug mode

–foreground, -f

False

set foreground mode, print log content to terminal

  • Examples

    create a new experiment with the default port: 8080

    nnictl create --config nni/examples/trials/mnist-pytorch/config.yml
    

    create a new experiment with specified port 8088

    nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088
    

    create a new experiment with specified port 8088 and debug mode

    nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088 --debug
    

Note:

Debug mode will disable version check function in Trialkeeper.

nnictl resume

  • Description

    You can use this command to resume a stopped experiment.

  • Usage

    nnictl resume [OPTIONS]
    
  • Options

Name, shorthand

Required

Default

Description

id

True

The id of the experiment you want to resume

–port, -p

False

Rest port of the experiment you want to resume

–debug, -d

False

set debug mode

–foreground, -f

False

set foreground mode, print log content to terminal

–experiment_dir, -e

False

Resume experiment from external folder, specify the full path of experiment folder

  • Example

    resume an experiment with specified port 8088

    nnictl resume [experiment_id] --port 8088
    

nnictl view

  • Description

    You can use this command to view a stopped experiment.

  • Usage

    nnictl view [OPTIONS]
    
  • Options

Name, shorthand

Required

Default

Description

id

True

The id of the experiment you want to view

–port, -p

False

Rest port of the experiment you want to view

–experiment_dir, -e

False

View experiment from external folder, specify the full path of experiment folder

  • Example

    view an experiment with specified port 8088

    nnictl view [experiment_id] --port 8088
    

nnictl stop

  • Description

    You can use this command to stop a running experiment or multiple experiments.

  • Usage

    nnictl stop [Options]
    
  • Options

Name, shorthand

Required

Default

Description

id

False

The id of the experiment you want to stop

–port, -p

False

Rest port of the experiment you want to stop

–all, -a

False

Stop all of experiments

  • Details & Examples

    1. If there is no id specified, and there is an experiment running, stop the running experiment, or print error message.

      nnictl stop
      
    2. If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.

      nnictl stop [experiment_id]
      
    3. If there is a port specified, and an experiment is running on that port, the experiment will be stopped.

      nnictl stop --port 8080
      
    4. Users could use ‘nnictl stop –all’ to stop all experiments.

      nnictl stop --all
      
    5. If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.

    6. If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.

    7. If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.

nnictl update

  • nnictl update searchspace

    • Description

      You can use this command to update an experiment’s search space.

    • Usage

      nnictl update searchspace [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

–filename, -f

True

the file storing your new search space

  • Example

    update experiment's new search space with file dir 'examples/trials/mnist-pytorch/search_space.json'

    nnictl update searchspace [experiment_id] --filename examples/trials/mnist-pytorch/search_space.json
    
  • nnictl update concurrency

    • Description

      You can use this command to update an experiment’s concurrency.

    • Usage

      nnictl update concurrency [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

–value, -v

True

the number of allowed concurrent trials

  • Example

    update experiment’s concurrency

    nnictl update concurrency [experiment_id] --value [concurrency_number]
    
  • nnictl update duration

    • Description

      You can use this command to update an experiment’s duration.

    • Usage

      nnictl update duration [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

–value, -v

True

Strings like ‘1m’ for one minute or ‘2h’ for two hours. SUFFIX may be ‘s’ for seconds, ‘m’ for minutes, ‘h’ for hours or ‘d’ for days.

  • Example

    update experiment’s duration

    nnictl update duration [experiment_id] --value [duration]
    
  • nnictl update trialnum

    • Description

      You can use this command to update an experiment’s maxtrialnum.

    • Usage

      nnictl update trialnum [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

–value, -v

True

the new number of maxtrialnum you want to set

  • Example

    update experiment’s trial num

    nnictl update trialnum [experiment_id] --value [trial_num]
    

nnictl trial

  • nnictl trial ls

    • Description

      You can use this command to show trial’s information. Note that if head or tail is set, only complete trials will be listed.

    • Usage

      nnictl trial ls
      nnictl trial ls --head 10
      nnictl trial ls --tail 10
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

–head

False

the number of items to be listed with the highest default metric

–tail

False

the number of items to be listed with the lowest default metric

  • nnictl trial kill

    • Description

      You can use this command to kill a trial job.

    • Usage

      nnictl trial kill [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

Experiment ID of the trial

–trial_id, -T

True

ID of the trial you want to kill.

  • Example

    kill trail job

    nnictl trial kill [experiment_id] --trial_id [trial_id]
    

nnictl top

  • Description

    Monitor all of running experiments.

  • Usage

    nnictl top
    
  • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

–time, -t

False

The interval to update the experiment status, the unit of time is second, and the default value is 3 second.

Manage experiment information

  • nnictl experiment show

    • Description

      Show the information of experiment.

    • Usage

      nnictl experiment show
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

  • nnictl experiment status

    • Description

      Show the status of experiment.

    • Usage

      nnictl experiment status
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

  • nnictl experiment list

    • Description

      Show the information of all the (running) experiments.

    • Usage

      nnictl experiment list [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

–all

False

list all of experiments

  • nnictl experiment delete

    • Description

      Delete one or all experiments, it includes log, result, environment information and cache. It uses to delete useless experiment result, or save disk space.

    • Usage

      nnictl experiment delete [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment

–all

False

delete all of experiments

  • nnictl experiment export

    • Description

      You can use this command to export reward & hyper-parameter of trial jobs to a csv file.

    • Usage

      nnictl experiment export [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment

–filename, -f

True

File path of the output file

–type

True

Type of output file, only support “csv” and “json”

–intermediate, -i

False

Are intermediate results included

  • Examples

    export all trial data in an experiment as json format

    nnictl experiment export [experiment_id] --filename [file_path] --type json --intermediate
    
  • nnictl experiment import

    • Description

      You can use this command to import several prior or supplementary trial hyperparameters & results for NNI hyperparameter tuning. The data are fed to the tuning algorithm (e.g., tuner or advisor).

    • Usage

      nnictl experiment import [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

The id of the experiment you want to import data into

–filename, -f

True

a file with data you want to import in json format

  • Details

    NNI supports users to import their own data, please express the data in the correct format. An example is shown below:

    [
      {"parameter": {"x": 0.5, "y": 0.9}, "value": 0.03},
      {"parameter": {"x": 0.4, "y": 0.8}, "value": 0.05},
      {"parameter": {"x": 0.3, "y": 0.7}, "value": 0.04}
    ]
    

    Every element in the top level list is a sample. For our built-in tuners/advisors, each sample should have at least two keys: parameter and value. The parameter must match this experiment’s search space, that is, all the keys (or hyperparameters) in parameter must match the keys in the search space. Otherwise, tuner/advisor may have unpredictable behavior. Value should follow the same rule of the input in nni.report_final_result, that is, either a number or a dict with a key named default. For your customized tuner/advisor, the file could have any json content depending on how you implement the corresponding methods (e.g., import_data).

    You also can use nnictl experiment export to export a valid json file including previous experiment trial hyperparameters and results.

    Currently, following tuner and advisor support import data:

    builtinTunerName: TPE, Anneal, GridSearch, MetisTuner
    builtinAdvisorName: BOHB
    

    If you want to import data to BOHB advisor, user are suggested to add “TRIAL_BUDGET” in parameter as NNI do, otherwise, BOHB will use max_budget as “TRIAL_BUDGET”. Here is an example:

    [
      {"parameter": {"x": 0.5, "y": 0.9, "TRIAL_BUDGET": 27}, "value": 0.03}
    ]
    
  • Examples

    import data to a running experiment

    nnictl experiment import [experiment_id] -f experiment_data.json
    
  • nnictl experiment save

    • Description

      Save nni experiment metadata and code data.

    • Usage

      nnictl experiment save [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

id

True

The id of the experiment you want to save

–path, -p

False

the folder path to store nni experiment data, default current working directory

–saveCodeDir, -s

False

save codeDir data of the experiment, default False

  • Examples

    save an expeirment

    nnictl experiment save [experiment_id] --saveCodeDir
    
  • nnictl experiment load

    • Description

      Load an nni experiment.

    • Usage

      nnictl experiment load [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

–path, -p

True

the file path of nni package

–codeDir, -c

True

the path of codeDir for loaded experiment, this path will also put the code in the loaded experiment package

–logDir, -l

False

the path of logDir for loaded experiment

–searchSpacePath, -s

True

the path of search space file for loaded experiment, this path contains file name. Default in $codeDir/search_space.json

  • Examples

    load an expeirment

    nnictl experiment load --path [path] --codeDir [codeDir]
    

Manage platform information

  • nnictl platform clean

    • Description

      It uses to clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.

    • Note

      if the target platform is being used by other users, it may cause unexpected errors to others.

    • Usage

      nnictl platform clean [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

–config

True

the path of yaml config file used when create an experiment

nnictl config show

  • Description

    Display the current context information.

  • Usage

    nnictl config show
    

Manage log

  • nnictl log stdout

    • Description

      Show the stdout log content.

    • Usage

      nnictl log stdout [options]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

–head, -h

False

show head lines of stdout

–tail, -t

False

show tail lines of stdout

–path, -p

False

show the path of stdout file

  • Example

    Show the tail of stdout log content

    nnictl log stdout [experiment_id] --tail [lines_number]
    
  • nnictl log stderr

    • Description

      Show the stderr log content.

    • Usage

      nnictl log stderr [options]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

–head, -h

False

show head lines of stderr

–tail, -t

False

show tail lines of stderr

–path, -p

False

show the path of stderr file

  • nnictl log trial

    • Description

      Show trial log path.

    • Usage

      nnictl log trial [options]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

Experiment ID of the trial

–trial_id, -T

False

ID of the trial to be found the log path, required when id is not empty.

Manage webui

  • nnictl webui url

    • Description

      Show an experiment’s webui url

    • Usage

      nnictl webui url [options]
      
    • Options

Name, shorthand

Required

Default

Description

id

False

Experiment ID

Manage builtin algorithms

  • nnictl algo register

    • Description

      Register customized algorithms as builtin tuner/assessor/advisor.

    • Usage

      nnictl algo register --meta <path_to_meta_file>
      

      <path_to_meta_file> is the path to the meta data file in yml format, which has following keys:

      • algoType: type of algorithms, could be one of tuner, assessor, advisor

      • builtinName: builtin name used in experiment configuration file

      • className: tuner class name, including its module name, for example: demo_tuner.DemoTuner

      • classArgsValidator: class args validator class name, including its module name, for example: demo_tuner.MyClassArgsValidator

    • Example

      Install a customized tuner in nni examples

      cd nni/examples/tuners/customized_tuner
      python3 setup.py develop
      nnictl algo register --meta meta_file.yml
      
  • nnictl algo show

    • Description

      Show the detailed information of specified registered algorithms.

    • Usage

      nnictl algo show <builtinName>
      
    • Example

      nnictl algo show SMAC
      
  • nnictl package list

    • Description

      List the registered builtin algorithms.

    • Usage

      nnictl algo list
      
  • Example

    nnictl algo list
    
  • nnictl algo unregister

    • Description

      Unregister a registered customized builtin algorithms. The NNI provided builtin algorithms can not be unregistered.

    • Usage

      nnictl algo unregister <builtinName>
      
    • Example

      nnictl algo unregister demotuner
      

Generate search space

  • nnictl ss_gen

    • Description

      Generate search space from user trial code which uses NNI NAS APIs.

    • Usage

      nnictl ss_gen [OPTIONS]
      
    • Options

Name, shorthand

Required

Default

Description

–trial_command

True

The command of the trial code

–trial_dir

False

./

The directory of the trial code

–file

False

nni_auto_gen_search_space.json

The file for storing generated search space

  • Example

    Generate a search space

    nnictl ss_gen --trial_command="python3 mnist.py" --trial_dir=./ --file=ss.json
    

Check NNI version

  • nnictl –version

    • Description

      Describe the current version of NNI installed.

    • Usage

      nnictl --version