nnictl

Introduction

nnictl is a command line tool, which can be used to control experiments, such as start/stop/resume an experiment, start/stop NNIBoard, etc.

Commands

nnictl support commands:

• nnictl create
• nnictl resume
• nnictl view
• nnictl stop
• nnictl update
• nnictl trial
• nnictl top
• nnictl experiment
• nnictl platform
• nnictl config
• nnictl log
• nnictl webui
• nnictl tensorboard
• nnictl package
• nnictl ss_gen
• nnictl –version

Manage an experiment

nnictl create

• Description

  You can use this command to create a new experiment, using the configuration specified in config file.

  After this command is successfully done, the context will be set as this experiment, which means the following command you issued is associated with this experiment, unless you explicitly changes the context(not supported yet).

• Usage

  nnictl create [OPTIONS]
  

• Options

Name, shorthand	Required	Default	Description
--config, -c	True		YAML configure file of the experiment
--port, -p	False		the port of restful server
--debug, -d	False		set debug mode
--foreground, -f	False		set foreground mode, print log content to terminal

• Examples

  create a new experiment with the default port: 8080

  nnictl create --config nni/examples/trials/mnist-tfv1/config.yml
  

  create a new experiment with specified port 8088

  nnictl create --config nni/examples/trials/mnist-tfv1/config.yml --port 8088
  

  create a new experiment with specified port 8088 and debug mode

  nnictl create --config nni/examples/trials/mnist-tfv1/config.yml --port 8088 --debug
  

Note:

Debug mode will disable version check function in Trialkeeper.

nnictl resume

• Description

  You can use this command to resume a stopped experiment.

• Usage

  nnictl resume [OPTIONS]
  

• Options

Name, shorthand	Required	Default	Description
id	True		The id of the experiment you want to resume
--port, -p	False		Rest port of the experiment you want to resume
--debug, -d	False		set debug mode
--foreground, -f	False		set foreground mode, print log content to terminal

• Example

  resume an experiment with specified port 8088

  nnictl resume [experiment_id] --port 8088
  

nnictl view

• Description

  You can use this command to view a stopped experiment.

• Usage

  nnictl view [OPTIONS]
  

• Options

Name, shorthand	Required	Default	Description
id	True		The id of the experiment you want to view
--port, -p	False		Rest port of the experiment you want to view

• Example

  view an experiment with specified port 8088

  nnictl view [experiment_id] --port 8088
  

nnictl stop

• Description

  You can use this command to stop a running experiment or multiple experiments.

• Usage

  nnictl stop [Options]
  

• Options

Name, shorthand	Required	Default	Description
id	False		The id of the experiment you want to stop
--port, -p	False		Rest port of the experiment you want to stop
--all, -a	False		Stop all of experiments

• Details & Examples

  1. If there is no id specified, and there is an experiment running, stop the running experiment, or print error message.

     nnictl stop
     

  2. If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.

     nnictl stop [experiment_id]
     

  3. If there is a port specified, and an experiment is running on that port, the experiment will be stopped.

     nnictl stop --port 8080
     

  4. Users could use ‘nnictl stop –all’ to stop all experiments.

     nnictl stop --all
     

  5. If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.

  6. If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.

  7. If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.

nnictl update

• nnictl update searchspace

  • Description

    You can use this command to update an experiment’s search space.

  • Usage

    nnictl update searchspace [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
--filename, -f	True		the file storing your new search space

• Example

  update experiment's new search space with file dir 'examples/trials/mnist-tfv1/search_space.json'

  nnictl update searchspace [experiment_id] --filename examples/trials/mnist-tfv1/search_space.json
  

• nnictl update concurrency

  • Description

    You can use this command to update an experiment’s concurrency.

  • Usage

    nnictl update concurrency [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
--value, -v	True		the number of allowed concurrent trials

• Example

  update experiment’s concurrency

  nnictl update concurrency [experiment_id] --value [concurrency_number]
  

• nnictl update duration

  • Description

    You can use this command to update an experiment’s duration.

  • Usage

    nnictl update duration [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
--value, -v	True		the experiment duration will be NUMBER seconds. SUFFIX may be 's' for seconds (the default), 'm' for minutes, 'h' for hours or 'd' for days.

• Example

  update experiment’s duration

  nnictl update duration [experiment_id] --value [duration]
  

• nnictl update trialnum

  • Description

    You can use this command to update an experiment’s maxtrialnum.

  • Usage

    nnictl update trialnum [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
--value, -v	True		the new number of maxtrialnum you want to set

• Example

  update experiment’s trial num

  nnictl update trialnum --id [experiment_id] --value [trial_num]
  

nnictl trial

• nnictl trial ls

  • Description

    You can use this command to show trial’s information.

  • Usage

    nnictl trial ls
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set

• nnictl trial kill

  • Description

    You can use this command to kill a trial job.

  • Usage

    nnictl trial kill [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		Experiment ID of the trial
--trial_id, -T	True		ID of the trial you want to kill.

• Example

  kill trail job

  nnictl trial [trial_id] --experiment [experiment_id]
  

nnictl top

• Description

  Monitor all of running experiments.

• Usage

  nnictl top
  

• Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
--time, -t	False		The interval to update the experiment status, the unit of time is second, and the default value is 3 second.

Manage experiment information

• nnictl experiment show

  • Description

    Show the information of experiment.

  • Usage

    nnictl experiment show
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set

• nnictl experiment status

  • Description

    Show the status of experiment.

  • Usage

    nnictl experiment status
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set

• nnictl experiment list

  • Description

    Show the information of all the (running) experiments.

  • Usage

    nnictl experiment list [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
--all	False		list all of experiments

• nnictl experiment delete

  • Description

    Delete one or all experiments, it includes log, result, environment information and cache. It uses to delete useless experiment result, or save disk space.

  • Usage

    nnictl experiment delete [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment
--all	False		delete all of experiments

• nnictl experiment export

  • Description

    You can use this command to export reward & hyper-parameter of trial jobs to a csv file.

  • Usage

    nnictl experiment export [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment
--filename, -f	True		File path of the output file
--type	True		Type of output file, only support "csv" and "json"

• Examples

export all trial data in an experiment as json format

nnictl experiment export [experiment_id] --filename [file_path] --type json

• nnictl experiment import

  • Description

    You can use this command to import several prior or supplementary trial hyperparameters & results for NNI hyperparameter tuning. The data are fed to the tuning algorithm (e.g., tuner or advisor).

  • Usage

    nnictl experiment import [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		The id of the experiment you want to import data into
--filename, -f	True		a file with data you want to import in json format

• Details

  NNI supports users to import their own data, please express the data in the correct format. An example is shown below:

  [
    {"parameter": {"x": 0.5, "y": 0.9}, "value": 0.03},
    {"parameter": {"x": 0.4, "y": 0.8}, "value": 0.05},
    {"parameter": {"x": 0.3, "y": 0.7}, "value": 0.04}
  ]
  

  Every element in the top level list is a sample. For our built-in tuners/advisors, each sample should have at least two keys: parameter and value. The parameter must match this experiment’s search space, that is, all the keys (or hyperparameters) in parameter must match the keys in the search space. Otherwise, tuner/advisor may have unpredictable behavior. Value should follow the same rule of the input in nni.report_final_result, that is, either a number or a dict with a key named default. For your customized tuner/advisor, the file could have any json content depending on how you implement the corresponding methods (e.g., import_data).

  You also can use nnictl experiment export to export a valid json file including previous experiment trial hyperparameters and results.

  Currently, following tuner and advisor support import data:

  builtinTunerName: TPE, Anneal, GridSearch, MetisTuner
  builtinAdvisorName: BOHB
  

  If you want to import data to BOHB advisor, user are suggested to add “TRIAL_BUDGET” in parameter as NNI do, otherwise, BOHB will use max_budget as “TRIAL_BUDGET”. Here is an example:

  [
    {"parameter": {"x": 0.5, "y": 0.9, "TRIAL_BUDGET": 27}, "value": 0.03}
  ]
  

• Examples

  import data to a running experiment

  nnictl experiment import [experiment_id] -f experiment_data.json
  

Manage platform information

• nnictl platform clean

  • Description

    It uses to clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.

  • Note

    if the target platform is being used by other users, it may cause unexpected errors to others.

  • Usage

    nnictl platform clean [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
--config	True		the path of yaml config file used when create an experiment

nnictl config show

• Description

  Display the current context information.

• Usage

  nnictl config show
  

Manage log

• nnictl log stdout

  • Description

    Show the stdout log content.

  • Usage

    nnictl log stdout [options]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
--head, -h	False		show head lines of stdout
--tail, -t	False		show tail lines of stdout
--path, -p	False		show the path of stdout file

• Example

  Show the tail of stdout log content

  nnictl log stdout [experiment_id] --tail [lines_number]
  

• nnictl log stderr

  • Description

    Show the stderr log content.

  • Usage

    nnictl log stderr [options]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
--head, -h	False		show head lines of stderr
--tail, -t	False		show tail lines of stderr
--path, -p	False		show the path of stderr file

• nnictl log trial

  • Description

    Show trial log path.

  • Usage

    nnictl log trial [options]
    

  • Options

Name, shorthand	Required	Default	Description
id	False		Experiment ID of the trial
--trial_id, -T	False		ID of the trial to be found the log path, required when id is not empty.

Manage webui

• nnictl webui url

Manage tensorboard

• nnictl tensorboard start

  • Description

    Start the tensorboard process.

  • Usage

    nnictl tensorboard start
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
--trial_id, -T	False		ID of the trial
--port	False	6006	The port of the tensorboard process

• Detail

  1. NNICTL support tensorboard function in local and remote platform for the moment, other platforms will be supported later.
  2. If you want to use tensorboard, you need to write your tensorboard log data to environment variable [NNI_OUTPUT_DIR] path.
  3. In local mode, nnictl will set –logdir=[NNI_OUTPUT_DIR] directly and start a tensorboard process.
  4. In remote mode, nnictl will create a ssh client to copy log data from remote machine to local temp directory firstly, and then start a tensorboard process in your local machine. You need to notice that nnictl only copy the log data one time when you use the command, if you want to see the later result of tensorboard, you should execute nnictl tensorboard command again.
  5. If there is only one trial job, you don’t need to set trial id. If there are multiple trial jobs running, you should set the trial id, or you could use [nnictl tensorboard start –trial_id all] to map –logdir to all trial log paths.

• nnictl tensorboard stop

  • Description

    Stop all of the tensorboard process.

  • Usage

    nnictl tensorboard stop
    

  • Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set

Manage package

• nnictl package install

  • Description

    Install the packages needed in nni experiments.

  • Usage

    nnictl package install [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
--name	True		The name of package to be installed

• Example

  Install the packages needed in tuner SMAC

  nnictl package install --name=SMAC
  

• nnictl package show

  • Description

    List the packages supported.

  • Usage

    nnictl package show
    

Generate search space

• nnictl ss_gen

  • Description

    Generate search space from user trial code which uses NNI NAS APIs.

  • Usage

    nnictl ss_gen [OPTIONS]
    

  • Options

Name, shorthand	Required	Default	Description
--trial_command	True		The command of the trial code
--trial_dir	False	./	The directory of the trial code
--file	False	nni_auto_gen_search_space.json	The file for storing generated search space

• Example

  Generate a search space

  nnictl ss_gen --trial_command="python3 mnist.py" --trial_dir=./ --file=ss.json
  

Check NNI version

• nnictl –version

  • Description

    Describe the current version of NNI installed.

  • Usage

    nnictl --version