nnictl¶

Introduction¶

nnictl is a command line tool, which can be used to control experiments, such as start/stop/resume an experiment, start/stop NNIBoard, etc.

Commands¶

nnictl support commands:

nnictl create
nnictl resume
nnictl view
nnictl stop
nnictl update
nnictl trial
nnictl top
nnictl experiment
nnictl platform
nnictl config
nnictl log
nnictl webui
nnictl algo
nnictl ss_gen
nnictl –version

Manage an experiment¶

nnictl create¶

Description

You can use this command to create a new experiment, using the configuration specified in config file.

After this command is successfully done, the context will be set as this experiment, which means the following command you issued is associated with this experiment, unless you explicitly changes the context(not supported yet).
Usage
```
nnictl create [OPTIONS]
```
Options

Name, shorthand	Required	Description
–config, -c	True	YAML configure file of the experiment
–port, -p	False	the port of restful server
–debug, -d	False	set debug mode
–foreground, -f	False	set foreground mode, print log content to terminal

Examples

create a new experiment with the default port: 8080

nnictl create --config nni/examples/trials/mnist-pytorch/config.yml

create a new experiment with specified port 8088

nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088

create a new experiment with specified port 8088 and debug mode

nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088 --debug

Note:

Debug mode will disable version check function in Trialkeeper.

nnictl resume¶

Description

You can use this command to resume a stopped experiment.
Usage
```
nnictl resume [OPTIONS]
```
Options

Name, shorthand	Required	Description
id	True	The id of the experiment you want to resume
–port, -p	False	Rest port of the experiment you want to resume
–debug, -d	False	set debug mode
–foreground, -f	False	set foreground mode, print log content to terminal
–experiment_dir, -e	False	Resume experiment from external folder, specify the full path of experiment folder

Example

resume an experiment with specified port 8088
```
nnictl resume [experiment_id] --port 8088
```

nnictl view¶

Description

You can use this command to view a stopped experiment.
Usage
```
nnictl view [OPTIONS]
```
Options

Name, shorthand	Required	Description
id	True	The id of the experiment you want to view
–port, -p	False	Rest port of the experiment you want to view
–experiment_dir, -e	False	View experiment from external folder, specify the full path of experiment folder

Example

view an experiment with specified port 8088
```
nnictl view [experiment_id] --port 8088
```

nnictl stop¶

Description

You can use this command to stop a running experiment or multiple experiments.
Usage
```
nnictl stop [Options]
```
Options

Name, shorthand	Required	Description
id	False	The id of the experiment you want to stop
–port, -p	False	Rest port of the experiment you want to stop
–all, -a	False	Stop all of experiments

Details & Examples
1. If there is no id specified, and there is an experiment running, stop the running experiment, or print error message.
```
nnictl stop
```
2. If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.
```
nnictl stop [experiment_id]
```
3. If there is a port specified, and an experiment is running on that port, the experiment will be stopped.
```
nnictl stop --port 8080
```
4. Users could use ‘nnictl stop –all’ to stop all experiments.
```
nnictl stop --all
```
5. If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.
6. If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.
7. If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.

nnictl update¶

nnictl update searchspace
- Description
  
  You can use this command to update an experiment’s search space.
- Usage
```
nnictl update searchspace [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
–filename, -f	True		the file storing your new search space

Example

update experiment's new search space with file dir 'examples/trials/mnist-pytorch/search_space.json'

nnictl update searchspace [experiment_id] --filename examples/trials/mnist-pytorch/search_space.json

nnictl update concurrency
- Description
  
  You can use this command to update an experiment’s concurrency.
- Usage
```
nnictl update concurrency [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
–value, -v	True		the number of allowed concurrent trials

Example

update experiment’s concurrency

nnictl update concurrency [experiment_id] --value [concurrency_number]

nnictl update duration
- Description
  
  You can use this command to update an experiment’s duration.
- Usage
```
nnictl update duration [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
–value, -v	True		Strings like ‘1m’ for one minute or ‘2h’ for two hours. SUFFIX may be ‘s’ for seconds, ‘m’ for minutes, ‘h’ for hours or ‘d’ for days.

Example

update experiment’s duration

nnictl update duration [experiment_id] --value [duration]

nnictl update trialnum
- Description
  
  You can use this command to update an experiment’s maxtrialnum.
- Usage
```
nnictl update trialnum [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
–value, -v	True		the new number of maxtrialnum you want to set

Example

update experiment’s trial num

nnictl update trialnum [experiment_id] --value [trial_num]

nnictl trial¶

nnictl trial ls
- Description
  
  You can use this command to show trial’s information. Note that if head or tail is set, only complete trials will be listed.
- Usage
```
nnictl trial ls
nnictl trial ls --head 10
nnictl trial ls --tail 10
```
- Options

Name, shorthand	Required	Description
id	False	ID of the experiment you want to set
–head	False	the number of items to be listed with the highest default metric
–tail	False	the number of items to be listed with the lowest default metric

nnictl trial kill
- Description
  
  You can use this command to kill a trial job.
- Usage
```
nnictl trial kill [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
id	False		Experiment ID of the trial
–trial_id, -T	True		ID of the trial you want to kill.

Example

kill trail job

nnictl trial kill [experiment_id] --trial_id [trial_id]

nnictl top¶

Description

Monitor all of running experiments.
Usage
```
nnictl top
```
Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set
–time, -t	False		The interval to update the experiment status, the unit of time is second, and the default value is 3 second.

Manage experiment information¶

nnictl experiment show
- Description
  
  Show the information of experiment.
- Usage
```
nnictl experiment show
```
- Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set

nnictl experiment status
- Description
  
  Show the status of experiment.
- Usage
```
nnictl experiment status
```
- Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment you want to set

nnictl experiment list
- Description
  
  Show the information of all the (running) experiments.
- Usage
```
nnictl experiment list [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
–all	False		list all of experiments

nnictl experiment delete
- Description
  
  Delete one or all experiments, it includes log, result, environment information and cache. It uses to delete useless experiment result, or save disk space.
- Usage
```
nnictl experiment delete [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
id	False		ID of the experiment
–all	False		delete all of experiments

nnictl experiment export
- Description
  
  You can use this command to export reward & hyper-parameter of trial jobs to a csv file.
- Usage
```
nnictl experiment export [OPTIONS]
```
- Options

Name, shorthand	Required	Description
id	False	ID of the experiment
–filename, -f	True	File path of the output file
–type	True	Type of output file, only support “csv” and “json”
–intermediate, -i	False	Are intermediate results included

Examples

export all trial data in an experiment as json format

nnictl experiment export [experiment_id] --filename [file_path] --type json --intermediate

nnictl experiment import
- Description
  
  You can use this command to import several prior or supplementary trial hyperparameters & results for NNI hyperparameter tuning. The data are fed to the tuning algorithm (e.g., tuner or advisor).
- Usage
```
nnictl experiment import [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
id	False		The id of the experiment you want to import data into
–filename, -f	True		a file with data you want to import in json format

Details

NNI supports users to import their own data, please express the data in the correct format. An example is shown below:
```
[
  {"parameter": {"x": 0.5, "y": 0.9}, "value": 0.03},
  {"parameter": {"x": 0.4, "y": 0.8}, "value": 0.05},
  {"parameter": {"x": 0.3, "y": 0.7}, "value": 0.04}
]
```
Every element in the top level list is a sample. For our built-in tuners/advisors, each sample should have at least two keys: parameter and value. The parameter must match this experiment’s search space, that is, all the keys (or hyperparameters) in parameter must match the keys in the search space. Otherwise, tuner/advisor may have unpredictable behavior. Value should follow the same rule of the input in nni.report_final_result, that is, either a number or a dict with a key named default. For your customized tuner/advisor, the file could have any json content depending on how you implement the corresponding methods (e.g., import_data).

You also can use nnictl experiment export to export a valid json file including previous experiment trial hyperparameters and results.

Currently, following tuner and advisor support import data:
```
builtinTunerName: TPE, Anneal, GridSearch, MetisTuner
builtinAdvisorName: BOHB
```
If you want to import data to BOHB advisor, user are suggested to add “TRIAL_BUDGET” in parameter as NNI do, otherwise, BOHB will use max_budget as “TRIAL_BUDGET”. Here is an example:
```
[
  {"parameter": {"x": 0.5, "y": 0.9, "TRIAL_BUDGET": 27}, "value": 0.03}
]
```

Examples

import data to a running experiment

nnictl experiment import [experiment_id] -f experiment_data.json

nnictl experiment save
- Description
  
  Save nni experiment metadata and code data.
- Usage
```
nnictl experiment save [OPTIONS]
```
- Options

Name, shorthand	Required	Description
id	True	The id of the experiment you want to save
–path, -p	False	the folder path to store nni experiment data, default current working directory
–saveCodeDir, -s	False	save codeDir data of the experiment, default False

Examples

save an expeirment

nnictl experiment save [experiment_id] --saveCodeDir

nnictl experiment load
- Description
  
  Load an nni experiment.
- Usage
```
nnictl experiment load [OPTIONS]
```
- Options

Name, shorthand	Required	Description
–path, -p	True	the file path of nni package
–codeDir, -c	True	the path of codeDir for loaded experiment, this path will also put the code in the loaded experiment package
–logDir, -l	False	the path of logDir for loaded experiment
–searchSpacePath, -s	True	the path of search space file for loaded experiment, this path contains file name. Default in $codeDir/search_space.json

Examples

load an expeirment

nnictl experiment load --path [path] --codeDir [codeDir]

Manage platform information¶

nnictl platform clean
- Description
  
  It uses to clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.
- Note
  
  if the target platform is being used by other users, it may cause unexpected errors to others.
- Usage
```
nnictl platform clean [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
–config	True		the path of yaml config file used when create an experiment

nnictl config show¶

Description

Display the current context information.
Usage
```
nnictl config show
```

Manage log¶

nnictl log stdout
- Description
  
  Show the stdout log content.
- Usage
```
nnictl log stdout [options]
```
- Options

Name, shorthand	Required	Description
id	False	ID of the experiment you want to set
–head, -h	False	show head lines of stdout
–tail, -t	False	show tail lines of stdout
–path, -p	False	show the path of stdout file

Example

Show the tail of stdout log content

nnictl log stdout [experiment_id] --tail [lines_number]

nnictl log stderr
- Description
  
  Show the stderr log content.
- Usage
```
nnictl log stderr [options]
```
- Options

Name, shorthand	Required	Description
id	False	ID of the experiment you want to set
–head, -h	False	show head lines of stderr
–tail, -t	False	show tail lines of stderr
–path, -p	False	show the path of stderr file

nnictl log trial
- Description
  
  Show trial log path.
- Usage
```
nnictl log trial [options]
```
- Options

Name, shorthand	Required	Default	Description
id	False		Experiment ID of the trial
–trial_id, -T	False		ID of the trial to be found the log path, required when id is not empty.

Manage webui¶

nnictl webui url
- Description
  
  Show an experiment’s webui url
- Usage
```
nnictl webui url [options]
```
- Options

Name, shorthand	Required	Default	Description
id	False		Experiment ID

Manage builtin algorithms¶

nnictl algo register
- Description
  
  Register customized algorithms as builtin tuner/assessor/advisor.
- Usage
```
nnictl algo register --meta <path_to_meta_file>
```
  <path_to_meta_file> is the path to the meta data file in yml format, which has following keys:
  - algoType: type of algorithms, could be one of tuner, assessor, advisor
  - builtinName: builtin name used in experiment configuration file
  - className: tuner class name, including its module name, for example: demo_tuner.DemoTuner
  - classArgsValidator: class args validator class name, including its module name, for example: demo_tuner.MyClassArgsValidator
- Example
  
  Install a customized tuner in nni examples
```
cd nni/examples/tuners/customized_tuner
python3 setup.py develop
nnictl algo register --meta meta_file.yml
```
nnictl algo show
- Description
  
  Show the detailed information of specified registered algorithms.
- Usage
```
nnictl algo show <builtinName>
```
- Example
```
nnictl algo show SMAC
```
nnictl package list
- Description
  
  List the registered builtin algorithms.
- Usage
```
nnictl algo list
```
Example
```
nnictl algo list
```
nnictl algo unregister
- Description
  
  Unregister a registered customized builtin algorithms. The NNI provided builtin algorithms can not be unregistered.
- Usage
```
nnictl algo unregister <builtinName>
```
- Example
```
nnictl algo unregister demotuner
```

Generate search space¶

nnictl ss_gen
- Description
  
  Generate search space from user trial code which uses NNI NAS APIs.
- Usage
```
nnictl ss_gen [OPTIONS]
```
- Options

Name, shorthand	Required	Default	Description
–trial_command	True		The command of the trial code
–trial_dir	False	./	The directory of the trial code
–file	False	nni_auto_gen_search_space.json	The file for storing generated search space

Example

Generate a search space

nnictl ss_gen --trial_command="python3 mnist.py" --trial_dir=./ --file=ss.json

Check NNI version¶

nnictl –version
- Description
  
  Describe the current version of NNI installed.
- Usage
```
nnictl --version
```