nnictl Commands¶

nnictl is a command line tool, used to control experiments, such as start/stop/resume an experiment, start/stop WebUI, etc.

usage: nnictl [-h] [--version]
              {create,resume,view,update,stop,trial,experiment,platform,webui,config,log,algo,trainingservice,package,top,ss_gen,jupyter-extension,hello}
              ...

Name / Shorthand	Required	Default	Description
`--version, -v`	False	`False`	Describe the current version of NNI installed

nnictl create¶

Create a new experiment.

nnictl create [-h] --config CONFIG [--port PORT] [--debug]
              [--url_prefix URL_PREFIX] [--foreground]

Name / Shorthand	Required	Default	Description
`--config, -c`	True		Path to YAML configuration file of the experiment
`--port, -p`	False	`8080`	The port of restful server
`--debug, -d`	False	`False`	Set debug mode
`--url_prefix, -u`	False		Set prefix url
`--foreground, -f`	False	`False`	Set foreground mode, print log content to terminal

You can use this command to create a new experiment, using the configuration specified in config file.

After this command is successfully done, the context will be set as this experiment, which means the following command you issued is associated with this experiment, unless you explicitly change the context (not supported yet).

Examples:

Create a new experiment with the default port 8080:

nnictl create --config nni/examples/trials/mnist-pytorch/config.yml

Create a new experiment with specified port 8088:

nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088

Create a new experiment with specified port 8088 and debug mode:

nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088 --debug

Note

Debug mode will disable version check function in trial_keeper.

nnictl resume¶

Resume an experiment.

nnictl resume [-h] [--port PORT] [--debug] [--foreground]
              [--experiment_dir EXPERIMENT_DIR]
              id

Name / Shorthand	Required	Default	Description
`id`	True		The ID of the experiment you want to resume
`--port, -p`	False	`8080`	The port of restful server
`--debug, -d`	False	`False`	Set debug mode
`--foreground, -f`	False	`False`	Set foreground mode, print log content to terminal
`--experiment_dir, -e`	False		Resume experiment from external folder, specify the full path of experiment folder

You can use this command to resume a stopped experiment.

Example: resume an experiment with specified port 8088.

nnictl resume [experiment_id] --port 8088

nnictl view¶

View a stopped experiment.

nnictl view [-h] [--port PORT] [--experiment_dir EXPERIMENT_DIR] id

Name / Shorthand	Required	Default	Description
`id`	True		The ID of the experiment you want to view
`--port, -p`	False	`8080`	The port of restful server
`--experiment_dir, -e`	False		View experiment from external folder, specify the full path of experiment folder

nnictl update¶

Update the configuration of an experiment.

nnictl update [-h] {searchspace,concurrency,duration,trialnum} ...

update searchspace¶

Update the search space of an experiment.

nnictl update searchspace [-h] --filename FILENAME [id]

Name / Shorthand	Required	Default	Description
`id`	False		ID of the experiment you want to set
`--filename, -f`	True		Path to new search space file

You can use this command to update an experiment’s search space.

Example: update experiment’s new search space with file dir examples/trials/mnist-pytorch/search_space.json.

nnictl update searchspace [experiment_id] --filename examples/trials/mnist-pytorch/search_space.json

update concurrency¶

Update the concurrency of an experiment.

nnictl update concurrency [-h] --value VALUE [id]

Name / Shorthand	Required	Default	Description
`id`	False		ID of the experiment you want to set
`--value, -v`	True		The number of allowed concurrent trials

update duration¶

Update an experiment’s maximum allowed duration.

nnictl update duration [-h] --value VALUE [id]

Name / Shorthand	Required	Default	Description
`id`	False		ID of the experiment you want to set
`--value, -v`	True		Strings like ‘1m’ for one minute or ‘2h’ for two hours. SUFFIX may be ‘s’ for seconds, ‘m’ for minutes, ‘h’ for hours or ‘d’ for days.

update trialnum¶

Update an experiment’s maximum trial number.

nnictl update trialnum [-h] --value VALUE [id]

Name / Shorthand	Required	Default	Description
`id`	False		ID of the experiment you want to set
`--value, -v`	True		The new number of maxtrialnum you want to set

nnictl stop¶

nnictl stop [-h] [--port PORT] [--all] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID you want to stop
`--port, -p`	False		The port of restful server you want to stop
`--all, -a`	False	`False`	Stop all the experiments

You can use this command to stop a running experiment or multiple experiments.

Details & Examples:

If there is no id specified, and there is an experiment running, stop the running experiment, or print error message.
```
nnictl stop
```
If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.
```
nnictl stop [experiment_id]
```
If there is a port specified, and an experiment is running on that port, the experiment will be stopped.
```
nnictl stop --port 8080
```
Users could use nnictl stop --all to stop all experiments.
If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.
If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.
If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.

nnictl trial¶

Get information of trials.

nnictl trial [-h] {ls,kill} ...

trial ls¶

List trial jobs in one experiment.

nnictl trial ls [-h] [--head HEAD] [--tail TAIL] [id]

Name / Shorthand	Required	Description
`id`	False	Experiment ID
`--head`	False	The number of items to be listed with the highest default metric
`--tail`	False	The number of items to be listed with the lowest default metric

You can use this command to show trials in an experiment.

Note

If head or tail is set, only complete trials will be listed.

trial kill¶

Kill a trial job.

nnictl trial kill [-h] --trial_id TRIAL_ID [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID
`--trial_id, -T`	True		The ID of trial to be killed

nnictl experiment¶

Get information of, or operate on experiments.

nnictl experiment [-h] {show,status,list,delete,import,export,save,load} ...

experiment show¶

Show the information of experiment.

nnictl experiment show [-h] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID

experiment status¶

Show the status of experiment.

nnictl experiment status [-h] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID

experiment list¶

Show the information of all the (running) experiments.

nnictl experiment list [-h] [--all]

Name / Shorthand	Required	Default	Description
`--all`	False	`False`	List all of experiments

experiment delete¶

Delete one or all experiments, it includes log, result, environment information and cache. It can be used to delete useless experiment result, or save disk space.

nnictl experiment delete [-h] [--all] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID
`--all`	False	`False`	Delete all of experiments

experiment import¶

Import additional tuning data into an experiment.

nnictl experiment import [-h] --filename FILENAME [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID
`--filename, -f`	True		A file with data you want to import in json format

You can use this command to import several prior or supplementary trial hyperparameters & results for NNI hyperparameter tuning. The data are fed to the tuning algorithm (e.g., tuner or advisor).

NNI supports users to import their own data, please express the data in the correct format. An example is shown below:

[
    {"parameter": {"x": 0.5, "y": 0.9}, "value": 0.03},
    {"parameter": {"x": 0.4, "y": 0.8}, "value": 0.05},
    {"parameter": {"x": 0.3, "y": 0.7}, "value": 0.04}
]

Every element in the top level list is a sample. For our built-in tuners/advisors, each sample should have at least two keys: parameter and value. The parameter must match this experiment’s search space, that is, all the keys (or hyperparameters) in parameter must match the keys in the search space. Otherwise, tuner/advisor may have unpredictable behavior. value should follow the same rule of the input in nni.report_final_result, that is, either a number or a dict with a key named default. For your customized tuner/advisor, the file could have any json content depending on how you implement the corresponding methods (e.g., import_data).

Note

You can’t see imported data on the web portal when you import data into the experiment. Because currently the import data only has the metric and hyper-parameters, while to be visible on the web portal, one trial must have a complete record, including fields like sequence ID, intermediate results and etc.

You also can use nnictl experiment export to export a valid json file including previous experiment trial hyperparameters and results.

Currently, the following tuners/advisors support import data:

TPE
Anneal
GridSearch
MetisTuner
BOHB

Note

If you want to import data to BOHB advisor, user are suggested to add TRIAL_BUDGET in parameter as NNI do, otherwise, BOHB will use max_budget as TRIAL_BUDGET. Here is an example:

[
    {"parameter": {"x": 0.5, "y": 0.9, "TRIAL_BUDGET": 27}, "value": 0.03}
]

experiment export¶

Export trial job results.

nnictl experiment export [-h] --type {json,csv} --filename PATH
                         [--intermediate]
                         [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID
`--type, -t`	True		Possible choices: json, csv Target file type
`--filename, -f`	True		File path of the output file
`--intermediate, -i`	False	`False`	Whether intermediate results are included

Export results for trial jobs in an experiment as json format.

Example:

nnictl experiment export [experiment_id] --filename [file_path] --type json --intermediate

Note

Import/export are used to deal with trial jobs in a structured format. If you are looking for ways to dump the whole experiment and continue running it on another machine, save/load might intrigue you.

experiment save¶

Dump the metadata and code data of an experiment into a package.

nnictl experiment save [-h] [--path PATH] [--saveCodeDir] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID
`--path, -p`	False		The folder to store nni experiment data. Default: current working directory.
`--saveCodeDir, -s`	False	`False`	Copy code directory into the saved package.

experiment load¶

Load an experiment dumped with save command.

nnictl experiment load [-h] --path PATH --codeDir CODEDIR [--logDir LOGDIR]
                       [--searchSpacePath SEARCHSPACEPATH]

Name / Shorthand	Required	Description
`--path, -p`	True	Path to the packaged experiment.
`--codeDir, -c`	True	Where to put the code for the loaded experiment. Code in the package will be unzipped here.
`--logDir, -l`	False	Path to `logDir` for the loaded experiment
`--searchSpacePath, -s`	False	The file path (not folder) to put the search space file for the loaded experiment. Default: `$codeDir/search_space.json`

nnictl platform¶

nnictl platform [-h] {clean} ...

platform clean¶

Clean up the specified platform.

nnictl platform clean [-h] --config CONFIG

Name / Shorthand	Required	Default	Description
`--config, -c`	True		Path to yaml config file used when creating an experiment on that platform.

Clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.

Note

If the target platform is being used by other users, it may cause unexpected errors to others.

nnictl webui¶

nnictl webui [-h] {url} ...

webui url¶

Show an experiment’s webui url.

nnictl webui url [-h] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID

nnictl config¶

nnictl config [-h] {show} ...

config show¶

Show the config of an experiment.

nnictl config show [-h] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID

nnictl log¶

Manage logs.

nnictl log [-h] {stdout,stderr,trial} ...

log stdout¶

Show the stdout log content.

nnictl log stdout [-h] [--tail TAIL] [--head HEAD] [--path] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID
`--tail, -T`	False		Show tail lines of stdout
`--head, -H`	False		Show head lines of stdout
`--path`	False	`False`	Get the path of stdout file

log stderr¶

Show the stderr log content.

nnictl log stderr [-h] [--tail TAIL] [--head HEAD] [--path] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID
`--tail, -T`	False		Show tail lines of stderr
`--head, -H`	False		Show head lines of stderr
`--path`	False	`False`	Get the path of stderr file

log trial¶

Show trial log path.

nnictl log trial [-h] [--trial_id TRIAL_ID] [id]

Name / Shorthand	Required	Default	Description
`id`	False		Experiment ID
`--trial_id, -T`	False		Trial ID to find the log path, required when experiment ID is set

nnictl algo¶

Manage algorithms.

nnictl algo [-h] {register,reg,unregister,unreg,show,list} ...

algo register (reg)¶

Register customized algorithms as builtin tuner/assessor/advisor.

nnictl algo register [-h] --meta_path META_PATH

Name / Shorthand	Required	Default	Description
`--meta_path, -m`	True		Path to the meta file

Register algorithms so that it can be used like a built-in algorithm.

META_PATH is the path to the meta data file in yml format, which has following keys:

algoType: type of algorithms, could be one of tuner, assessor, advisor.
builtinName: builtin name used in experiment configuration file.
className: tuner class name, including its module name, for example: demo_tuner.DemoTuner.
classArgsValidator: class args validator class name, including its module name, for example: demo_tuner.MyClassArgsValidator.

Example: install a customized tuner in nni examples.

cd nni/examples/tuners/customized_tuner
python3 setup.py develop
nnictl algo register -m meta_file.yml

algo unregister (unreg)¶

Unregister a registered customized builtin algorithms. The NNI-provided builtin algorithms can not be unregistered.

nnictl algo unregister [-h] name

Name / Shorthand	Required	Default	Description
`name`	True		Builtin name of the algorithm

algo show¶

Show the detailed information of specific registered algorithms.

nnictl algo show [-h] name

Name / Shorthand	Required	Default	Description
`name`	True		Builtin name of the algorithm

algo list¶

List the registered builtin algorithms.

nnictl algo list [-h]

nnictl trainingservice¶

(internal preview) Manage 3rd-party training services.

nnictl trainingservice [-h] {register,unregister,list} ...

trainingservice register¶

Register training service.

nnictl trainingservice register [-h] --package PACKAGE

Name / Shorthand	Required	Default	Description
`--package`	True		Package name

trainingservice unregister¶

Unregister training service.

nnictl trainingservice unregister [-h] --package PACKAGE

Name / Shorthand	Required	Default	Description
`--package`	True		Package name

trainingservice list¶

List custom training services.

nnictl trainingservice list [-h]

nnictl package¶

This argument is replaced by algo.

nnictl package [-h] ...

Name / Shorthand	Required	Default	Description
`args`	True

nnictl top¶

Monitor the list of all running experiments.

nnictl top [-h] [--time TIME]

Name / Shorthand	Required	Default	Description
`--time, -t`	False	`3`	The interval to update the experiment status. The unit of time is second, and the default value is 3 seconds.

nnictl ss_gen¶

(deprecated) Automatically generate search space file from trial code.

nnictl ss_gen [-h] --trial_command TRIAL_COMMAND [--trial_dir TRIAL_DIR]
              [--file FILE]

Name / Shorthand	Required	Default	Description
`--trial_command, -t`	True		The command for running trial code
`--trial_dir, -d`	False	`./`	The directory for running the command
`--file, -f`	False	`nni_auto_gen_search_space.json`	The path of search space file

nnictl jupyter-extension¶

(internal preview) Install or uninstall JupyterLab extension.

nnictl jupyter-extension [-h] {install,uninstall} ...

jupyter-extension install¶

Install JupyterLab extension.

nnictl jupyter-extension install [-h]

jupyter-extension uninstall¶

Uninstall JupyterLab extension.

nnictl jupyter-extension uninstall [-h]

nnictl hello¶

Create “hello nni” example in current directory.

nnictl hello [-h]