nnictl Commands

nnictl is a command line tool, used to control experiments, such as start/stop/resume an experiment, start/stop WebUI, etc.

usage: nnictl [-h] [--version]
              {create,resume,view,update,stop,trial,experiment,platform,webui,config,log,algo,trainingservice,package,top,ss_gen,jupyter-extension,hello}
              ...

Name / Shorthand

Required

Default

Description

--version, -v

False

False

Describe the current version of NNI installed

nnictl create

Create a new experiment.

nnictl create [-h] --config CONFIG [--port PORT] [--debug]
              [--url_prefix URL_PREFIX] [--foreground]

Name / Shorthand

Required

Default

Description

--config, -c

True

Path to YAML configuration file of the experiment

--port, -p

False

8080

The port of restful server

--debug, -d

False

False

Set debug mode

--url_prefix, -u

False

Set prefix url

--foreground, -f

False

False

Set foreground mode, print log content to terminal

You can use this command to create a new experiment, using the configuration specified in config file.

After this command is successfully done, the context will be set as this experiment, which means the following command you issued is associated with this experiment, unless you explicitly change the context (not supported yet).

Examples:

  • Create a new experiment with the default port 8080:

    nnictl create --config nni/examples/trials/mnist-pytorch/config.yml
    
  • Create a new experiment with specified port 8088:

    nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088
    
  • Create a new experiment with specified port 8088 and debug mode:

    nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088 --debug
    

Note

Debug mode will disable version check function in trial_keeper.

nnictl resume

Resume an experiment.

nnictl resume [-h] [--port PORT] [--debug] [--foreground]
              [--experiment_dir EXPERIMENT_DIR]
              id

Name / Shorthand

Required

Default

Description

id

True

The ID of the experiment you want to resume

--port, -p

False

8080

The port of restful server

--debug, -d

False

False

Set debug mode

--foreground, -f

False

False

Set foreground mode, print log content to terminal

--experiment_dir, -e

False

Resume experiment from external folder, specify the full path of experiment folder

You can use this command to resume a stopped experiment.

Example: resume an experiment with specified port 8088.

nnictl resume [experiment_id] --port 8088

nnictl view

View a stopped experiment.

nnictl view [-h] [--port PORT] [--experiment_dir EXPERIMENT_DIR] id

Name / Shorthand

Required

Default

Description

id

True

The ID of the experiment you want to view

--port, -p

False

8080

The port of restful server

--experiment_dir, -e

False

View experiment from external folder, specify the full path of experiment folder

nnictl update

Update the configuration of an experiment.

nnictl update [-h] {searchspace,concurrency,duration,trialnum} ...

update searchspace

Update the search space of an experiment.

nnictl update searchspace [-h] --filename FILENAME [id]

Name / Shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

--filename, -f

True

Path to new search space file

You can use this command to update an experiment’s search space.

Example: update experiment’s new search space with file dir examples/trials/mnist-pytorch/search_space.json.

nnictl update searchspace [experiment_id] --filename examples/trials/mnist-pytorch/search_space.json

update concurrency

Update the concurrency of an experiment.

nnictl update concurrency [-h] --value VALUE [id]

Name / Shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

--value, -v

True

The number of allowed concurrent trials

update duration

Update an experiment’s maximum allowed duration.

nnictl update duration [-h] --value VALUE [id]

Name / Shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

--value, -v

True

Strings like ‘1m’ for one minute or ‘2h’ for two hours. SUFFIX may be ‘s’ for seconds, ‘m’ for minutes, ‘h’ for hours or ‘d’ for days.

update trialnum

Update an experiment’s maximum trial number.

nnictl update trialnum [-h] --value VALUE [id]

Name / Shorthand

Required

Default

Description

id

False

ID of the experiment you want to set

--value, -v

True

The new number of maxtrialnum you want to set

nnictl stop

nnictl stop [-h] [--port PORT] [--all] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID you want to stop

--port, -p

False

The port of restful server you want to stop

--all, -a

False

False

Stop all the experiments

You can use this command to stop a running experiment or multiple experiments.

Details & Examples:

  • If there is no id specified, and there is an experiment running, stop the running experiment, or print error message.

    nnictl stop
    
  • If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.

    nnictl stop [experiment_id]
    
  • If there is a port specified, and an experiment is running on that port, the experiment will be stopped.

    nnictl stop --port 8080
    
  • Users could use nnictl stop --all to stop all experiments.

  • If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.

  • If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.

  • If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.

nnictl trial

Get information of trials.

nnictl trial [-h] {ls,kill} ...

trial ls

List trial jobs in one experiment.

nnictl trial ls [-h] [--head HEAD] [--tail TAIL] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

--head

False

The number of items to be listed with the highest default metric

--tail

False

The number of items to be listed with the lowest default metric

You can use this command to show trials in an experiment.

Note

If head or tail is set, only complete trials will be listed.

trial kill

Kill a trial job.

nnictl trial kill [-h] --trial_id TRIAL_ID [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

--trial_id, -T

True

The ID of trial to be killed

nnictl experiment

Get information of, or operate on experiments.

nnictl experiment [-h] {show,status,list,delete,import,export,save,load} ...

experiment show

Show the information of experiment.

nnictl experiment show [-h] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

experiment status

Show the status of experiment.

nnictl experiment status [-h] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

experiment list

Show the information of all the (running) experiments.

nnictl experiment list [-h] [--all]

Name / Shorthand

Required

Default

Description

--all

False

False

List all of experiments

experiment delete

Delete one or all experiments, it includes log, result, environment information and cache. It can be used to delete useless experiment result, or save disk space.

nnictl experiment delete [-h] [--all] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

--all

False

False

Delete all of experiments

experiment import

Import additional tuning data into an experiment.

nnictl experiment import [-h] --filename FILENAME [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

--filename, -f

True

A file with data you want to import in json format

You can use this command to import several prior or supplementary trial hyperparameters & results for NNI hyperparameter tuning. The data are fed to the tuning algorithm (e.g., tuner or advisor).

NNI supports users to import their own data, please express the data in the correct format. An example is shown below:

[
    {"parameter": {"x": 0.5, "y": 0.9}, "value": 0.03},
    {"parameter": {"x": 0.4, "y": 0.8}, "value": 0.05},
    {"parameter": {"x": 0.3, "y": 0.7}, "value": 0.04}
]

Every element in the top level list is a sample. For our built-in tuners/advisors, each sample should have at least two keys: parameter and value. The parameter must match this experiment’s search space, that is, all the keys (or hyperparameters) in parameter must match the keys in the search space. Otherwise, tuner/advisor may have unpredictable behavior. value should follow the same rule of the input in nni.report_final_result, that is, either a number or a dict with a key named default. For your customized tuner/advisor, the file could have any json content depending on how you implement the corresponding methods (e.g., import_data).

Note

You can’t see imported data on the web portal when you import data into the experiment. Because currently the import data only has the metric and hyper-parameters, while to be visible on the web portal, one trial must have a complete record, including fields like sequence ID, intermediate results and etc.

You also can use nnictl experiment export to export a valid json file including previous experiment trial hyperparameters and results.

Currently, the following tuners/advisors support import data:

  • TPE

  • Anneal

  • GridSearch

  • MetisTuner

  • BOHB

Note

If you want to import data to BOHB advisor, user are suggested to add TRIAL_BUDGET in parameter as NNI do, otherwise, BOHB will use max_budget as TRIAL_BUDGET. Here is an example:

[
    {"parameter": {"x": 0.5, "y": 0.9, "TRIAL_BUDGET": 27}, "value": 0.03}
]

experiment export

Export trial job results.

nnictl experiment export [-h] --type {json,csv} --filename PATH
                         [--intermediate]
                         [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

--type, -t

True

Possible choices: json, csv

Target file type

--filename, -f

True

File path of the output file

--intermediate, -i

False

False

Whether intermediate results are included

Export results for trial jobs in an experiment as json format.

Example:

nnictl experiment export [experiment_id] --filename [file_path] --type json --intermediate

Note

Import/export are used to deal with trial jobs in a structured format. If you are looking for ways to dump the whole experiment and continue running it on another machine, save/load might intrigue you.

experiment save

Dump the metadata and code data of an experiment into a package.

nnictl experiment save [-h] [--path PATH] [--saveCodeDir] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

--path, -p

False

The folder to store nni experiment data. Default: current working directory.

--saveCodeDir, -s

False

False

Copy code directory into the saved package.

experiment load

Load an experiment dumped with save command.

nnictl experiment load [-h] --path PATH --codeDir CODEDIR [--logDir LOGDIR]
                       [--searchSpacePath SEARCHSPACEPATH]

Name / Shorthand

Required

Default

Description

--path, -p

True

Path to the packaged experiment.

--codeDir, -c

True

Where to put the code for the loaded experiment. Code in the package will be unzipped here.

--logDir, -l

False

Path to logDir for the loaded experiment

--searchSpacePath, -s

False

The file path (not folder) to put the search space file for the loaded experiment. Default: $codeDir/search_space.json

nnictl platform

nnictl platform [-h] {clean} ...

platform clean

Clean up the specified platform.

nnictl platform clean [-h] --config CONFIG

Name / Shorthand

Required

Default

Description

--config, -c

True

Path to yaml config file used when creating an experiment on that platform.

Clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.

Note

If the target platform is being used by other users, it may cause unexpected errors to others.

nnictl webui

nnictl webui [-h] {url} ...

webui url

Show an experiment’s webui url.

nnictl webui url [-h] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

nnictl config

nnictl config [-h] {show} ...

config show

Show the config of an experiment.

nnictl config show [-h] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

nnictl log

Manage logs.

nnictl log [-h] {stdout,stderr,trial} ...

log stdout

Show the stdout log content.

nnictl log stdout [-h] [--tail TAIL] [--head HEAD] [--path] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

--tail, -T

False

Show tail lines of stdout

--head, -H

False

Show head lines of stdout

--path

False

False

Get the path of stdout file

log stderr

Show the stderr log content.

nnictl log stderr [-h] [--tail TAIL] [--head HEAD] [--path] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

--tail, -T

False

Show tail lines of stderr

--head, -H

False

Show head lines of stderr

--path

False

False

Get the path of stderr file

log trial

Show trial log path.

nnictl log trial [-h] [--trial_id TRIAL_ID] [id]

Name / Shorthand

Required

Default

Description

id

False

Experiment ID

--trial_id, -T

False

Trial ID to find the log path, required when experiment ID is set

nnictl algo

Manage algorithms.

nnictl algo [-h] {register,reg,unregister,unreg,show,list} ...

algo register (reg)

Register customized algorithms as builtin tuner/assessor/advisor.

nnictl algo register [-h] --meta_path META_PATH

Name / Shorthand

Required

Default

Description

--meta_path, -m

True

Path to the meta file

Register algorithms so that it can be used like a built-in algorithm.

META_PATH is the path to the meta data file in yml format, which has following keys:

  • algoType: type of algorithms, could be one of tuner, assessor, advisor.

  • builtinName: builtin name used in experiment configuration file.

  • className: tuner class name, including its module name, for example: demo_tuner.DemoTuner.

  • classArgsValidator: class args validator class name, including its module name, for example: demo_tuner.MyClassArgsValidator.

Example: install a customized tuner in nni examples.

cd nni/examples/tuners/customized_tuner
python3 setup.py develop
nnictl algo register -m meta_file.yml

algo unregister (unreg)

Unregister a registered customized builtin algorithms. The NNI-provided builtin algorithms can not be unregistered.

nnictl algo unregister [-h] name

Name / Shorthand

Required

Default

Description

name

True

Builtin name of the algorithm

algo show

Show the detailed information of specific registered algorithms.

nnictl algo show [-h] name

Name / Shorthand

Required

Default

Description

name

True

Builtin name of the algorithm

algo list

List the registered builtin algorithms.

nnictl algo list [-h]

nnictl trainingservice

(internal preview) Manage 3rd-party training services.

nnictl trainingservice [-h] {register,unregister,list} ...

trainingservice register

Register training service.

nnictl trainingservice register [-h] --package PACKAGE

Name / Shorthand

Required

Default

Description

--package

True

Package name

trainingservice unregister

Unregister training service.

nnictl trainingservice unregister [-h] --package PACKAGE

Name / Shorthand

Required

Default

Description

--package

True

Package name

trainingservice list

List custom training services.

nnictl trainingservice list [-h]

nnictl package

This argument is replaced by algo.

nnictl package [-h] ...

Name / Shorthand

Required

Default

Description

args

True

nnictl top

Monitor the list of all running experiments.

nnictl top [-h] [--time TIME]

Name / Shorthand

Required

Default

Description

--time, -t

False

3

The interval to update the experiment status. The unit of time is second, and the default value is 3 seconds.

nnictl ss_gen

(deprecated) Automatically generate search space file from trial code.

nnictl ss_gen [-h] --trial_command TRIAL_COMMAND [--trial_dir TRIAL_DIR]
              [--file FILE]

Name / Shorthand

Required

Default

Description

--trial_command, -t

True

The command for running trial code

--trial_dir, -d

False

./

The directory for running the command

--file, -f

False

nni_auto_gen_search_space.json

The path of search space file

nnictl jupyter-extension

(internal preview) Install or uninstall JupyterLab extension.

nnictl jupyter-extension [-h] {install,uninstall} ...

jupyter-extension install

Install JupyterLab extension.

nnictl jupyter-extension install [-h]

jupyter-extension uninstall

Uninstall JupyterLab extension.

nnictl jupyter-extension uninstall [-h]

nnictl hello

Create “hello nni” example in current directory.

nnictl hello [-h]