nnictl¶
Introduction¶
nnictl is a command line tool, which can be used to control experiments, such as start/stop/resume an experiment, start/stop NNIBoard, etc.
Commands¶
nnictl support commands:
Manage an experiment¶
nnictl create¶
Description
You can use this command to create a new experiment, using the configuration specified in config file.
After this command is successfully done, the context will be set as this experiment, which means the following command you issued is associated with this experiment, unless you explicitly changes the context(not supported yet).
Usage
nnictl create [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
–config, -c |
True |
YAML configure file of the experiment |
|
–port, -p |
False |
the port of restful server |
|
–debug, -d |
False |
set debug mode |
|
–foreground, -f |
False |
set foreground mode, print log content to terminal |
Examples
create a new experiment with the default port: 8080
nnictl create --config nni/examples/trials/mnist-pytorch/config.yml
create a new experiment with specified port 8088
nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088
create a new experiment with specified port 8088 and debug mode
nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088 --debug
Note:
Debug mode will disable version check function in Trialkeeper.
nnictl resume¶
Description
You can use this command to resume a stopped experiment.
Usage
nnictl resume [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
True |
The id of the experiment you want to resume |
|
–port, -p |
False |
Rest port of the experiment you want to resume |
|
–debug, -d |
False |
set debug mode |
|
–foreground, -f |
False |
set foreground mode, print log content to terminal |
|
–experiment_dir, -e |
False |
Resume experiment from external folder, specify the full path of experiment folder |
Example
resume an experiment with specified port 8088
nnictl resume [experiment_id] --port 8088
nnictl view¶
Description
You can use this command to view a stopped experiment.
Usage
nnictl view [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
True |
The id of the experiment you want to view |
|
–port, -p |
False |
Rest port of the experiment you want to view |
|
–experiment_dir, -e |
False |
View experiment from external folder, specify the full path of experiment folder |
Example
view an experiment with specified port 8088
nnictl view [experiment_id] --port 8088
nnictl stop¶
Description
You can use this command to stop a running experiment or multiple experiments.
Usage
nnictl stop [Options]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
The id of the experiment you want to stop |
|
–port, -p |
False |
Rest port of the experiment you want to stop |
|
–all, -a |
False |
Stop all of experiments |
Details & Examples
If there is no id specified, and there is an experiment running, stop the running experiment, or print error message.
nnictl stop
If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.
nnictl stop [experiment_id]
If there is a port specified, and an experiment is running on that port, the experiment will be stopped.
nnictl stop --port 8080
Users could use ‘nnictl stop –all’ to stop all experiments.
nnictl stop --all
If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.
If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.
If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.
nnictl update¶
nnictl update searchspace
Description
You can use this command to update an experiment’s search space.
Usage
nnictl update searchspace [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
|
–filename, -f |
True |
the file storing your new search space |
Example
update experiment's new search space with file dir 'examples/trials/mnist-pytorch/search_space.json'
nnictl update searchspace [experiment_id] --filename examples/trials/mnist-pytorch/search_space.json
nnictl update concurrency
Description
You can use this command to update an experiment’s concurrency.
Usage
nnictl update concurrency [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
|
–value, -v |
True |
the number of allowed concurrent trials |
Example
update experiment’s concurrency
nnictl update concurrency [experiment_id] --value [concurrency_number]
nnictl update duration
Description
You can use this command to update an experiment’s duration.
Usage
nnictl update duration [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
|
–value, -v |
True |
Strings like ‘1m’ for one minute or ‘2h’ for two hours. SUFFIX may be ‘s’ for seconds, ‘m’ for minutes, ‘h’ for hours or ‘d’ for days. |
Example
update experiment’s duration
nnictl update duration [experiment_id] --value [duration]
nnictl update trialnum
Description
You can use this command to update an experiment’s maxtrialnum.
Usage
nnictl update trialnum [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
|
–value, -v |
True |
the new number of maxtrialnum you want to set |
Example
update experiment’s trial num
nnictl update trialnum [experiment_id] --value [trial_num]
nnictl trial¶
nnictl trial ls
Description
You can use this command to show trial’s information. Note that if
head
ortail
is set, only complete trials will be listed.Usage
nnictl trial ls nnictl trial ls --head 10 nnictl trial ls --tail 10
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
|
–head |
False |
the number of items to be listed with the highest default metric |
|
–tail |
False |
the number of items to be listed with the lowest default metric |
nnictl trial kill
Description
You can use this command to kill a trial job.
Usage
nnictl trial kill [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
Experiment ID of the trial |
|
–trial_id, -T |
True |
ID of the trial you want to kill. |
Example
kill trail job
nnictl trial kill [experiment_id] --trial_id [trial_id]
nnictl top¶
Description
Monitor all of running experiments.
Usage
nnictl top
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
|
–time, -t |
False |
The interval to update the experiment status, the unit of time is second, and the default value is 3 second. |
Manage experiment information¶
nnictl experiment show
Description
Show the information of experiment.
Usage
nnictl experiment show
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
nnictl experiment status
Description
Show the status of experiment.
Usage
nnictl experiment status
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
nnictl experiment list
Description
Show the information of all the (running) experiments.
Usage
nnictl experiment list [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
–all |
False |
list all of experiments |
nnictl experiment delete
Description
Delete one or all experiments, it includes log, result, environment information and cache. It uses to delete useless experiment result, or save disk space.
Usage
nnictl experiment delete [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment |
|
–all |
False |
delete all of experiments |
nnictl experiment export
Description
You can use this command to export reward & hyper-parameter of trial jobs to a csv file.
Usage
nnictl experiment export [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment |
|
–filename, -f |
True |
File path of the output file |
|
–type |
True |
Type of output file, only support “csv” and “json” |
|
–intermediate, -i |
False |
Are intermediate results included |
Examples
export all trial data in an experiment as json format
nnictl experiment export [experiment_id] --filename [file_path] --type json --intermediate
nnictl experiment import
Description
You can use this command to import several prior or supplementary trial hyperparameters & results for NNI hyperparameter tuning. The data are fed to the tuning algorithm (e.g., tuner or advisor).
Usage
nnictl experiment import [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
The id of the experiment you want to import data into |
|
–filename, -f |
True |
a file with data you want to import in json format |
Details
NNI supports users to import their own data, please express the data in the correct format. An example is shown below:
[ {"parameter": {"x": 0.5, "y": 0.9}, "value": 0.03}, {"parameter": {"x": 0.4, "y": 0.8}, "value": 0.05}, {"parameter": {"x": 0.3, "y": 0.7}, "value": 0.04} ]
Every element in the top level list is a sample. For our built-in tuners/advisors, each sample should have at least two keys:
parameter
andvalue
. Theparameter
must match this experiment’s search space, that is, all the keys (or hyperparameters) inparameter
must match the keys in the search space. Otherwise, tuner/advisor may have unpredictable behavior.Value
should follow the same rule of the input innni.report_final_result
, that is, either a number or a dict with a key nameddefault
. For your customized tuner/advisor, the file could have any json content depending on how you implement the corresponding methods (e.g.,import_data
).You also can use nnictl experiment export to export a valid json file including previous experiment trial hyperparameters and results.
Currently, following tuner and advisor support import data:
builtinTunerName: TPE, Anneal, GridSearch, MetisTuner builtinAdvisorName: BOHB
If you want to import data to BOHB advisor, user are suggested to add “TRIAL_BUDGET” in parameter as NNI do, otherwise, BOHB will use max_budget as “TRIAL_BUDGET”. Here is an example:
[ {"parameter": {"x": 0.5, "y": 0.9, "TRIAL_BUDGET": 27}, "value": 0.03} ]
Examples
import data to a running experiment
nnictl experiment import [experiment_id] -f experiment_data.json
nnictl experiment save
Description
Save nni experiment metadata and code data.
Usage
nnictl experiment save [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
True |
The id of the experiment you want to save |
|
–path, -p |
False |
the folder path to store nni experiment data, default current working directory |
|
–saveCodeDir, -s |
False |
save codeDir data of the experiment, default False |
Examples
save an expeirment
nnictl experiment save [experiment_id] --saveCodeDir
nnictl experiment load
Description
Load an nni experiment.
Usage
nnictl experiment load [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
–path, -p |
True |
the file path of nni package |
|
–codeDir, -c |
True |
the path of codeDir for loaded experiment, this path will also put the code in the loaded experiment package |
|
–logDir, -l |
False |
the path of logDir for loaded experiment |
|
–searchSpacePath, -s |
True |
the path of search space file for loaded experiment, this path contains file name. Default in $codeDir/search_space.json |
Examples
load an expeirment
nnictl experiment load --path [path] --codeDir [codeDir]
Manage platform information¶
nnictl platform clean
Description
It uses to clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.
Note
if the target platform is being used by other users, it may cause unexpected errors to others.
Usage
nnictl platform clean [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
–config |
True |
the path of yaml config file used when create an experiment |
nnictl config show¶
Description
Display the current context information.
Usage
nnictl config show
Manage log¶
nnictl log stdout
Description
Show the stdout log content.
Usage
nnictl log stdout [options]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
|
–head, -h |
False |
show head lines of stdout |
|
–tail, -t |
False |
show tail lines of stdout |
|
–path, -p |
False |
show the path of stdout file |
Example
Show the tail of stdout log content
nnictl log stdout [experiment_id] --tail [lines_number]
nnictl log stderr
Description
Show the stderr log content.
Usage
nnictl log stderr [options]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
ID of the experiment you want to set |
|
–head, -h |
False |
show head lines of stderr |
|
–tail, -t |
False |
show tail lines of stderr |
|
–path, -p |
False |
show the path of stderr file |
nnictl log trial
Description
Show trial log path.
Usage
nnictl log trial [options]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
Experiment ID of the trial |
|
–trial_id, -T |
False |
ID of the trial to be found the log path, required when id is not empty. |
Manage webui¶
nnictl webui url
Description
Show an experiment’s webui url
Usage
nnictl webui url [options]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
Experiment ID |
Manage builtin algorithms¶
nnictl algo register
Description
Register customized algorithms as builtin tuner/assessor/advisor.
Usage
nnictl algo register --meta <path_to_meta_file>
<path_to_meta_file>
is the path to the meta data file in yml format, which has following keys:algoType
: type of algorithms, could be one oftuner
,assessor
,advisor
builtinName
: builtin name used in experiment configuration fileclassName
: tuner class name, including its module name, for example:demo_tuner.DemoTuner
classArgsValidator
: class args validator class name, including its module name, for example:demo_tuner.MyClassArgsValidator
Example
Install a customized tuner in nni examples
cd nni/examples/tuners/customized_tuner python3 setup.py develop nnictl algo register --meta meta_file.yml
nnictl algo show
Description
Show the detailed information of specified registered algorithms.
Usage
nnictl algo show <builtinName>
Example
nnictl algo show SMAC
nnictl package list
Description
List the registered builtin algorithms.
Usage
nnictl algo list
Example
nnictl algo list
nnictl algo unregister
Description
Unregister a registered customized builtin algorithms. The NNI provided builtin algorithms can not be unregistered.
Usage
nnictl algo unregister <builtinName>
Example
nnictl algo unregister demotuner
Generate search space¶
nnictl ss_gen
Description
Generate search space from user trial code which uses NNI NAS APIs.
Usage
nnictl ss_gen [OPTIONS]
Options
Name, shorthand |
Required |
Default |
Description |
---|---|---|---|
–trial_command |
True |
The command of the trial code |
|
–trial_dir |
False |
./ |
The directory of the trial code |
–file |
False |
nni_auto_gen_search_space.json |
The file for storing generated search space |
Example
Generate a search space
nnictl ss_gen --trial_command="python3 mnist.py" --trial_dir=./ --file=ss.json
Check NNI version¶
nnictl –version
Description
Describe the current version of NNI installed.
Usage
nnictl --version