nnictl Commands¶
nnictl is a command line tool, used to control experiments, such as start/stop/resume an experiment, start/stop WebUI, etc.
usage: nnictl [-h] [--version]
{create,resume,view,update,stop,trial,experiment,platform,webui,config,log,algo,trainingservice,package,top,ss_gen,jupyter-extension,hello}
...
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--version, -v |
False |
False |
Describe the current version of NNI installed |
nnictl create¶
Create a new experiment.
nnictl create [-h] --config CONFIG [--port PORT] [--debug]
[--url_prefix URL_PREFIX] [--foreground]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--config, -c |
True |
|
Path to YAML configuration file of the experiment |
--port, -p |
False |
8080 |
The port of restful server |
--debug, -d |
False |
False |
Set debug mode |
--url_prefix, -u |
False |
|
Set prefix url |
--foreground, -f |
False |
False |
Set foreground mode, print log content to terminal |
You can use this command to create a new experiment, using the configuration specified in config file.
After this command is successfully done, the context will be set as this experiment, which means the following command you issued is associated with this experiment, unless you explicitly change the context (not supported yet).
Examples:
Create a new experiment with the default port 8080:
nnictl create --config nni/examples/trials/mnist-pytorch/config.yml
Create a new experiment with specified port 8088:
nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088
Create a new experiment with specified port 8088 and debug mode:
nnictl create --config nni/examples/trials/mnist-pytorch/config.yml --port 8088 --debug
Note
Debug mode will disable version check function in trial_keeper
.
nnictl resume¶
Resume an experiment.
nnictl resume [-h] [--port PORT] [--debug] [--foreground]
[--experiment_dir EXPERIMENT_DIR]
id
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
True |
|
The ID of the experiment you want to resume |
--port, -p |
False |
8080 |
The port of restful server |
--debug, -d |
False |
False |
Set debug mode |
--foreground, -f |
False |
False |
Set foreground mode, print log content to terminal |
--experiment_dir, -e |
False |
|
Resume experiment from external folder, specify the full path of experiment folder |
You can use this command to resume a stopped experiment.
Example: resume an experiment with specified port 8088.
nnictl resume [experiment_id] --port 8088
nnictl view¶
View a stopped experiment.
nnictl view [-h] [--port PORT] [--experiment_dir EXPERIMENT_DIR] id
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
True |
|
The ID of the experiment you want to view |
--port, -p |
False |
8080 |
The port of restful server |
--experiment_dir, -e |
False |
|
View experiment from external folder, specify the full path of experiment folder |
nnictl update¶
Update the configuration of an experiment.
nnictl update [-h] {searchspace,concurrency,duration,trialnum} ...
update searchspace¶
Update the search space of an experiment.
nnictl update searchspace [-h] --filename FILENAME [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
ID of the experiment you want to set |
--filename, -f |
True |
|
Path to new search space file |
You can use this command to update an experiment’s search space.
Example: update experiment’s new search space with file dir examples/trials/mnist-pytorch/search_space.json.
nnictl update searchspace [experiment_id] --filename examples/trials/mnist-pytorch/search_space.json
update concurrency¶
Update the concurrency of an experiment.
nnictl update concurrency [-h] --value VALUE [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
ID of the experiment you want to set |
--value, -v |
True |
|
The number of allowed concurrent trials |
update duration¶
Update an experiment’s maximum allowed duration.
nnictl update duration [-h] --value VALUE [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
ID of the experiment you want to set |
--value, -v |
True |
|
Strings like ‘1m’ for one minute or ‘2h’ for two hours. SUFFIX may be ‘s’ for seconds, ‘m’ for minutes, ‘h’ for hours or ‘d’ for days. |
update trialnum¶
Update an experiment’s maximum trial number.
nnictl update trialnum [-h] --value VALUE [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
ID of the experiment you want to set |
--value, -v |
True |
|
The new number of maxtrialnum you want to set |
nnictl stop¶
nnictl stop [-h] [--port PORT] [--all] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID you want to stop |
--port, -p |
False |
|
The port of restful server you want to stop |
--all, -a |
False |
False |
Stop all the experiments |
You can use this command to stop a running experiment or multiple experiments.
Details & Examples:
If there is no id specified, and there is an experiment running, stop the running experiment, or print error message.
nnictl stop
If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.
nnictl stop [experiment_id]
If there is a port specified, and an experiment is running on that port, the experiment will be stopped.
nnictl stop --port 8080
Users could use
nnictl stop --all
to stop all experiments.If the id ends with
*
, nnictl will stop all experiments whose ids matchs the regular.If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.
If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.
nnictl trial¶
Get information of trials.
nnictl trial [-h] {ls,kill,codegen} ...
trial ls¶
List trial jobs in one experiment.
nnictl trial ls [-h] [--head HEAD] [--tail TAIL] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--head |
False |
|
The number of items to be listed with the highest default metric |
--tail |
False |
|
The number of items to be listed with the lowest default metric |
You can use this command to show trials in an experiment.
Note
If head
or tail
is set, only complete trials will be listed.
trial kill¶
Kill a trial job.
nnictl trial kill [-h] --trial_id TRIAL_ID [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--trial_id, -T |
True |
|
The ID of trial to be killed |
trial codegen¶
Generate trial code for a specific trial. Useful in annotation mode.
nnictl trial codegen [-h] --trial_id TRIAL_ID [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--trial_id, -T |
True |
|
The ID of trial to do code generation |
nnictl experiment¶
Get information of, or operate on experiments.
nnictl experiment [-h] {show,status,list,delete,import,export,save,load} ...
experiment show¶
Show the information of experiment.
nnictl experiment show [-h] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
experiment status¶
Show the status of experiment.
nnictl experiment status [-h] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
experiment list¶
Show the information of all the (running) experiments.
nnictl experiment list [-h] [--all]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--all |
False |
False |
List all of experiments |
experiment delete¶
Delete one or all experiments, it includes log, result, environment information and cache. It can be used to delete useless experiment result, or save disk space.
nnictl experiment delete [-h] [--all] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--all |
False |
False |
Delete all of experiments |
experiment import¶
Import additional tuning data into an experiment.
nnictl experiment import [-h] --filename FILENAME [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--filename, -f |
True |
|
A file with data you want to import in json format |
You can use this command to import several prior or supplementary trial hyperparameters & results for NNI hyperparameter tuning. The data are fed to the tuning algorithm (e.g., tuner or advisor).
NNI supports users to import their own data, please express the data in the correct format. An example is shown below:
[
{"parameter": {"x": 0.5, "y": 0.9}, "value": 0.03},
{"parameter": {"x": 0.4, "y": 0.8}, "value": 0.05},
{"parameter": {"x": 0.3, "y": 0.7}, "value": 0.04}
]
Every element in the top level list is a sample.
For our built-in tuners/advisors, each sample should have at least two keys: parameter
and value
.
The parameter
must match this experiment’s search space, that is,
all the keys (or hyperparameters) in parameter
must match the keys in the search space.
Otherwise, tuner/advisor may have unpredictable behavior.
value
should follow the same rule of the input in nni.report_final_result
,
that is, either a number or a dict with a key named default
.
For your customized tuner/advisor, the file could have any json content depending on
how you implement the corresponding methods (e.g., import_data
).
You also can use nnictl experiment export to export a valid json file including previous experiment trial hyperparameters and results.
Currently, the following tuners/advisors support import data:
TPE
Anneal
GridSearch
MetisTuner
BOHB
Note
If you want to import data to BOHB advisor, user are suggested to add TRIAL_BUDGET
in parameter as NNI do,
otherwise, BOHB will use max_budget as TRIAL_BUDGET
. Here is an example:
[
{"parameter": {"x": 0.5, "y": 0.9, "TRIAL_BUDGET": 27}, "value": 0.03}
]
experiment export¶
Export trial job results.
nnictl experiment export [-h] --type {json,csv} --filename PATH
[--intermediate]
[id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--type, -t |
True |
|
Possible choices: json, csv Target file type |
--filename, -f |
True |
|
File path of the output file |
--intermediate, -i |
False |
False |
Whether intermediate results are included |
Export results for trial jobs in an experiment as json format.
Example:
nnictl experiment export [experiment_id] --filename [file_path] --type json --intermediate
Note
Import/export are used to deal with trial jobs in a structured format. If you are looking for ways to dump the whole experiment and continue running it on another machine, save/load might intrigue you.
experiment save¶
Dump the metadata and code data of an experiment into a package.
nnictl experiment save [-h] [--path PATH] [--saveCodeDir] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--path, -p |
False |
|
The folder to store nni experiment data. Default: current working directory. |
--saveCodeDir, -s |
False |
False |
Copy code directory into the saved package. |
experiment load¶
Load an experiment dumped with save
command.
nnictl experiment load [-h] --path PATH --codeDir CODEDIR [--logDir LOGDIR]
[--searchSpacePath SEARCHSPACEPATH]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--path, -p |
True |
|
Path to the packaged experiment. |
--codeDir, -c |
True |
|
Where to put the code for the loaded experiment. Code in the package will be unzipped here. |
--logDir, -l |
False |
|
Path to |
--searchSpacePath, -s |
False |
|
The file path (not folder) to put the search space file for the loaded experiment. Default: |
nnictl platform¶
nnictl platform [-h] {clean} ...
platform clean¶
Clean up the specified platform.
nnictl platform clean [-h] --config CONFIG
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--config, -c |
True |
|
Path to yaml config file used when creating an experiment on that platform. |
Clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.
Note
If the target platform is being used by other users, it may cause unexpected errors to others.
nnictl webui¶
nnictl webui [-h] {url} ...
webui url¶
Show an experiment’s webui url.
nnictl webui url [-h] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
nnictl config¶
nnictl config [-h] {show} ...
config show¶
Show the config of an experiment.
nnictl config show [-h] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
nnictl log¶
Manage logs.
nnictl log [-h] {stdout,stderr,trial} ...
log stdout¶
Show the stdout log content.
nnictl log stdout [-h] [--tail TAIL] [--head HEAD] [--path] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--tail, -T |
False |
|
Show tail lines of stdout |
--head, -H |
False |
|
Show head lines of stdout |
--path |
False |
False |
Get the path of stdout file |
log stderr¶
Show the stderr log content.
nnictl log stderr [-h] [--tail TAIL] [--head HEAD] [--path] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--tail, -T |
False |
|
Show tail lines of stderr |
--head, -H |
False |
|
Show head lines of stderr |
--path |
False |
False |
Get the path of stderr file |
log trial¶
Show trial log path.
nnictl log trial [-h] [--trial_id TRIAL_ID] [id]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
id |
False |
|
Experiment ID |
--trial_id, -T |
False |
|
Trial ID to find the log path, required when experiment ID is set |
nnictl algo¶
Manage algorithms.
nnictl algo [-h] {register,reg,unregister,unreg,show,list} ...
algo register (reg)¶
Register customized algorithms as builtin tuner/assessor/advisor.
nnictl algo register [-h] --meta_path META_PATH
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--meta_path, -m |
True |
|
Path to the meta file |
Register algorithms so that it can be used like a built-in algorithm.
META_PATH
is the path to the meta data file in yml format, which has following keys:
algoType
: type of algorithms, could be one oftuner
,assessor
,advisor
.builtinName
: builtin name used in experiment configuration file.className
: tuner class name, including its module name, for example:demo_tuner.DemoTuner
.classArgsValidator
: class args validator class name, including its module name, for example:demo_tuner.MyClassArgsValidator
.
Example: install a customized tuner in nni examples.
cd nni/examples/tuners/customized_tuner
python3 setup.py develop
nnictl algo register -m meta_file.yml
algo unregister (unreg)¶
Unregister a registered customized builtin algorithms. The NNI-provided builtin algorithms can not be unregistered.
nnictl algo unregister [-h] name
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
name |
True |
|
Builtin name of the algorithm |
algo show¶
Show the detailed information of specific registered algorithms.
nnictl algo show [-h] name
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
name |
True |
|
Builtin name of the algorithm |
algo list¶
List the registered builtin algorithms.
nnictl algo list [-h]
nnictl trainingservice¶
(internal preview) Manage 3rd-party training services.
nnictl trainingservice [-h] {register,unregister,list} ...
trainingservice register¶
Register training service.
nnictl trainingservice register [-h] --package PACKAGE
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--package |
True |
|
Package name |
trainingservice unregister¶
Unregister training service.
nnictl trainingservice unregister [-h] --package PACKAGE
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--package |
True |
|
Package name |
trainingservice list¶
List custom training services.
nnictl trainingservice list [-h]
nnictl package¶
This argument is replaced by algo.
nnictl package [-h] ...
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
args |
True |
|
nnictl top¶
Monitor the list of all running experiments.
nnictl top [-h] [--time TIME]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--time, -t |
False |
3 |
The interval to update the experiment status. The unit of time is second, and the default value is 3 seconds. |
nnictl ss_gen¶
(deprecated) Automatically generate search space file from trial code.
nnictl ss_gen [-h] --trial_command TRIAL_COMMAND [--trial_dir TRIAL_DIR]
[--file FILE]
Name / Shorthand |
Required |
Default |
Description |
---|---|---|---|
--trial_command, -t |
True |
|
The command for running trial code |
--trial_dir, -d |
False |
./ |
The directory for running the command |
--file, -f |
False |
nni_auto_gen_search_space.json |
The path of search space file |
nnictl jupyter-extension¶
(internal preview) Install or uninstall JupyterLab extension.
nnictl jupyter-extension [-h] {install,uninstall} ...
jupyter-extension install¶
Install JupyterLab extension.
nnictl jupyter-extension install [-h]
jupyter-extension uninstall¶
Uninstall JupyterLab extension.
nnictl jupyter-extension uninstall [-h]
nnictl hello¶
Create “hello nni” example in current directory.
nnictl hello [-h]