Construct Model SpaceΒΆ

NNI provides powerful (and multi-level) APIs for users to easily express model space (or search space).

  • Mutation Primitives: high-level APIs (e.g., ValueChoice, LayerChoice) that are utilities to build blocks in search space. In most cases, mutation pritimives should be straightforward yet expressive enough. We strongly recommend users to try them first, and report issues if those APIs are not satisfying.

  • Hyper-module Library: plug-and-play modules that are proved useful. They are usually well studied in research, and comes with pre-searched results. (For example, the optimal activation function in AutoActivation is reported to be Swish).

  • Mutator: for advanced users only. NNI provides interface to customize new mutators for expressing more complicated model spaces.

The following table summarizes all the APIs we have provided for constructing search space.

Name

Category

Brief Description

LayerChoice

Mutation Primitives

Select from some PyTorch modules

InputChoice

Mutation Primitives

Select from some inputs (tensors)

ValueChoice

Mutation Primitives

Select from some candidate values

Repeat

Mutation Primitives

Repeat a block by a variable number of times

Cell

Mutation Primitives

Cell structure popularly used in literature

NasBench101Cell

Mutation Primitives

Cell structure (variant) proposed by NAS-Bench-101

NasBench201Cell

Mutation Primitives

Cell structure (variant) proposed by NAS-Bench-201

AutoActivation

Hyper-modules Library

Searching for activation functions

Mutator

Mutator

Flexible mutations on graphs. See tutorial here