Retiarii for Neural Architecture Search¶
注意
NNI's latest NAS supports are all based on Retiarii Framework, users who are still on early version using NNI NAS v1.0 shall migrate your work to Retiarii as soon as possible.
Contents
Motivation¶
Automatic neural architecture search is playing an increasingly important role in finding better models. Recent research has proven the feasibility of automatic NAS and has led to models that beat many manually designed and tuned models. Representative works include NASNet, ENAS, DARTS, Network Morphism, and Evolution. In addition, new innovations continue to emerge.
However, it is pretty hard to use existing NAS work to help develop common DNN models. Therefore, we designed Retiarii, a novel NAS/HPO framework, and implemented it in NNI. It helps users easily construct a model space (or search space, tuning space), and utilize existing NAS algorithms. The framework also facilitates NAS innovation and is used to design new NAS algorithms.
Overview¶
There are three key characteristics of the Retiarii framework:
Simple APIs are provided for defining model search space within PyTorch/TensorFlow model.
SOTA NAS algorithms are built-in to be used for exploring model search space.
System-level optimizations are implemented for speeding up the exploration.
There are two types of model space exploration approach: Multi-trial NAS and One-shot NAS. Mutli-trial NAS trains each sampled model in the model space independently, while One-shot NAS samples the model from a super model. After constructing the model space, users can use either exploration appraoch to explore the model space.
Multi-trial NAS¶
Multi-trial NAS means each sampled model from model space is trained independently. A typical multi-trial NAS is NASNet. The algorithm to sample models from model space is called exploration strategy. NNI has supported the following exploration strategies for multi-trial NAS.
Exploration Strategy Name |
Brief Introduction of Algorithm |
|---|---|
Random Strategy |
Randomly sampling new model(s) from user defined model space. ( |
Grid Search |
Sampling new model(s) from user defined model space using grid search algorithm. ( |
Regularized Evolution |
Generating new model(s) from generated models using regularized evolution algorithm . ( |
TPE Strategy |
Sampling new model(s) from user defined model space using TPE algorithm . ( |
RL Strategy |
It uses PPO algorithm to sample new model(s) from user defined model space. ( |
Please refer to here for detailed usage of multi-trial NAS.
One-shot NAS¶
One-shot NAS means building model space into a super-model, training the super-model with weight sharing, and then sampling models from the super-model to find the best one. DARTS is a typical one-shot NAS. Below is the supported one-shot NAS algorithms. More one-shot NAS will be supported soon.
One-shot Algorithm Name |
Brief Introduction of Algorithm |
|---|---|
Efficient Neural Architecture Search via Parameter Sharing. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. It uses parameter sharing between child models to achieve fast speed and excellent performance. |
|
DARTS: Differentiable Architecture Search introduces a novel algorithm for differentiable network architecture search on bilevel optimization. |
|
Single Path One-Shot Neural Architecture Search with Uniform Sampling constructs a simplified supernet trained with a uniform path sampling method and applies an evolutionary algorithm to efficiently search for the best-performing architectures. |
|
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. It removes proxy, directly learns the architectures for large-scale target tasks and target hardware platforms. |
Please refer to here for detailed usage of one-shot NAS algorithms.
Reference and Feedback¶
To report a bug for this feature in GitHub ;
To file a feature or improvement request for this feature in GitHub .