Neural Architecture Search Comparison

Posted by Anonymous Author

Train and Compare NAS (Neural Architecture Search) models including Autokeras, DARTS, ENAS and NAO.

Their source code link is as below:

Experiment Description

To avoid over-fitting in CIFAR-10, we also compare the models in the other five datasets including Fashion-MNIST, CIFAR-100, OUI-Adience-Age, ImageNet-10-1 (subset of ImageNet), ImageNet-10-2 (another subset of ImageNet). We just sample a subset with 10 different labels from ImageNet to make ImageNet-10-1 or ImageNet-10-2.

Dataset

Training Size

Numer of Classes

Descriptions

Fashion-MNIST

60,000

10

T-shirt/top, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag and ankle boot.

CIFAR-10

50,000

10

Airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks.

CIFAR-100

50,000

100

Similar to CIFAR-10 but with 100 classes and 600 images each.

OUI-Adience-Age

26,580

8

8 age groups/labels (0-2, 4-6, 8-13, 15-20, 25-32, 38-43, 48-53, 60-).

ImageNet-10-1

9,750

10

Coffee mug, computer keyboard, dining table, wardrobe, lawn mower, microphone, swing, sewing machine, odometer and gas pump.

ImageNet-10-2

9,750

10

Drum, banj, whistle, grand piano, violin, organ, acoustic guitar, trombone, flute and sax.

We do not change the default fine-tuning technique in their source code. In order to match each task, the codes of input image shape and output numbers are changed.

Search phase time for all NAS methods is two days as well as the retrain time. Average results are reported based on three repeat times. Our evaluation machines have one Nvidia Tesla P100 GPU, 112GB of RAM and one 2.60GHz CPU (Intel E5-2690).

For NAO, it requires too much computing resources, so we only use NAO-WS which provides the pipeline script.

For AutoKeras, we used 0.2.18 version because it was the latest version when we started the experiment.

NAS Performance

NAS

AutoKeras (%)

ENAS (macro) (%)

ENAS (micro) (%)

DARTS (%)

NAO-WS (%)

Fashion-MNIST

91.84

95.44

95.53

95.74

95.20

CIFAR-10

75.78

95.68

96.16

94.23

95.64

CIFAR-100

43.61

78.13

78.84

79.74

75.75

OUI-Adience-Age

63.20

80.34

78.55

76.83

72.96

ImageNet-10-1

61.80

77.07

79.80

80.48

77.20

ImageNet-10-2

37.20

58.13

56.47

60.53

61.20

Unfortunately, we cannot reproduce all the results in the paper.

The best or average results reported in the paper:

NAS

AutoKeras(%)

ENAS (macro) (%)

ENAS (micro) (%)

DARTS (%)

NAO-WS (%)

CIFAR- 10

88.56(best)

96.13(best)

97.11(best)

97.17(average)

96.47(best)

For AutoKeras, it has relatively worse performance across all datasets due to its random factor on network morphism.

For ENAS, ENAS (macro) shows good results in OUI-Adience-Age and ENAS (micro) shows good results in CIFAR-10.

For DARTS, it has a good performance on some datasets but we found its high variance in other datasets. The difference among three runs of benchmarks can be up to 5.37% in OUI-Adience-Age and 4.36% in ImageNet-10-1.

For NAO-WS, it shows good results in ImageNet-10-2 but it can perform very poorly in OUI-Adience-Age.

Reference

  1. Jin, Haifeng, Qingquan Song, and Xia Hu. “Efficient neural architecture search with network morphism.” arXiv preprint arXiv:1806.10282 (2018).

  2. Liu, Hanxiao, Karen Simonyan, and Yiming Yang. “Darts: Differentiable architecture search.” arXiv preprint arXiv:1806.09055 (2018).

  3. Pham, Hieu, et al. “Efficient Neural Architecture Search via Parameters Sharing.” international conference on machine learning (2018): 4092-4101.

  4. Luo, Renqian, et al. “Neural Architecture Optimization.” neural information processing systems (2018): 7827-7838.