Efficient Training Under Limited Resources

As an initial step, we started with implementing a training pipeline. We based on a provided code from the challenge website, and we added new modules such as data preprocessing, model building, and trainer. In the preprocessing module, we added a data sampler to build a dataset requested by the competition: 5k images for the train set and 1k images for the validation set. We selected CIFAR-10 as a proxy dataset because it is the most common dataset with ten classes in classification. Its size allowed us to build various subsets by shuffling the selected image indices before subsampling the data and helped our candidate model generalize better. Model building module provides a designed model as we previewed a known model’s NAS variant. The trainer module runs the training loop for precisely ten minutes to simulate the competition environment.

The larger the search space, the longer and more expensive NAS will be. Efficient use of resources was a top priority for the challenge. Therefore, instead of starting an architecture design from scratch, the second step of our approach was finding a baseline network candidate from which we would start the search. We came up with a list of well-known classifiers, mainly the winner of the ImageNet ILSVRC challenge. We adapted the implementation of most of these classifiers from Kuang (2021). Then, we trained each classifier ten times with distinct subsets drawn from CIFAR-10. Training networks multiple times was important to smooth the variance effect induced by the small training sets. We trained all classifiers using our pipeline with 10 minutes training time budget and the same dataset. Other hyperparameters were similar to the respective paper of each model. We finally selected the top two models with the highest average accuracy. Table 1 shows that SENet with 77.0% and ResNet-18 with 75.5% of validation accuracy are on top of the list.

In order to reduce the training epoch time, we performed Neural Architecture Search on SENet and ResNet-18 by using Formulation 1. The goal of this third step was to find a smaller variant of the model that can perform higher or equal to the baseline model. We used the NOMAD (Le Digabel, 2011) blackbox optimization software which implements the MADS algorithm (Audet & Dennis, Jr., 2006). NOMAD allows optimizing an objective function under certain constraints without requiring knowledge about the internals of the function. We defined $g({\bm{\phi}})$ as a function that denotes the resource consumption, an explicit function such as the number of Multiply-Accumulate (MAC) operations. We also constrained the model’s performance to be higher or equal to the baseline one. We followed Howard et al. (2017) to scale the baseline model by defining three multipliers for three dimensions of the model: depth, width, and input resolution. The depth multiplier determines the number of blocks for a model. The role of the width multiplier is to thin a network by reducing the number of input and output channels of each layer uniformly. Input resolution multiplier scales the input image and subsequently the internal representation of every layer. For each NAS trial, we return the number of MAC calculated based on the new dimensions of the model and the accuracy of the model to NOMAD.

Where $\mathbf{w}$ is the neural networks weights, $\Phi$ denotes the hyperparameter space of the neural network, ${\bm{\phi}}\in\Phi$ represents a vector of values for depth, width, and resolution that define the network architecture, and $(\mathbf{x},\mathbf{y})$ the data features and labels. We note $f({\bm{\phi}},\mathbf{w}\mid\mathbf{x},\mathbf{y})$ the validation accuracy of the network after training; And $g({\bm{\phi}})$ the function that reflects the number of MAC operations.

Given the small size of the dataset, we observed that our candidate model is prone to overfitting. Since the dataset is tiny, the model does not gain enough performance during training by using the basic techniques of image augmentation. We used the AutoAugment (Cubuk et al., 2018) approach, which is an automatic way of selecting optimal data augmentation policies. Also, we added the CUTOUT (DeVries & Taylor, 2017) technique which consists of masking out random sections of input images during training.

Finally, we perform the HPO on our candidate model to find the best value of the hyperparameters by using NOMAD. Formulation 2 shows the objective function that we proposed.

Where $\lambda$ is a vector of values from hyperparameter space of $\Lambda$. $f(\lambda,\mathbf{w}\mid\mathbf{x},\mathbf{y})$ denotes the validation accuracy of the neural network. We selected three standard hyperparameters known to strongly influence validation set performance: initial learning rate, weight decay, and the optimizer type. Since the dataset is tiny, we chose to tune the batch size as well. As the range of values for learning rate is different from an optimizer to another, we adapted the learning rate based on an expected value of learning rate for each optimizer during training. For example, $0.1$ is usually used for SGD, but $0.001$ is used for ADAM. In the end, the search space of the HPO experiment was as follows:

• •

  Learning rate of uniform type and in [0.6;1E-3]

• •

  Weight decay of discrete type and in {0, 0.00005, 0.0005, 0.005, 0.05, 0.5}

• •

  Optimizer of discrete type and in {“Adadelta”(1) , “Adagrad” (0.01) , “SGD” (0.1), “Adam” (0.01), “Adamw” (0.01), “Adamax” (0.002), “ASGD”}

• •

  Batch size of discrete type and in {128, 256, 512}

The HPO increased the accuracy of our model by 1.1%.

CIFAR-10 is a collection of 60K color images of 32$\times$32, divided as follows: 50K for train set and 10K for validation set. It has ten classes. Since it is so close to the description of the competition evaluation dataset and its size is big enough to have several distinct subsamples, we decided to use it during the development phase as a proxy dataset. In all our experiments, the size of the dataset is as follows: the train set is 5K, and the validation set is 1K. Table 1 shows the intermediate results of our search to find a baseline model. All the classifiers are trained for ten minutes and on a subset of CIFAR-10. SENet-18 with 77.0% and ResNet-18 with 75.5% achieved better accuracy than other classifiers.

By applying NAS on SENet-18 and ResNet-18, we were able to find a smaller variant for each model that performs similarly to its baseline model. The NOMAD-NAS-SENet-18 has the same depth and input resolution as SENet-18, but the width of the network is 67% of the original one. The NOMAD-NAS-ResNet-18 has the same depth as ResNet-18, but its width and input resolution are 73% and 118% of the original ones, respectively. Although the input resolution is 18% bigger than its original model, the NOMAD-NAS-ResNet-18 has 22% less MAC operations and 47% fewer parameters than ResNet-18. We observed that both candidate models are prone to overfitting. We improved our data transformation list by adding AutoAugment and CUTOUT techniques. Table 2 shows the performance of SENet-18 and ResNet-18 after applying NAS and DA. After this step, we continued on hyperparameters optimization with NOMAD-NAS-SENet-18. Table 3 shows the validation accuracy of our candidate model, NOMAD-NAS-SENet-18 after applying HPO. We used NOMAD to find the best hyperparameter values of the model. To this end, we performed a blackbox optimization by using MADS algorithm over multiple parameters of our final candidate model. As described in 3.5, we came up with a set of tuned values for the hyperparameters: learning rate = 0.042, weight decay = 0.005, optimizer = SGD, batch size = 512. We performed several tests to ensure these best values can achieve optimal accuracy. We observed an improvement of validation accuracy from 86.7% to 87.8%.

After the competition deadline, the ICLR HAET Challenge committee announced the results and the name of the evaluation dataset. Table 4 shows the final result of our model on a subset of Mini-ImageNet. There is a considerable gap of 1.8% between our evaluation on CIFAR-10 and committee evaluation on Mini-ImageNet. Some other potential differences, such as hardware and training loop, can justify this gap. However, the main factor affecting our model’s performance is an extra operation in data transformation that wasn’t included in the challenge description. The images in Mini-ImageNet are 80$\times$80. The committee added a resize operation to the data transformation list in order to convert images to the initial announcement of 32$\times$32. We used Cosine Annealing for the learning rate decay scheduler with the max epoch of 240 for ten minutes. This extra data transformation is costly and delayed our training process for several epochs, consequently reducing our model’s performance. We found the results fair because this extra cost affects all other submissions.