Discovering Robust Convolutional Architecture at Targeted Capacity:
A Multi-Shot Approach

In the inner search process (Line 23), we run a tournament-based evolutionary search with population size 20 and tournament size 5. We conduct $N^{p}\times 50=25k$ evolutionary search steps, and in each step, a mutated topology is assessed by the predictor, which is very efficient. And in every 50 steps, a topology with the highest predicted score in the current population is decided for the actual multi-shot evaluation. In each step, the topology with the lowest predicted score is removed from the population.

As for the predictor construction, we adapt a recent graph-based encoder [22] to our search space. Specifically, we encode the topology of each cell into a continuous vector, and then concatenate the embeddings of four cell topologies (S0, S1, S2, R) as the architecture embedding. Then the architecture embedding is fed into an MLP to get a predicted score. The operation embedding dimension, node embedding dimension, and hidden dimension are all 48. And the output dimension of each cell topology is set to 32. Thus, the dimension of one full topology is 128. Then this 128-dim vector is fed into a 3-layer MLP with 256 hidden units and output a final score. In each predictor training process, the predictor is trained for 100 epochs with batch size 50 and an Adam optimizer (1e-3 learning rate). We use hinge pair-wise ranking loss with margin $m=0.1$ to train the predictor following [22]. A dropout of 0.1 before the MLP is used.

For the final comparison of all the architectures, we adversarially train them for 110 epochs on CIFAR-10/CIFAR-100 and 50 epochs on SVHN. As for the adversarial attack, we adopt the PGD⁷ attack with $\epsilon=0.031(8/255)$ and step size $\eta=0.0078(2/255)$. The batch size is set to 48, and we use an SGD optimizer with momentum 0.9, gradient clipping 5.0, and weight decay 5e-4. The learning rate is set to 0.05 initially. On CIFAR-10 and CIFAR-100, the learning rate is decayed by 10 at epoch 100 and epoch 105. And on SVHN¹¹1To successfully train a VGG-16 model on SVHN, we first conduct a normal training process for 50 epochs, and finetune the normally-trained model with adversarial training., it is decayed by 10 at epoch 20, epoch 40 and epoch 45. In addition, following a recent study [23], we employ the label smoothing regularization [32] with weight 0.2. No cutout and dropout is applied. All models are trained with the same settings on a single 2080Ti GPU.

On Tiny-ImageNet, the image size is 64$\times$64 instead of 32$\times$32, and we change the stem from a 3x3 convolution to a 7x7 convolution with stride 2. Due to this modification, the overall FLOPs is still close to $C_{T}$. For all the architectures, we first conduct a normal training process for 110 epochs with batch size 256. The learning rate is set to 0.1 initially and decayed by 10 at epoch 30, 60 and 100. Using the normally-trained model as the pre-trained model, we adversarially train the model for 70 epochs with PGD⁵ attack ($\epsilon=0.0157(4/255)$, step size $\eta=0.0039(1/255)$). The learning rate is linearly warmed up from 0 to 0.05 in 10 epochs, and decayed by 10 at epoch 45 and 60. In both the normal and adversarial training processes, an SGD optimizer with momentum 0.9, gradient clipping 5.0, and weight decay 3e-4 is used.

The settings of the distance attacks on CIFAR-10: 1) DeepFool: 100 steps; 2) Basic Iterative Method (BIM): 20 binary search steps.

Fig. A3 shows that without BatchNorm calibration (No CalibBN, green points), the evaluation results are unmeaningful. And after BN calibration with several batches, the accuracies and rankings of 200 randomly sampled sub-architectures become meaningful. Note that we calculate the ranking correlation with the CalibBN-15 results (calibration using 15 batches), since we fail to tune the architectures to the same level of accuracies with only 3 epochs of training as in [10].

We train MSRobNets and all baseline architectures with identical settings that have been described before, and show the comparison in Tab. A1 and Tab. A2. We can see MSRobNets outperform the baseline architectures, with comparable FLOPs and fewer parameters.

An interesting observation is that the architectures demonstrate slightly different relative robustness on these two datasets. For example, MSRobNet-1000 outperforms MSRobNet-1560 in adversarial accuracies on SVHN/Tiny-ImageNet, and RobNet-free outperforms MSRobNet-1560 on SVHN. This may arise from that larger dataset difference degrades the architecture transferability. Thus, the MSRobNet-1560 architecture discovered on CIFAR-10 is suboptimal on SVHN/Tiny-ImageNet for the targeted FLOPs of 1560M.

We compare with the architectures discovered by baseline one-shot workflows at targeted FLOPs 1560M and 2000M in Tab. A3. All the one-shot searches are conducted on the supernet with init channel number 44. After the search, the discovered topology is augmented to the targeted capacity $C_{T}$. We can see that MSRobNets indeed outperform those architectures discovered by the one-shot methods consistently.

Compared with normal training and evaluation, adversarial training and adversarial robustness evaluation are much more time-consuming. Since NAS itself faces the computational challenge, how to evaluate the adversarial robustness efficiently in the search phase is an important problem. We wonder whether we can use a weaker and faster attack in the search phase while still managing to discover a robust architecture under the stronger attack. For example, while the final comparison is conducted using PGD⁷ (PGD attack with 7 steps), a recent study [5] tried to use the weaker FGSM attack during the whole search phase for efficiency.

As described in the main text, the search phase includes the supernet training phase and the parameter-sharing search phase. We conduct the following experiments to verify whether it is suitable to use FGSM as a substitute in the search phase: 1) FGSM-FGSM: Train another supernet with FGSM adversarial training, and use FGSM to evaluate the 200 randomly sampled topologies. 2) PGD⁷-FGSM: Reuse the supernet trained with PGD⁷ adversarial training, but use FGSM to evaluate the 200 randomly sampled topologies. The notation “ATTACK1-ATTACK2” means that “ATTACK1” is used in supernet adversarial training and “ATTACK2” is used in the parameter-sharing evaluation, and the step size of the FGSM attack is set to 8/255. Fig. A4 shows the rankings of these evaluations and the Spearman coefficient with the PGD⁷-PGD⁷ evaluations. The results show that using a much weaker attack FGSM for adversarial training causes the supernet evaluation to be uncorrelated. Thus, one can use FGSM for acceleration during the parameter-sharing search phase, while should keep using a stronger attack during the supernet training phase.

In the stage-wise search space, we also calculate the Kendall Tau, Spearman correlation, and P@K [22] of using PGD⁷-FGSM as a proxy reward for PGD⁷-PGD⁷. The P@K criterion reports the proportion of topologies with top-K rewards evaluated by PGD⁷-FGSM in the top-K topologies evaluated by PGD⁷-PGD⁷, and is a criterion that focus more on well-performing topologies. Results in Tab. A4 show that most of the high-score topologies evaluated by PGD⁷-FGSM also perform well under PGD⁷-PGD⁷ evaluation, which demonstrates that FGSM is indeed a suitable proxy attack for PGD during the search phase.

In our paper, we use a reward formulation that is a weighted sum of clean and adversarial accuracies.

where $\lambda$ is a hyperparameter making a trade-off between the clean and adversarial accuracies. During our search, $\lambda$ is set to 0.5. In other words, we use the average of clean and adversarial accuracies as the reward to guide the search.

Another type of robustness criterion is the distance-based criteria. The Foolbox toolbox [28] implements various kinds of distance attacks that attempt to find a minimum successful perturbation. Examples that have been misclassified without any adversarial perturbation are assigned zero attack distance. In this way, no averaging coefficient is needed to trade off the clean and adversarial accuracies, since the clean accuracy is handled implicitly.

We try to explore how these different robustness criteria correlate with each other. We evaluate the 200 randomly sampled architectures on the validation dataset in $\chi_{2}$ (init channel number 24) under five kinds of distance attacks. The five attacks are Basic Iterative Method (BIM) $\ell_{2}$ / $\ell_{\infty}$ [15], DeepFool $\ell_{2}$ / $\ell_{\infty}$ [20] and C&W $\ell_{2}$ [4], where $\ell_{2}$ or $\ell_{\infty}$ denotes the type of the $\ell_{p}$ distance measure in the input space. Due to the low efficiency of C&W attack, we only run it on 500 images ($1/20$) in the validation set. We use the median distance as one architecture’s performance criterion under the preformed attack, since it is insensitive to outliers (e.g., when the attack fails to find a proper adversarial perturbation within limited attack steps). For comparison, we also run PGD⁷ and FGSM attacks and calculate their rewards as in Eq. 6 with $\lambda=0.5$.

The Kendall Tau correlation coefficients between different criteria are shown in Fig. A5. We can see that

1. 1.

   Different robustness criteria are indeed correlated.

2. 2.

   Compared with the relatively high correlation between FGSM and PGD adversarial accuracies and rewards ($>0.8$), their correlations with distance criteria are relatively weaker. The clean accuracy is better correlated with FGSM/PGD rewards (0.718, 0.669) than the distance-based criteria (0.366-0.465). This is intuitive since the clean accuracy is added into FGSM/PGD rewards explicitly, while only handled implicitly (attack distance$=0$ or $>0$) in distance-based attacks. On the other hand, they are all higher than the correlations of clean accuracy and FGSM/PGD adversarial accuracies (0.324, 0.282), which verifies that an implicit or explicit trade-off between clean and adversarial accuracies must be considered in the robustness criterion.

3. 3.

   Intuitively, compared with $\ell_{\infty}$ attack distances, $\ell_{2}$ attack distances are better correlated with other $\ell_{2}$ attack distances, and vice versa. For example, $Corr(\mbox{DeepFool }\ell_{\infty},\mbox{BIM }\ell_{\infty})=0.724>Corr(\mbox{DeepFool }\ell_{\infty},\mbox{BIM }\ell_{2})=0.644$. There are indeed differences brought by input space measure, and this motivates us to use criteria with different input space measure to assess model robustness.

4. 4.

   FGSM/PGD attacks are conducted with $\ell_{\infty}$ input space measure, thus the correlation of their rewards with $\ell_{\infty}$ attack distances is higher than $\ell_{2}$ attack distances.

As mentioned in the main text, the FGSM proxy reward and fewer validation images accelerate the evaluation process by roughly 8$\times$ (from 25 min to 3 min for evaluating each topology on a single 2080Ti GPU).

After been trained only once, the supernets can be reused to discover architectures targeting different capacities (e.g., 1000M, 1560M, 2000M), or in search space with different macro layouts (e.g., cell-wise, stage-wise). Thus, for each targeted capacity, denoting the epoch-wise training and evaluation time of one architecture as $t^{s}$ and $t^{e}$, the search time can be estimated as $(N+T^{e})Kt^{e}$. By using the BatchNorm calibration technique, we obviate the need of separate architecture training phases during the search. If each architecture needs to be finetuned for $n$ epochs (e.g., $n=3$ in [10]), the search time becomes $(\pi+T^{e})(t^{e}+nt^{s})$. We can estimate the coefficients of $T^{e}$ in the multi-shot method and a NAS method with separate training phases as $Kt^{e}\approx 8t^{e}$ and $t^{e}+nt^{s}\approx 25t^{e}$, respectively.²²2In our experiments, we find that $t^{s}/t^{e}$ is approximately $2$ times of the train-valid data portion ratio $t^{s}/t^{e}\approx 2s/(1-s)=8$. This means that, compared with the NAS methods that have separate training phases, the relative efficiency of multi-shot NAS emerges as the number of explored architectures increases and the speedup would approaches 25/8 asymptotically.

In the predictor-based search flow, $N+T^{p}\times N^{p}=200+400=600$ topologies in total are assesed using the multi-shot evaluation strategy, and the whole search process can be finished in 1.3 GPU days.