RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies

Abstract

Neural architecture search (NAS) has made tremendous progress in the automatic design of effective neural network structures but suffers from a heavy computational burden. One-shot NAS significantly alleviates the burden through weight sharing and improves computational efficiency. Zero-shot NAS further reduces the cost by predicting the performance of the network from its initial state, which conducts no training. Both methods aim to distinguish between ”good” and ”bad” architectures, i.e., ranking consistency of predicted and true performance. In this paper, we propose Ranking Distillation one-shot NAS (RD-NAS) to enhance ranking consistency, which utilizes zero-cost proxies as the cheap teacher and adopts the margin ranking loss to distill the ranking knowledge. Specifically, we propose a margin subnet sampler to distill the ranking knowledge from zero-shot NAS to one-shot NAS by introducing Group distance as margin. Our evaluation of the NAS-Bench-201 and ResNet-based search space demonstrates that RD-NAS achieve 10.7% and 9.65% improvements in ranking ability, respectively. Our codes are available at https://github.com/pprp/CVPR2022-NAS-competition-Track1-3th-solution

Index Terms— neural architecture search, one-shot NAS, zero-shot NAS, rank consistency

1 Introduction

Refer to caption — Fig. 1: The ranking consistency of one-shot NAS (SPOS [1]) and zero-shot NAS(FLOPs [2], Params [2], Snip [2], Synflow [2], ZenNAS [3], NASWOT [4], Grasp [2]). Our proposed RD-NAS achieves higher Kendall’s Tau than both one-shot and zero-shot NAS and converges faster than the baseline [1]

Neural Architecture Search (NAS) has sparked increased interest due to its remarkable progress in a variety of computer vision tasks [5, 3, 6, 7, 8]. It aims to reduce the cost of human efforts in manually designing network architectures and discover promising models automatically. Early NAS works [9, 10, 11] take thousands of GPU hours in search cost, and ENAS [12] first attempt at weight-sharing techniques to accelerate the searching process. The key to weight-sharing-based methods is an over-parameterized network, a.k.a. supernet, that encompasses all candidate architectures in the search space. The weight-sharing-based NAS approaches [1, 13] are called one-shot NAS since it only requires the cost of training one supernet. Despite the high efficiency of one-shot NAS, it is theoretically based on the ranking consistency assumption, a.k.a. the estimated performance of candidate architectures in supernet should be highly correlated with the true performance of the corresponding architecture when trained from scratch. However, due to the nature of the weight-sharing approach, the subnet architectures interfere with each other, and the estimated accuracy from the supernet is inevitably averaged.

A recent trend of NAS focuses on zero-cost proxies [4, 2], which aims to estimate the relative performance of candidate architecture from a few mini-batch of data without training the model. The approach of inferring the trained accuracy directly from the initial state of the model is called zero-shot NAS since it does not involve any training. However, there is a clear preference for zero-shot NAS that affects its ability to find good architectures [14].

To alleviate the ranking disorder caused by weight-sharing, some predictor-based methods [15, 16] use the ranking loss to enhance the ranking ability of the model, but the challenge is that acquiring training samples (pairs of (model, accuracy)) is computationally expensive. To improve the ranking consistency economically, we resort to zero-shot NAS as an inexpensive teacher and measure the ranking relationship between a pair of subnets without employing a predictor. In recent years, the experimental investigations [2] enhanced the existing NAS algorithms using zero-cost proxies to initialize the search algorithm at the beginning of the search process, which accelerated the convergence speed of the supernet. Unlike its limited use, we incorporate zero-cost proxies into the whole training procedure with ranking knowledge distillation [17, 18, 19, 20, 21, 22, 23].

We observe the complementarity between one-shot NAS and zero-shot NAS in Fig.1. Zero-cost proxies show good initial ranking consistency, but the upper-performance limit is not as high as one-shot NAS, while one-shot NAS has lower ranking performance in the early stage. This inspires us to bridge one-shot NAS and zero-shot NAS by utilizing zero-cost proxies as ranking teachers. In this paper, we propose a novel Ranking Distillation one-shot NAS (RD-NAS) to transfer the ranking knowledge from Zero-cost proxies to one-shot supernet.

Our contributions are summarized as follows: (1) We propose a Ranking Distillation framework via margin ranking loss to distill knowledge from zero-cost proxies and improve the ranking ability of one-shot NAS. (2) We propose a margin subnet sampler to reduce the uncertainty caused by zero-cost proxies by introducing group distance. (3) We demonstrate the effectiveness of RD-NAS on two types of benchmark, a.k.a. ResNet-like search space, and NAS-Bench-201, which achieve 10.7% and 9.65% improvement on Spearman and Kendall’s Tau, respectively.

2 Ranking Distillation One-shot NAS

Ranking Distillation one-shot NAS is a framework for distilling ranking knowledge from zero-shot NAS to improve the ranking ability of one-shot NAS. In Sec.2.1, we introduce how to measure the effectiveness of zero-cost proxies in the context of ranking and introduce a new criteria. In Sec.2.2, we propose the margin subnet sampler with margin ranking loss to perform the ranking distillation.

2.1 How to measure Zero-cost Proxies

Given a candidate architecture $\alpha_{i}\in\Omega,(i=1,2,...,N)$ in the search space $\Omega$ , zero-cost proxies can quickly estimate the relative score $\pi^{z}_{i}\in\mathbb{R}$ of the $i$ -th architecture, which can obtain the relative ranking relationship with other counterparts. However, there is a certain gap between the score $\pi^{z}_{i}$ obtained by zero-cost proxies and truth scores $\pi^{*}_{i}\in\mathbb{R}$ obtained by stand-alone training. We introduce $\sigma_{i}$ to estimate the variance of the zero-cost score of the $i$ -th architecture, and the equation $\pi^{*}_{i}=\pi^{z}_{i}+\sigma_{i}$ holds. We expect to introduce truth ranking correlation between architectures into the supernet optimization process. To measure the estimation ability of zero-cost proxies, we incorporate the objective function:

	$\displaystyle max_{i,j}P(\pi^{}_{i}>\pi^{}_{j})$	$\displaystyle=max_{i,j}P(\pi^{z}_{i}-\pi^{z}_{j}+\sigma_{i}-\sigma_{j}>0)$
		$\displaystyle\approx max_{i,j}P(\pi^{z}_{i}-\pi^{z}_{j}>0)$		(1)

The variance of a certain zero-cost proxy can be considered as a constant scalar, so that $\mathbb{E}[\sigma_{i}-\sigma_{j}]\approx 0$ , where we use zero-cost score $\pi^{z}$ to estimate the ground truth $\pi^{*}$ . Here we simplify Kendall’s tau and introduce Concordant Pair Ratio (CPR) $\delta$ to evaluate the pair-wise ranking consistency. We define the subnet pairs that satisfying $(\pi^{*}_{i}-\pi^{*}_{j})(\pi^{z}_{i}-\pi^{z}_{j})>0$ as the concordant pair. The Concordant Pair Ratio coefficient $\delta$ is defined as:

\delta=\frac{\sum_{i,j}1((\pi^{*}_{i}-\pi^{*}_{j})(\pi^{z}_{i}-\pi^{z}_{j})>0)}{\sum_{i,j}1}

(2)

which is in the range $0\leq\delta\leq 1$ . The closer $\delta$ is to 1, the better estimation of the zero-cost proxy for ground truth and vice versa.

2.2 Ranking Distillation

To transfer the ranking knowledge obtained from zero-cost proxies to one-shot supernet, we propose a ranking distillation to further improve the ranking consistency of the one-shot NAS. In Sec.2.2.1, we introduce the margin ranking loss to distill the ranking knowledge from zero-cost proxies, but the margin is hard to set in one-shot training. Therefore, we propose a margin subnet sampler to sample subnet pairs from the subnet space in Sec.2.2.2.

2.2.1 Margin Ranking Loss

Margin ranking loss is adopted to constrain the optimization of the supernet from the perspective of ranking distillation. Unlike the standard pairwise ranking loss [24], we do not have the true rank, only the estimated rank provided by zero-cost proxies, which suffer from uncertainty. Margin ranking loss can handle difficult subnet pairs by introducing a fixed penalty margin $m$ . The margin ranking loss of random-sampled subnet pair ( $\alpha_{i}$ , $\alpha_{j}$ ) is as follows:

$\mathcal{L}^{rd}_{(\alpha_{i},\alpha_{j})}\left(\theta^{s}\right)=\max\left[0,m-\left(\mathcal{L}^{ce}\left(x,\theta_{\alpha_{i}}^{s}\right)-\mathcal{L}^{ce}\left(x,\theta_{\alpha_{j}}^{s}\right)\right)\right]$

(3)

where $\theta^{s}$ denotes the supernet in one-shot NAS, and $\mathcal{L}^{ce}(\cdot)$ denotes the cross-entropy loss. However, such a fixed learning objective is not realistic as the training goes with inconsistent variation, which might limit the discriminative training of supernet. Instead of using a fixed margin $m$ , we proposed a margin subnet sampler to control the margin in the subnet sampling process.

Table 1: The comparison of Kendall’s Tau w.r.t different methods on ResNet-like search space. ”MSS” denotes the margin subnet sampler.

Type	Method	Spearman $r$ (%)
Zero-cost Proxies	Params [2]	67.23
	FLOPs [2]	78.43
	ZenNAS [3]	81.27
One-shot NAS	SPOS [1]	72.49
	Sandwich [25]	74.86
	RD-NAS w/o MSS	81.14
	RD-NAS w/ MSS	83.20

2.2.2 Margin Subnet Sampler

It is difficult to tune using a fixed margin, and the margin changes as training progresses. The effectiveness of margin ranking loss is greatly affected by the subnet pair chosen. To tackle the problems, we propose a margin subnet sampler, which affects the margin by controlling the sampled subnets. Intuitively, we expect the sampled subnet pairs to cover the entire search space, which can improve the generalization ability. We first randomly select $n$ pairs of subnet $\{(\alpha_{i},\alpha_{j})\}_{n}$ from the search space, then we calculate the distance of the subnet pair with distance function $f(\cdot)$ mapping from subnet space $\mathbb{R}^{+}$ to ranking space $\mathbb{R}$ and filter out the $c$ pairs of subnet $\{(\alpha_{i},\alpha_{j})\}_{c}=\{(\alpha_{i},\alpha_{j})|f(\alpha_{i},\alpha_{j})>\mu\}$ , where distances less than the threshold $\mu$ is selected.

Concerning the subnet distance measure $f(\cdot)$ , by encoding the subnet as a sequence composed of candidate choices, the Hamming distance $f_{h}(\cdot)$ is computed as the number of elements that are different in the two subnet sequences. However, Hamming distance treats all candidate operations as equal and ignores the variability between them. To make the distance measure more discriminative, we propose the Margin Subnet Sampler (MSS) with Group distance $f_{g}(\cdot)$ , which divides the candidate operations into groups with and without parameters. The intra-group distance is set small, and the inter-group distance is set large.

f_{g}^{k}(\alpha_{i}^{k},\alpha_{j}^{k})=\left\{\begin{array}[]{rcl}2f_{h}(\alpha_{i}^{k},\alpha_{j}^{k}),&&\beta(\alpha_{i}^{k},\alpha_{j}^{k})=0\\ 0.5f_{h}(\alpha_{i}^{k},\alpha_{j}^{k}),&&\beta(\alpha_{i}^{k},\alpha_{j}^{k})=1\end{array}\right.

(4)

where $\beta(\cdot)$ is an indicator function of whether $(\alpha_{i}^{k},\alpha_{j}^{k})$ are intra-group and $f_{g}^{k}(\cdot)$ denotes the group distance of the $k$ -th operation of the subnet encoding.

Table 2: Ranking correlation on NAS-Bench-201 CIFAR-10.

\tau

, r,

\rho

, and

\delta

refers to Kendall’s Tau, Pearson, Spearman and CPR, respectively. ”Random” refers to the rank correlation on random initialized model.

	$\tau$ (%)	$r$ (%)	$\rho$ (%)	$\delta$ (%)
FLOPs [2]	52.92	47.12	72.44	78.49
Params [2]	56.64	49.14	74.70	77.49
Snip [2]	51.16	28.20	69.12	80.49
Synflow [2]	58.83	44.90	77.64	84.99
ZenNAS [3]	48.80	55.72	67.83	44.99
NASWOT [4]	55.13	86.71	74.93	70.99
Random	-26.32	-21.43	-22.48	1.99
SPOS [1]	77.43	89.55	92.73	100.0
FairNAS [26]	70.62	82.81	88.74	100.0
RD-NAS(Ours)	87.08	95.00	97.34	100.0

Table 3: Searching results on NAS-Bench-201 dataset [27]. Our method obtains 1.12%, 3.76%, and 4.18% absolute accuracy improvement compared with SPOS on the test sets of CIFAR-10, CIFAR-100, and ImageNet-16-120, respectively. Note that results for other weight-sharing methods are reported by NAS-Bench-201 benchmark [27], which only uses the labels of the dataset as the true performance of the searched architecture for a fair comparison.

Method	CIFAR-10 Acc. (%)		CIFAR-100 Acc. (%)		ImageNet-16-120 Acc. (%)
Method	Validation	Test	Validation	Test	Validation	Test
GDAS [28]	90.00 $\pm$ 0.21	93.51 $\pm$ 0.13	71.14 $\pm$ 0.27	70.61 $\pm$ 0.26	41.70 $\pm$ 1.26	41.84 $\pm$ 0.90
DSNAS [29]	89.66 $\pm$ 0.29	93.08 $\pm$ 0.13	30.87 $\pm$ 16.40	31.01 $\pm$ 16.38	40.61 $\pm$ 0.09	41.07 $\pm$ 0.09
PC-Darts [30]	89.96 $\pm$ 0.15	93.41 $\pm$ 0.30	67.12 $\pm$ 0.39	67.48 $\pm$ 0.89	40.83 $\pm$ 0.08	41.31 $\pm$ 0.22
NASWOT [4]	89.14 $\pm$ 1.14	92.44 $\pm$ 1.13	68.50 $\pm$ 2.03	68.62 $\pm$ 2.04	41.09 $\pm$ 3.97	41.31 $\pm$ 4.11
EPENAS [31]	89.90 $\pm$ 0.21	92.63 $\pm$ 0.32	69.78 $\pm$ 2.44	70.10 $\pm$ 1.71	41.73 $\pm$ 3.60	41.92 $\pm$ 4.25
Random	83.20 $\pm$ 13.28	86.61 $\pm$ 13.46	60.70 $\pm$ 12.55	60.83 $\pm$ 12.58	33.34 $\pm$ 9.39	33.13 $\pm$ 9.66
SPOS [1]	88.40 $\pm$ 1.07	92.24 $\pm$ 1.16	67.84 $\pm$ 2.00	68.07 $\pm$ 2.25	39.28 $\pm$ 3.00	40.28 $\pm$ 3.00
RD-NAS(Ours)	90.44 $\pm$ 0.27	93.36 $\pm$ 0.04	70.96 $\pm$ 2.12	71.83 $\pm$ 1.33	43.81 $\pm$ 0.09	44.46 $\pm$ 1.58
Optimal	91.61	94.37	73.49	73.51	46.77	47.31

3 Experiments

The experiments in Sec.3.1 and Sec.3.2 verify the effectiveness of the RD-NAS on the ResNet-like search space and NAS-Bench-201. Then, we conduct the ablation study for RD-NAS.

3.1 Searching on ResNet-like Search Space

Setup. Presented in CVPR NAS Workshop, the ResNet-like search space is based on ResNet48 on the ImageNet [32], whose depth and expansion ratio are searchable. Specifically, the search space consists of four stages with different depths {5,5,8,5}, while the candidate expansion ratio is ranging from 0.7 to 1.0 at 0.05 intervals, and the channel number should be divided by 8. The search space size is $5.06\times 10^{19}$ in total. Our proposed RD-NAS won the third place at CVPR 2022 NAS Track1. We conduct a comprehensive comparison of the zero-shot [3] and one-shot methods [1, 26, 25], and the Spearman’s ranking correlation is adopted to measure the ranking ability. We first pre-train the supernet for 90 epochs in ImageNet and then train the supernet with a learning rate of 0.001 with a batch size of 256 for 70 epochs. The SGD optimizer is adopted with a momentum of 0.9.

Results. The results of Spearman’s ranking correlation are shown in Tab.1. RD-NAS could achieve significantly higher ranking correlations than both the zero-shot and one-shot NAS, especially using the margin subnet sampler. The Kendall’s Tau of RD-NAS surpasses the baseline method [1] by 10.71% and surpasses the zero-cost proxy - ”FLOPs”, demonstrating that using zero-cost proxies as teachers for distillation can effectively mitigate the ranking disorder problem.

3.2 Searching on NAS-Bench-201

Setup. NAS-Bench-201 [27] provides the performance of 15,625 subnets on CIFAR-10, CIFAR-100, and ImageNet-16-120. In our experiments, the training settings are consistent with NAS-Bench-201. Following SPOS [1], we formulate the search space of NAS-Bench-201 [27] as a single-path supernet. The distance threshold $\mu$ is set to 6.7.

Evaluation Criteria. The correlation criteria used in NAS-Bench-201 are Pearson $r$ , Kendall’ Tau $\tau$ [33], Spearman $\rho$ , and CPR $\delta$ . Pearson measures the linear relationship between the variables, while Kendall’s Tau, Spearman, and CPR measure the monotonic relationship. The ranking correlations are computed using 200 randomly selected subnets.

Results. Our method obtains 2.04%, 3.12%, and 4.53% absolute accuracy improvement on the validation sets of CIFAR-10, CIFAR-100, and ImageNet-16-120 in Tab.3, compared with SPOS, which further certifies that RD-NAS enables to boost the ranking ability of one-shot NAS. Even compared with the recent NAS methods (e.g., PC-Darts [30] and GDAS [28]), our method achieve comparable performance on the NAS-Bench-201. Especially for ImageNet-16-120, our RD-NAS achieves state-of-the-art performance that surpasses all other methods. The results on Tab.2 reveal that our RD-NAS achieve better ranking correlation than one-shot and zero-shot NAS.

3.3 Ablation Study

The ablation study of the distance measurement and training strategy on NAS-Bench-201 CIFAR-10 is shown in Tab.4. The one-shot training with uniform sampling strategy[1] is employed as the ”Random” baseline. The results of Ranking Distillation exhibit the effectiveness of margin ranking loss. Moreover, using hamming or adaptive distance to sample subnets can further improve the rank consistency. The combination of margin subnet sampler and ranking distillation improves the Kendall’s Tau from 77% to 87%.

Table 4: Ablation study of RD-NAS on NAS-Bench-201 CIFAR-10. ”Random” denotes the baseline SPOS[1], ”Margin” and ”Hamming” denotes margin subnet sampler with Group and Hamming distance, respectively. ”RD” denotes Ranking Distillation.

Random	Margin	Hamming	RD	Kendall’ Tau(%)
$\checkmark$	-	-	-	77.43
-	-	-	$\checkmark$	79.37
-	-	$\checkmark$	$\checkmark$	84.18
-	$\checkmark$	-	$\checkmark$	87.08

4 Conclusion

In this paper, we present a novel ranking knowledge distillation for one-shot NAS, which we call RD-NAS, to mitigate the rank disorder problem. Specifically, our RD-NAS employs zero-cost proxies as teachers and transfers the ranking knowledge to one-shot supernet by margin ranking loss. To reduce the uncertainty of zero-cost proxies, we introduce the margin subnet sampler to increase the reliability of ranking knowledge. We conduct detailed experiments on ResNet-like search space and NAS-Bench-201 to demonstrate the effectiveness of RD-NAS. It is hoped that our work will inspire further research to exploit the strengths of zero-shot NAS.

References

[1] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun, “Single path one-shot neural architecture search with uniform sampling,” in ECCV, 2020.
[2] Mohamed Saleh Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas D. Lane, “Zero-cost proxies for lightweight nas,” ArXiv, vol. abs/2101.08134, 2021.
[3] Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin, “Zen-nas: A zero-shot nas for high-performance image recognition,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 337–346, 2021.
[4] Joseph Charles Mellor, Jack Turner, Amos J. Storkey, and Elliot J. Crowley, “Neural architecture search without training,” ArXiv, vol. abs/2006.04647, 2021.
[5] Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Ken Chen, Wanli Ouyang, and Dong Xu, “Block proposal neural architecture search,” IEEE Transactions on Image Processing, vol. 30, pp. 15–25, 2020.
[6] Yiming Hu, Xingang Wang, Lujun Li, and Qingyi Gu, “Improving one-shot nas with shrinking-and-expanding supernet,” Pattern Recognition, 2021.
[7] Peijie Dong, Xin Niu, Lujun Li, Linzhen Xie, Wenbin Zou, Tian Ye, Zimian Wei, and Hengyue Pan, “Prior-guided one-shot neural architecture search,” arXiv preprint arXiv:2206.13329, 2022.
[8] Kunlong Chen, Liu Yang, Yitian Chen, Kunjin Chen, Yidan Xu, and Lujun Li, “Gp-nas-ensemble: a model for the nas performance prediction,” in CVPRW, 2022.
[9] Barret Zoph and Quoc Le, “Neural architecture search with reinforcement learning,” in ICLR 2017 : International Conference on Learning Representations 2017, 2017.
[10] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le, “Regularized evolution for image classifier architecture search,” in Proceedings of the aaai conference on artificial intelligence, 2019, vol. 33, pp. 4780–4789.
[11] Zimian Wei, Hengyue Pan, Lujun Li Li, Menglong Lu, Xin Niu, Peijie Dong, and Dongsheng Li, “Convformer: Closing the gap between cnn and vision transformers,” arXiv preprint arXiv:2209.07738, 2022.
[12] Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean, “Efficient neural architecture search via parameter sharing,” arXiv preprint arXiv:1802.03268, 2018.
[13] Hanxiao Liu, Karen Simonyan, and Yiming Yang, “Darts: Differentiable architecture search,” arXiv preprint arXiv:1806.09055, 2018.
[14] Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Yu Wang, “Evaluating efficient performance estimators of neural architectures,” in NeurIPS, 2021.
[15] Chi Hu, Chenglong Wang, Xiangnan Ma, Xia Meng, Yinqiao Li, Tong Xiao, Jingbo Zhu, and Changliang Li, “Ranknas: Efficient neural architecture search by pairwise ranking,” in EMNLP, 2021.
[16] Ruochen Wang, Xiangning Chen, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh, “Rank-nosh: Efficient predictor-based architecture search via non-uniform successive halving,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10357–10366, 2021.
[17] Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao, “Norm: Knowledge distillation via n-to-one representation matching,” in ICLR, 2022.
[18] Jie Qin, Jie Wu, Xuefeng Xiao, Lujun Li, and Xingang Wang, “Activation modulation and recalibration scheme for weakly supervised semantic segmentation,” in AAAI, 2022.
[19] Lujun Li, “Self-regulated feature learning via teacher-free feature distillation,” in ECCV, 2022.
[20] Lujun Li, Liang Shiuan-Ni, Ya Yang, and Zhe Jin, “Boosting online feature transfer via separable feature fusion.,” in IJCNN, 2022.
[21] Lujun Li and Zhe Jin, “Shadow knowledge distillation: Bridging offline and online knowledge transfer,” in NeuIPS, 2022.
[22] Lujun Li, Yikai Wang, Anbang Yao, Yi Qian, Xiao Zhou, and Ke He, “Explicit connection distillation,” in ICLR, 2020.
[23] Lujun Li, Liang Shiuan-Ni, Ya Yang, and Zhe Jin, “Teacher-free distillation via regularizing intermediate representation,” in IJCNN, 2022.
[24] Kaicheng Yu, René Ranftl, and Mathieu Salzmann, “Landmark regularization: Ranking guided super-net training in neural architecture search,” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13718–13727, 2021.
[25] Jiahui Yu and Thomas Huang, “Autoslim: Towards one-shot architecture search for channel numbers,” arXiv: Computer Vision and Pattern Recognition, 2019.
[26] Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li, “Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search,” arXiv preprint arXiv:1907.01845, 2019.
[27] Xuanyi Dong and Yi Yang, “Nas-bench-201: Extending the scope of reproducible neural architecture search,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
[28] Xuanyi Dong and Yezhou Yang, “Searching for a robust neural architecture in four gpu hours,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1761–1770, 2019.
[29] Shoukang Hu, S. Xie, Hehui Zheng, C. Liu, Jianping Shi, Xunying Liu, and D. Lin, “Dsnas: Direct neural architecture search without parameter retraining,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12081–12089, 2020.
[30] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong, “PC-DARTS: partial channel connections for memory-efficient architecture search,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2019.
[31] Vasco Lopes, Saeid Alirezazadeh, and Luís A. Alexandre, “Epe-nas: Efficient performance estimation without training for neural architecture search,” in ICANN, 2021.
[32] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al., “ImageNet large scale visual recognition challenge,” IJCV, vol. 115, no. 3, pp. 211–252, 2015.
[33] Pranab Kumar Sen, “Estimates of the regression coefficient based on kendall’s tau,” Journal of the American Statistical Association, vol. 63, pp. 1379–1389, 1968.