Dense Optimizer : An Information Entropy-Guided
Structural Search Method for Dense-like Neural Network Design

DenseNet has garnered widespread attention and research interest in the field of computer vision [13]. Owing to its excellent performance and flexibility, a multitude of manual designs in recent years have leveraged to refine and augment the DenseNet architecture [14][15]. However, these designs still heavily rely on on human expertise and lack principles for structural design guidance [6]. When selecting hyperparameters such as channel growth rate and convolution kernels, extensive experimentation and tuning are often required[16]. Dense Optimizer provides an efficient and automatic structural search method for Dense-like networks and promote its performance at the same time.

Neural Architecture Search(NAS) have been greatly studied during the past few years to automatically design more effective architectures. Popular NAS algorithm options include genetic algorithms [17, 18, 19, 20], reinforcement learning (RL) [21, 22], differentiable search [23, 11] and many other types of optimization algorithms, e.g., Bayesian optimization and particle swarm optimization [24, 25, 26]. As a classic differentiable architecture search method, DARTS is the most widely attention-grabbing algorithms. It models the architecture design as a bi-level optimization problem, which requires training vast numbers of candidate networks to inform the search process, and often leads to high computational cost.

Recent NAS work Mellor et al. [27] start to explore indicators that can predict a network’s performance without training. Moreover, Xuan et al. [28] endeavor to decouple network weights from network architecture, focusing on the discovery of improved design principles through the exploration of structural parameters alone. These methods have represented an effective exploration in neural network architecture search that decouples structural parameters and weights, but still lack good design guidelines. Dense Optimizer conducts further in-depth research. Dense Optimizer circumventing the time-consuming issue of bi-level optimization, and uses information entropy as a design criterion to obtain a better structure by solving an optimization problems.

Information theory is a powerful tool for studying complex systems such as deep neural networks. Recently, mathematical architecture design(MAD) is proposed [28]. Unlike the existing hyperparameter optimization methods [29, 30, 31], MAD does not require any model training during optimization, allowing for the acquisition of optimized network structures within minutes. It maximizes the network entropy with three empirical guidelines and demonstrate an advancement in designing network structures using mathematical programming. However, MAD unable to characterize the information flow structure of the concatenation operation and accurately estimate the information entropy of the dense network structure. And its three empirical guidelines do not have strong theoretical foundation. Dense Optimizer not only specifically addresses these issues, but also constraints the distribution of entropy at different scales with power-law and proposes a new optimization model for dense-like network architectures.

The principle of maximum entropy is one of the most widely applied principles in information theory [32]. Some previous works have also attempted to establish a connection between entropy and the structure of neural networks. Here, we provide a re-derivation, and give the entropy upper bound of DenseBlock. Suppose that in an L-layer MLP f(·), the i-th layer has $\mathrm{w}_{i}$ input channels and $\mathrm{w}_{\mathrm{i}+1}$ output channels. The output $\mathrm{x}_{\mathrm{i}+1}$ and the input $\mathrm{x}_{i}$ are connected by $\mathrm{x}_{\mathrm{i}+1}=\mathrm{M}_{\mathrm{i}}\mathrm{x}_{i}$ , where $M_{i}\in R^{w_{i+1}\times w_{i}}$ is trainable weights. Then the structural parameters define how the input $\mathrm{x}_{i}$ propagates inside the network, which is capable of being depicted.

For a DenseBlock with L layers, we consider the information reuse caused by the network’s dense connections and the impact of information distribution brought about by the concatenation operation. Then we obtain the information entropy of the basic block of the dense network, defined in Proposition 1:

Proposition 1.The normalized Gaussian entropy upper bound of the DenseBlock f(·) is

where the $\mathrm{w}_{L}$ is the width of $L$-th layer, and $\mathrm{w}_{0}$ is the initial width of the DenseBlock. The whole derivation is given in Appendix A. The entropy measures the expressiveness of a DenseBlock. Following the Principle of Maximum Entropy [33, 34], we propose to maximize the entropy of DenseBlock under given computational budgets. When calculating the precise information entropy of a dense block, the number of the $i$-th dense layer, the input channels is $\mathrm{c}_{i}$, the number of output channels is $\mathrm{c}_{\mathrm{i}+1}$, and the kernel size is $\mathrm{k}_{i}$. Consequently, the ”width” of a dense block layer is projected as $\mathrm{c}_{i}\mathrm{k}_{i}^{2}$ in (1). Therefore, for a feature map with a resolution of $\mathrm{r}_{i}\times\mathrm{r}_{i}$, the entropy of a DenseBlock with L layers is defined by

Inspired by [35], taking logarithms can better formulate the ground-truth entropy for natural images.

Inspired by previous research, an infinitely deep network will become hard to train unless it meets particular structural requirements. Therefore, in Dense Optimizer, we propose to control the depth of the dense-like network to facilitate gradient flow throughout the entire architecture effectively. Typically, the depth and width of a network are relative; thus, the effectiveness of a network with $\mathrm{L}$ layers, where each layer possesses the same width $\mathrm{W}$, can be defined as follows:

Normally, the width $\mathrm{w}_{i}$ of each layer can be different. So the average width of an $\mathrm{L}$ layer network f(·) is defined by

But in DenseBlock, each layer connects to all the previous layers. This results in a relatively steady increase in the number of parameters with each layer. The growth rate $\mathrm{K}$ is always the same, and much smaller than the number of input channels. So the average of the width can be defined as:

So a DenseBlock has L-layers with $\mathrm{w}_{0}$ input width and growth rate $\mathrm{K}$, the effectiveness is defined by

In information theory, entropy reflects the amount of information or uncertainty within a system. Higher information entropy implies the system is more dispersed and diverse [36]. For multi-stage neural networks, they are akin to independent information systems. To eliminate the discrete distribution of information entropy and uncertainty across these systems, it is essential to propose a multi-scale entropy distribution under mathematical constraints. According to highly optimized tolerance(HOT)[37], when a complex system is in a HOT state, the system will satisfy power-laws, that is, a global optimization process can lead to power-law distributions: inputs with characteristic scales, after undergoing a global system’s “output” optimization process, can produce outputs with power-law characteristics[38].

Based on extensive experimental statistics, We find that the information entropy distribution of dense network follows a power-law distribution, see figure 1.

To reinforce this constraint, We propose a power-law in entropy distribution, and we use a two-parameter fitting function with parameters ’a’ and ’b’ to optimize the distribution. The object is to maximize the value of ’a’ and minimize the value of ’b’ under the same fitting parameter settings.

Here, we provide specific definitions. Following (2), The cumulative entropy distribution sequence at the current stage is:

The fitting expression of this sequence under the power law function is as follows:

where a, b are power index parameters, and $\mathrm{M}_{i}$ represent the i th stages. So the optimization target S is:

Subsequently, we establish optimization constraints to achieve the objective of maximizing a while minimizing b.

We gather everything together and present the final optimization model for Dense Optimizer. Suppose that we aim to design an L-layer dense-like model $F$ with $\mathrm{M}$ stages. The entropy of the $\mathrm{i}$-th stage is denoted as $\mathrm{H}_{i}$ defined in (2). We propose to optimize {$\mathrm{c}_{i}$,$\mathrm{k}_{i}$,$\mathrm{L}_{i}$} via the following optimization problem:

where $\alpha_{i}$ is the weights of entropies at different scales, $\beta$ is a tuning coefficient. $\rho$ controls the effectiveness of the network whose value is usually tuned in range $[10,20]$.

Integrating the complexity of above optimization problem and the particularity of the candidate solution space of network structure, we propose a branch-and-bound algorithm to integrate power-law in algorithm 1, to achieve efficient optimization search. By combining the properties of power-law distribution, we decompose the optimization problem into sub-optimization problems, gradually narrowing the search space and finding the optimal solution through searching.

Specifically, we first score the initial densely connected architecture through information entropy representation. Then, during the search process, we relax the initial network structure while employing regional reduction techniques during the relaxation process. Based on the network information entropy at a certain stage, we calculate the entropy space that conforms more to the power law distribution, and prune the search space. In the process of iterative optimization, we always retain the optimal solution.

In Dense Optimizer, the number of search populations N is 256, and the number of iterations is 500,000. The classic densenet121 is used as the initial backbone network. Search space parameters: the number of input and output channels for each block, convolution kernel size: [3,5,7], budget layers=130, maximum growth rate: [12,24,40]. Meanwhile the optimization problem is solved on CPU device.

Following previous works , SGD optimizer with momentum 0.9 is adopted to train the dense models. The weight decay is 5e-4 for CIFAR dataset . The initial learning rate is 0.1 with batch size of 32. We use cosine learning rate decay with 5 epochs of warm-up. The number of training epochs is 100 for CIFAR-100. All experiments use the following data augmentations : mix-up, label-smoothing , random erasing, random crop/resize/flip/lighting, and Auto-Augment .

As shown in table I and table II, after optimization with Dense Optimizer, the performance of the dense network on the CIFAR-100 dataset significantly surpassed the original network. Given diverse latency budgets, our method outperforms the compared NAS methods in terms of the accuracy of the generated/searched architectures. It is solved in CPU device with small search cost. The optimized model, with a size of 32M, achieved a TOP-1 error rate of 16.96%, while the larger network of 171M size had a TOP-1 error rate of 15.7%.

Moreover, we compare the searched architectures on CIFAR-10 and SVHN datasets, showing in table III and table IV.

The results demonstrate that Dense Optimizer achieves effective results on different datasets. Within the same model size, the accuracy of the optimized models is significantly higher than the original densenet, and they can easily outperform ResNet family.

In this section, we conducted ablation experiments on the CIFAR-100 dataset with power-law distribution constraints and performed statistical analysis on the information entropy of multi-stage power-law distributions. We utilized traditional dense-BC convolution blocks and networks generated with different information entropy distributions. The network structures found under different max channel growth rates $\mathrm{K}$ [40, 24, 12] were trained, and the fitting values of information entropy distribution at each stage were statistically analyzed.

As shown in table V, controlling the information entropy distribution at each stage can improve the performance of image classification tasks. When the distribution satisfies a power-law distribution, the network can achieve the best model performance in classification tasks. As can be seen in figure 2, the model performance is positively correlated with the power-law distribution hyperparameter a (Pearson correlation coefficient of $0.86$) and negatively correlated with the hyperparameter b (Pearson correlation coefficient of $-0.94$). Therefore, a better performance can be achieved by optimizing the structure of the model through power-law constraints.

We also statistically averaged the information entropy of all optimized dense network stages and fitted them with different functions. And we compared fitting errors, including SSE (Sum of Squared Errors), R-square (Coefficient of Determination), Adjusted R-square, and RMSE (Root Mean Squared Error). Figure 3 indicates that, compared to first-order and second-order polynomial functions and exponential functions, the power-law function also has the smallest fitting error and the highest fitting score.

Subsequently, we conducted ablation experiments under power-law constraints. In table V, we can observe that the model performance is significantly improved after optimization with the dense Optimizer and the addition of power-law constraint terms. This showcases how the model performance evolves with the strengthening of power-law constraints. The DenseNet, not optimized by power-law, achieved accuracy gains of +0.20%, +0.57%, and +2.45% on SVHN, CIFAR-10, and CIFAR-100 respectively. Meanwhile, the DenseNet optimized under power-law achieved accuracy gains of +0.23%, +1.66%, and +2.94% on SVHN, CIFAR-10, and CIFAR-100 respectively.

Moreover, $\beta$ parameter is crucial in tuning the model. It controls the relationship between the power-law distribution and the magnitude of information entropy. Our results indicate that when $\beta$ is set to 0.1, there is an optimal balance between information entropy and the power-law constraint, leading to the best model performance (See table VI).

Overall, the analysis proves that there is a strong correlation between hyperparameters a, b and model performance. The ablation result further validates the significance of power-law constraints in model design. And we analyze the differences between different distribution functions to show the fitness of power. Ultimately, we propose an optimal hyperparameter $\beta$ to achieve the best balance of entropy distribution.