Redundancy of Hidden Layers in Deep Learning: An Information Perspective

Deep neural networks (DNNs) have achieved significant success in many practical applications due to their strong expression capacity and powerful learning ability. However, the deep structure of DNNs may lead to complicated nonlinear mappings from input to output, giving rise to the problem of overfitting. Therefore, many studies have been devoted to developing approaches to improve the generalization capacity of DNNs in order to address the overfitting problem. One research direction is to constrain the model complexity; these methods include dropout Srivastava et al. (2014), weight decay Krogh & Hertz (1991), and CIF Zhao et al. (2018). Despite the high effectiveness of these methods, they may not fully leverage the expression capability of the model because they generally reduce the effective number of model parameters.

To improve the generalization capacity of DNNs while simultaneously maintaining their expressivity, another research direction to pursue is to explore the diversity among the hidden units of a single specified layer of DNNs and to improve it, encouraging hidden units to be as uncorrelated or independent from each other as possible. For instance, Cogswell et al. minimized the cross-covariance of hidden activations to obtain diverse representation of hidden units to reduce overfitting Cogswell et al. (2016). Gu et al. extended this method, treating nonoverlapping groups of hidden units as component learners in order to avoid the negative influence of the breakdown of correlations Gu et al. (2018). Impressively, by using mutual information among the hidden units as the measure of diversity, Brakel et al. proposed a method to learn independent features Brakel & Bengio (2018). Many other studies reported in the literature also investigated the positive role of feature independence in the feature encoding Bengio et al. (2017); Hjelm et al. (2019).

This paper follows the second research direction and further focuses on the role of label information in defining the diversity measure. In fact, while boosting the diversity among the hidden units of DNNs has been shown to be beneficial for the performance in classification tasks, there is no single widely accepted formal definition of diversity measure. It was found that the role of inductive biases should be made explicit and enforced in the process of learning disentangled hidden representations for downstream tasks Locatello et al. (2019). However, the measures of diversity in the current set mostly only considered the correlations among the hidden units over the whole mixed data distribution but neglected the local clustering feature of different classes, which may be harmful to the classification performance Grover & Ermon (2019). A feasible approach to understand the role of label information in defining the diversity measure and obtain an appropriate measure with embedded inductive biases is to investigate the measure of diversity under supervised settings by examining the relationship between diversity and generalization capacity.

To achieve this goal, we first introduce a generalization error bound of DNNs from an information perspective, following the work of Xu et al. Xu & Raginsky (2017), which formalizes the bound as mutual information between the activation values of the hidden units in the specified layer and model parameters from this layer to the end layer (see Fig.1). Intuitively, the new bound describes the information of the extracted features stored in the model parameters, which was proposed as a measure of the effective complexity of a network by Hinton and Van Camp and was used as a regularizer to simplify the networks by Achille and Soatto Hinton & Van Camp (1993); Achille & Soatto (2018). However, the direct usage of this bound is usually difficult due to its excessively complicated estimation in previous work. Nevertheless, compared to the traditional generalization error bounds that are based on hypothesis space, e.g., the Rademacher complexity Boucheron et al. (2005) and the uniform stability Bousquet & Elisseeff (2002), the new bound is tight and simpler in form, making it sufficient for our further analysis.

We then decompose the new bound and naturally obtain a measure of diversity formulated as a difference of two terms, named the label-based diversity measure (LDiversity), where the first term that has been used to obtain independent features Brakel & Bengio (2018) is a canonical unsupervised diversity measure and the second term is a label-based term that has not been considered by the current diversity measures in DNNs; this term is the main difference between our measure and the other measures, reflecting the inductive biases embedded in the hidden representations. Decreasing the defined label-based diversity measure is expected to improve the generalization capacity. Furthermore, if regarding the hidden units as base learners, the proposed diversity measure can also be viewed as the ensemble diversity proposed by Brown et al. Brown (2009); Zhou & Li (2010) for ensemble learning, where the ensemble diversity is shown to be the part of the upper bound of classification error. This means that decreasing LDiversity may also suppress the probability of classification error.

By using LDiversity as the regularizer, we develop a new regularization method named the LDiversity method (LDM), the goal of which is to minimize the classification loss and newly added LDiversity. In particular, the process of minimizing LDiversity is the same as that of training generative adversarial networks (GANs), where two additional “discriminators” are involved to estimate LDiversity by maximizing their output values. Finally, we apply this method to fully connected neural networks and convolutional neural networks. Many experiments show that LDM can effectively reduce the overfitting and decrease the generalization error compared to the methods without the LDiversity regularizer. Furthermore, LDiversity between hidden units is demonstrated to be a crucial factor for reducing the generalization error in DNNs.

Let $\mathcal{Z}=(\mathcal{X},\mathcal{Y})$ be an instance space, where $\mathcal{X}$ is a feature space and $\mathcal{Y}$ is a label space. A training set $S$ of size $n$ is an $n$-tuple, i.e.,

of i.i.d random elements of $Z=(X,Y)\in\mathcal{Z}$ with an unknown PDF $P_{Z}(z)$. Given a neural network with multilayers, let $\hat{h}=(h_{1},h_{2},...,h_{m})$ be the set of all hidden units $h_{i}$ in the discussed layer and $W\in\mathcal{W}$ be the collection of model parameters from the specified hidden layer to the end layer, where $\mathcal{W}$ is the hypothesis space of $W$. Due to the randomness in the realization of the dataset $S$, the values in $W$ are considered to be random variables.

We will make frequent use of the following standard information theoretical quantities Cover & Thomas (1991). For a stochastic variable $X$, its Shannon entropy is defined as

where $\mathbb{E}_{X}[\cdot]$ denotes the expectation of the random object within the brackets w.r.t to the subscript random variable $X$.

The mutual information of two stochastic variables is

which by capturing the nonlinear statistical dependencies between the variables can be reformulated as the Kullback-Leibler (KL-)divergence between the joint density and the product of the marginal densities, i.e.,

which is zero if and only if $h_{1}$ and $h_{2}$ are independent. For more than two variables, the multivariate mutual information is defined as

We will later use it to measure the entanglement in the hidden units and still call it the mutual information in the following for consistency. Consequently, the conditional mutual information of multiple variables given $Y$ is

and will be employed below to describe the class-conditional correlation.

This section gives an upper-bound of the generalization error from an information-theoretical perspective, formulating the absolute difference in the expectation between the expected risk and the empirical risk as the mutual information between the hidden units and the model parameters. We follow the framework proposed by Russo and Xu et al. Russo & Zou (2020); Xu & Raginsky (2017), where it is convenient to think of each unit in the discussed layer as a mapping from the datum to its activation value, i.e., $h_{i}:\mathcal{X}\to\mathbb{R}(i=1,...,m)$, to investigate the effect of entanglement of the hidden units on the generalization capacity since both the expected risk and the empirical risk are initially related to the datum rather than to the values of the hidden units; in fact, we are more interested in the mechanism that produces the activation value than in the activation value itself (see Fig. 1). Consequently, let $\hat{h}(X)=(h_{1}(X),...,h_{m}(X))$ , $f(Z)=(\hat{h}(X),Y)$; then, given $W\in\mathcal{W}$, the loss function $l$ on the sample $Z$ can be restated as a function w.r.t. $f(Z)$ and $W$, i.e., $l:\mathcal{W}\times f(\mathcal{Z})\to\mathbb{R}^{+}$, where $Z=(X,Y)$. Accordingly, let $f(S)=\left(f(Z^{1}),...,f(Z^{n})\right)$. Now, we are ready to obtain the upper bound.

The empirical risk of a hypothesis $W\in\mathcal{W}$ over the dataset $S$ is

The expected risk of $W$ on $P_{S}$ is

where $F^{i}=f(Z^{i})$ $(1\leq i\leq n)$ are i.i.d random variables. Taking expectation on the difference between $L_{f(S)}(W)$ and $L_{\overline{f(S)}}(W)$ with respect to the joint distribution $P_{(S,W)}(s,w)$, we obtain

Then, the generalization error can be decomposed as

We focus on $g(P_{S},P_{W|S})$, which reflects the quality of the generalization of the output hypothesis. Some further steps show that

where $\mathbb{E}_{f(S)\otimes W}$ means taking the expectation w.r.t the product of the marginal PDFs of $f(S)$ and $W$.

Xu and Raginsky (Lemma 1 in Xu & Raginsky (2017)) have justified that given two random variables $X$ and $Y$ with the joint PDF $P_{XY}$ and the product of the marginal PDFs $P_{\overline{X}\overline{Y}}=P_{X}\otimes P_{Y}$, if the function $c(X,Y)$ is a $\sigma-$subgaussian function under $P_{\overline{X}\overline{Y}}$, then

where a random variable $U$ is $\sigma-$subguassian if $\log\mathbb{E}\left[e^{\gamma(U-\mathbb{E}[U])}\right]\leq\gamma^{2}\sigma^{2}/2$ for all $\gamma\in\mathbb{R}$. In fact, if the loss function in Eqs. (7) and (8) is restricted as a function bounded in $[a,b]$, e.g., the sigmoid function, thereby being a $\sigma$-subgaussian function by Hoeffding’s lemma Massart et al. (2007), then $L_{f(S)}(W)$ in Eq. (11) is consequently a $\sigma/\sqrt{n}$-subgaussian function for $W$ due to the independence among $f(Z_{i})(1\leq i\leq n)$, where $\sigma=(b-a)/2$. Then, according to Eq. (12), by setting $X$ and $Y$ in Eq. (12) as $f(S)$ and $W$, respectively, we obtain the following lemma.

The generalization error bound deduced by Lemma 1 may be more tightly coupled to the generalization error than some existing bounds can because the new bound depends on almost all ingredients of learning problems, including the distribution of the dataset, the hypothesis space and the learning algorithm, while some existing bounds, such as VC dimension or Rademacher complexity, mainly depend on the hypothesis space and neglect of the learning algorithm, which potentially means a looser quantity of bound to unify the other ignored ingredients in the learning problems. Moreover, Lemma 1 implies that regularizing the empirical risk with $I(f(S);W)$ may lead to improved genelarization capacity. However, due to the high dimensions of hypothesis space $\mathcal{W}$, the direct usage of $I(f(S);W)$ is usually intractable. In this section, we decompose the upper bound in Eq. (13), remove the terms related to $W$ and naturally derive a label-based diversity measure among the hidden units.

Let us focus on the Eq. (14). There are five terms in the square root. Only the first two terms are completely unrelated to the sample size $n$ and the model parameters $W$, reflecting the relationships among the hidden units. Since the two terms are part of the decomposed upper bound, regularizing the empirical risk with their sum is expected to reduce the upper bound of the absolute value of $g(P_{S},P_{W|S})$ as well as the generalization error. We argue that the two terms naturally quantify the diversity among the hidden units (see Definition 1). For remaining terms: the third term, which is the sum of the respective relevancy of the hidden units $h_{i}(1\leq i\leq m)$ to the labels $Y$, demonstrating the classification ability of each hidden unit itself and as a whole having a positive correlation, with the second term to some extent, is not considered by our diversity measure; the forth term $H(Y)$ is non-optimizable w.r.t. the training process, which is also not considered by the new diversity measure; Compared to the other terms, the last term is the only one related to the sample size $n$ and the model parameters $W$, which makes it unsuitable for describing the diversity among hidden units. Moreover, when the sample size is relatively large, this term tends to be relatively small and consequently has a small effect on the generalization error since the upper bound in Lemma 1 for any reasonable hypothesis necessarily declines to a very small value when the sample size increases. Thus, we do not consider optimizing this term in this work.

As discussed in the introduction, the first term in the diversity measure is the canonical unsupervised diversity measure, which was used to learn independent data representations Brakel & Bengio (2018). It has also been shown that reducing such a term of correlations among the hidden units will lead to an improved generalization capabilityCogswell et al. (2016); Hjelm et al. (2019). The second term in LDiversity is a label-dependent term; it describes the local clustering feature captured by the hidden units. Improving this term may strengthen the class-conditional correlation and make the activations of the same class behave more collaboratively, which is usually important for a classification task. This term is the main difference between our diversity measure and other diversity measures.

It is worth noting that the diversity measure is identical in form to the ensemble diversity measure proposed by Brown (2009); Zhou & Li (2010) for ensemble learning. They derived the ensemble measure by analyzing the upper bound of the probability of the classification error,which is Hellman & Raviv (1970)

where we continue to use the previous symbols and see $\hat{h}=(h_{1},h_{2},...,h_{m})$ as a set of base classifiers for the sample $(X,Y)$; $C(\cdot)$ is any given combination function that minimizes the probatility $P(C(\hat{h})\neq Y)$. By decomposing the mutual information term in Eq. (19), they obtained

and defined the sum of the last two terms as the ensemble diversity measure. Although the two measures have the same form, the ensemble diversity measure is based on the the Bayesian learning framework where only 0-1 loss is permitted, which makes it not very suitable for the cases in deep learning; they also did not propose an effective process for using the measure in practice. Nonetheless, their work implies that if we can regard the hidden units as base classifers, the decrease in LDiversity may well lead to a decrease of the classification error.

In this section, a new regularization method named the label-based diversity method (LDM) is proposed by using LDiversity $D_{LB}$ as the regularizer. Its total loss function is formulated as

where $E_{loss}$ is the premier loss function of the DNNs without any regularizer, for instance, the cross-entropy between the outputs of DNNs and the labels; $D_{LB}$ controls the label-based diversity among the hidden units in one specified layer; and $\lambda>0$ is the balance parameter.

The regularizer $D_{LB}$ is actually the difference of two mutual information terms. Although the estimation of mutual information was recognized as a very difficult problem due to the continuity and high dimensions of data, recent studies Belghazi et al. (2018); Brakel & Bengio (2018) revealed that this problem can be solved in terms of the Donsker-Varadhan representation Donsker & Varadhan (1975) of KL-based mutual information, which is

where $T$ is usually realized as a neural network such that the two expectations are finite. Then, by Eq. (5), the mutual information is estimated by optimizing $T$ to narrow the divergence between the joint distribution and the product of the marginals. However, such strategy for KL-based mutual information may suffer from the instability problem; an alternative approach is to use Jensen-Shannon (JS-)divergence-based mutual information to replace KL-based mutual information as proposed by Brakel and Bengio, where the possible deviation of using JS-divergence is usually acceptable Brakel & Bengio (2018). That is,

where $D_{JS}$ is JS-divergence. It is estimated by

where $\overline{h}=(\overline{h_{1}},\overline{h_{2}},...,\overline{h_{m}})$ obeys the distribution $\otimes_{i=1}^{m}P_{h_{i}}$ and hereinafter $\sigma(\cdot)$ represents the sigmoid function. Similarly, for the conditional mutual information, we have

where taking expectation on $Y$ requires using Eq. (24) to first obtain the corresponding JS-divergence for any given $Y$ and combining the obtained divergence according to the prior probability of $Y$, which is estimated by the proportion of samples of each class to the total. To distinguish the network $T$ used in Eqs. (23) and (25), they are denoted by $T_{1}$ and $T_{2}$, respectively. For brevity, the JS-divergence for conditional mutual information is abbreviated as $D_{JS}^{L}$.

The learning algorithm is finally shown as an iterative min-max process:

where the maximization process guarantees a sufficient approximation to LDiversity $D_{LB}$ by $T_{1}$ and $T_{2}$; and the minimization process is the training process of the regularizing DNNs to obtain the classifier $\theta$. The overall training process of LDM is similar to that of generative adversarial networks (GANs) Goodfellow et al. (2014). In fact, both $T_{1}$ and $T_{2}$ in LDM play the same role as the discriminator in GANs (see Fig. 2).

To implement the learning algorithm presented in Eq. (26), it is important to estimate the expectations first. The expectation taken on the joint distribution $P_{\hat{h}}$ or $P_{\hat{h}|Y}$, can be estimated directly by its average value on the samples from the joint distribution. However, taking expectation on the product of marginals $P_{\overline{h}}$ or $P_{\overline{h}|Y}$ is not straightforward because there are no samples from such a distribution for the empirical estimation. In this work, two strategies are established to approximately obtain samples from the product of marginals according to the type of network. For the case of fully connected neural networks, each sample from the product of marginals with $M$ dimensions is obtained by randomly selecting $M$ samples from the joint distribution, taking the $i$th element from the $i$th selected sample and combining them. For convolutional neural networks (CNNs), after viewing each filter in the discussed layer as a map, each group of the mapped values of all the filters is seen as a sample from the joint distribution. For instance, in the case where there are 3 filters in the the specified CNNs layer, then the samples from the joint distribution are the vectors with 3 mapped values of all the 3 filters. Then, by the same method applied to the fully connected neural networks, we can obtain the samples from the product of marginals. The implementation of LDM is presented in Algorithm 1 (also see Fig. 2).

We apply LDM to fully connected neural networks and convolutional neural networks. We compare it with the method without a regularizer (NONE), dropout with a dropout rate of 0.5 Srivastava et al. (2014), the method with decorrelation regularizer (Decov) in which the hyperparameter was set to 0.1 Cogswell et al. (2016) and the method with the unsupervised diversity term in LDiversity as a regularizer (UDM), in which the balance parameter was set to 0.1 as proposed by Brakel and Bengio in Brakel & Bengio (2018). All the methods involved were implemented by TensorFlow Abadi et al. (2016).

In this paper, by investigating the upper bound of the generalization error from an information perspective, we found that we can naturally derive a measure of the diversity among the hidden units of DNNs, which differs from the other measures because it contains an inductive-bias term. Based on this insight, we designed a regularization method using the diversity measure as the regularizer. Our experiments verified the effectiveness of the proposed method and provided empirical evidence for the validity of our approach.