Implicit Regularization of Dropout

Consider a $L$-layer ($L\geq 2$) fully-connected neural network (FNN). We regard the input as the $0$th layer and the output as the $L$th layer. Let $m_{l}$ represent the number of neurons in the $l$th layer. In particular, $m_{0}=d$ and $m_{L}=d^{\prime}$. For any $i,k\in\mathbb{N}$ and $i<k$, we denote $[i:k]=\{i,i+1,\ldots,k\}$. In particular, we denote $[k]:=\{1,2,\ldots,k\}$.

Given weights $W^{[l]}\in\mathbb{R}^{m_{l}\times m_{l-1}}$ and biases $b^{[l]}\in\mathbb{R}^{m_{l}}$ for $l\in[L]$, we define the collection of parameters $\bm{\theta}$ as a $2L$-tuple (an ordered list of $2L$ elements) whose elements are matrices or vectors

where the $l$th layer parameters of $\bm{\theta}$ is the ordered pair $\bm{\theta}|_{l}=\Big{(}\bm{W}^{[l]},\bm{b}^{[l]}\Big{)},\quad l\in[L]$. We may misuse notation and identify $\bm{\theta}$ with its vectorization $\mathrm{vec}(\bm{\theta})\in\mathbb{R}^{M}$, where $M=\sum_{l=0}^{L-1}(m_{l}+1)m_{l+1}$.

Given $\bm{\theta}\in\mathbb{R}^{M}$, the FNN function $\bm{f}_{\bm{\theta}}(\cdot)$ is defined recursively. First, we denote $\bm{f}^{[0]}_{\bm{\theta}}(\bm{x})=\bm{x}$ for all $\bm{x}\in\mathbb{R}^{d}$. Then, for $l\in[L-1]$, $\bm{f}^{[l]}_{\bm{\theta}}$ is defined recursively as $\bm{f}^{[l]}_{\bm{\theta}}(\bm{x})=\sigma(\bm{W}^{[l]}\bm{f}^{[l-1]}_{\bm{\theta}}(\bm{x})+\bm{b}^{[l]})$, where $\sigma$ is a non-linear activation function. Finally, we denote

For notational simplicity, we denote

where $\bm{f}_{\bm{\theta}}^{j}(\bm{x}_{i}),\bm{W}^{[L]}_{j}\in\mathbb{R}^{m_{L}}$ is the $j$th column of $\bm{W}^{[L]}$, and $f^{[L-1]}_{\bm{\theta},j}(\bm{x}_{i})$ is the $j$th element of vector $\bm{f}^{[L-1]}_{\bm{\theta}}(\bm{x}_{i})$. In this work, we denote the $l_{2}$-norm as $\lVert\cdot\rVert$ for convenience.

The training data set is denoted as $S=\{(\bm{x}_{i},\bm{y}_{i})\}_{i=1}^{n}$, where $\bm{x}_{i}\in\mathbb{R}^{d}$ and $\bm{y}_{i}\in\mathbb{R}^{d^{\prime}}$. For simplicity, we assume an unknown function $\bm{y}$ satisfying $\bm{y}(\bm{x}_{i})=\bm{y}_{i}$ for $i\in[n]$. The empirical risk reads as

where the loss function $\ell(\cdot,\cdot)$ is differentiable and the derivative of $\ell$ with respect to its first argument is denoted by $\nabla\ell(\bm{y},\bm{y}^{*})$. The error with respect to data sample $(\bm{x}_{i},\bm{y}_{i})$ is defined as

For notation simplicity, we denote $\bm{e}(\bm{f}_{\bm{\theta}}(\bm{x}_{i}),\bm{y}_{i})=\bm{e}_{\bm{\theta},i}$.

For $\bm{f}_{\bm{\theta}}^{[l]}(\bm{x})\in\mathbb{R}^{m_{l}}$, we randomly sample a scaling vector $\bm{\eta}\in\mathbb{R}^{m_{l}}$ with coordinates of $\bm{\eta}$ are sampled i.i.d that,

where $p\in(0,1]$, $k\in[m_{l}]$ indices the coordinate of $\bm{f}_{\bm{\theta}}^{[l]}(\bm{x})$. It is important to note that $\bm{\eta}$ is a zero-mean random variable. We then apply dropout by computing

and use $\bm{f}_{\bm{\theta},\bm{\eta}}^{[l]}(\bm{x})$ instead of $\bm{f}_{\bm{\theta}}^{[l]}(\bm{x})$. Here we use $\odot$ for the Hadamard product of two matrices of the same dimension. To simplify notation, we let $\bm{\eta}$ denote the collection of such vectors over all layers. We denote the output of model $\bm{f}_{\bm{\theta}}(\bm{x})$ on input $\bm{x}$ using dropout noise $\bm{\eta}$ as $\bm{f}_{\bm{\theta},\bm{\eta}}^{\mathrm{drop}}(\bm{x})$. The empirical risk associated with the network with dropout layer $\bm{f}_{\bm{\theta},\bm{\eta}}^{\mathrm{drop}}$ is denoted by $R_{S}^{\mathrm{drop}}\left(\bm{\theta},\bm{\eta}\right)$, given by

As $R_{1}(\bm{\theta})$ is independent of the learning rate and $R_{2}(\bm{\theta})$ vanishes in the limit of zero learning rate, we select a small learning rate to verify the validity of $R_{1}(\bm{\theta})$. According to Equation (5), the modified equation of dropout training dynamics can be approximated by $R_{S}\left(\bm{\theta}\right)+R_{1}\left(\bm{\theta}\right)$ when the learning rate $\varepsilon$ is sufficiently small. Therefore, we verify the validity of $R_{1}(\bm{\theta})$ through the similarity of the NN trained by the two loss functions, i.e., $R_{S}^{\mathrm{drop}}(\bm{\theta},\bm{\eta})$ and $R_{S}\left(\bm{\theta}\right)+R_{1}\left(\bm{\theta}\right)$ under a small learning rate.

Fig. 2(a) presents the test accuracy of two losses trained under different dropout rates. For the network trained with $R_{S}\left(\bm{\theta}\right)+R_{1}\left(\bm{\theta}\right)$, there is no dropout layer, and the dropout rate affects the weight of $R_{1}\left(\bm{\theta}\right)$ in the loss function. For different dropout rates, the networks obtained by the two losses above exhibit similar test accuracy. It is worth mentioning that for the network trained with $R_{S}\left(\bm{\theta}\right)$, the obtained accuracy is only $79\%$, which is significantly lower than the accuracy of the network trained through the two loss functions above (over $88\%$ in Fig. 2(a)). In Fig. 2(b), we show the values of $R_{S}\left(\bm{\theta}\right)$ and $R_{1}\left(\bm{\theta}\right)$ for the two networks at different dropout rates. Note that for the network obtained by $R_{S}^{\mathrm{drop}}(\bm{\theta},\bm{\eta})$ training, we can calculate the two terms through the network’s parameters. It can be seen that for different dropout rates, the values of $R_{S}\left(\bm{\theta}\right)$ and $R_{1}\left(\bm{\theta}\right)$ of the two networks are almost indistinguishable.

As shown in Theorem 4, the modified loss $\tilde{R}_{S}^{\mathrm{drop}}(\bm{\theta},\bm{\eta})$ satisfies the equation:

In order to validate the effect of $R_{2}(\bm{\theta})$ in the training process, we verify the equivalence of the following two training methods: (i) training networks with a dropout layer by MSE $R_{S}^{\mathrm{drop}}\left(\bm{\theta},\bm{\eta}\right)$ with different learning rates $\varepsilon$; (ii) training networks with a dropout layer by MSE with an explicit regularization:

with different values of $\lambda$ and a fixed learning rate much smaller than $\varepsilon$. The exact form of $R_{2}(\bm{\theta})$ has an expectation with respect to $\bm{\eta}$, but in this subsection, we ignore this expectation in experiments for convenience.

As shown in Fig. 3, we train the NNs by the MSE $R_{S}^{\mathrm{drop}}\left(\bm{\theta},\bm{\eta}\right)$ with different learning rates (blue), and the regularized MSE $R_{S}^{\mathrm{regu}}\left(\bm{\theta},\bm{\eta}\right)$ with a fixed small learning rate and different values of $\lambda$ (orange). In Fig. 3(a), the learning rate $\varepsilon$ and the regularization coefficient $\lambda$ are close when they reach their corresponding maximum test accuracy (red point). In addition, as shown in Fig. 3(b), we study the value of $\mathbb{E}_{\bm{\eta}}\left\|R_{S}^{\mathrm{drop}}\left(\bm{\theta},\bm{\eta}\right)\right\|/\mathbb{E}_{\bm{\eta}}\left\|\nabla_{\bm{\theta}}R_{S}^{\mathrm{drop}}\left(\bm{\theta},\bm{\eta}\right)\right\|^{2}$ under different learning rates (blue) and regularization coefficients (orange). In practical experiments, we take 3000 different dropout noises $\bm{\eta}$ to approximate the expectation after the training process. The results indicate that the same learning rate $\varepsilon$ and regularization coefficient $\lambda$ result in similar ratios.

Due to the computational cost of full-batch GD, we only use a few training samples in the above experiments. We conduct similar experiments with dropout under different learning rates and regularization coefficients using SGD as detailed in Appendix C.1.

To empirically validate the effect of dropout on condensation, we examine ReLU and tanh activations in one-dimensional and high-dimensional fitting problems, as well as image classification problems. Due to space limitations, some experimental results and detailed experimental settings are left in Appendices A, C.

As can be seen from the implicit regularization term $R_{1}(\bm{\theta})$, dropout regularization imposes an additional $l_{2}$-norm constraint on the output of each neuron. The constraint has an effect on condensation. We illustrate the effect of $R_{1}(\bm{\theta})$ by a toy example of a two-layer ReLU network.

We use the following two-layer ReLU network to fit a one-dimensional function:

where $\bm{x}:=(x,1)^{\intercal}\in\mathbb{R}^{2}$, $\bm{w}_{j}:=(w_{j},b_{j})\in\mathbb{R}^{2}$, $\sigma(x)=\mathrm{ReLU}(x)$. For simplicity, we set $m=2$, and suppose the network can perfectly fit a training data set of two data points generated by a target function of $\sigma(\bm{w}^{*}\cdot\bm{x})$, denoted as $\bm{o}^{*}:=(\sigma(\bm{w}^{*}\cdot\bm{x}_{1}),\sigma(\bm{w}^{*}\cdot\bm{x}_{2}))$. We further assume $\bm{w}^{*}\cdot\bm{x}_{i}>0,i=1,2$. Denote the output of the $j$-th neuron over samples as

The network output should equal to the target on the training data points after long enough training, i.e.,

There are infinitely many pairs of $\bm{o}_{1}$ and $\bm{o}_{2}$ that can fit $\bm{o}^{*}$ well. However, the $R_{1}(\bm{\theta})$ term leads the training to a specific pair. $R_{1}(\bm{\theta})$ can be written as

and the components of $\bm{o}_{j}$ perpendicular to $\bm{o}^{*}$ need to cancel each other at the well-trained stage to minimize $R_{1}(\bm{\theta})$. As a result, $\bm{o}_{1}$ and $\bm{o}_{2}$ need to be parallel with $\bm{o}^{*}$, i.e., $\bm{w}_{1}//\bm{w}_{2}//\bm{w}^{*}$, which is the condensation phenomenon.

In the following, we show that minimizing $R_{1}(\bm{\theta})$ term can lead to condensation under several settings. We first give some definitions that capture the characteristic of ReLU neurons (also shown in Fig. 9).

Drawing inspiration from the methodology employed to establish the regularization effect of label noise SGD [28], we show that under the setting of two-layer ReLU NN and one-dimensional input data, the implicit bias of $R_{1}(\bm{\theta})$ term corresponds to “simple” functions that satisfy two conditions: (I) they have the minimum number of convexity changes required to fit the training points, and (ii) if the intercept points of neurons are in the same inner interval, and the neurons have the same direction, then their intercept points are identical.

It should be noted that not all functions trained with dropout exhibit obvious condensation. For example, a function with dropout shows no condensation while training with a dataset consisting of only one data point. However, for general datasets, such as the example shown in Fig. 1 and Fig. 4, NNs reach the condensed solution due to the limit of the convexity changes and the intercept points (also illustrated in Fig. 10).

Although the current study only demonstrates the result for ReLU NNs, it is expected that for general activation functions, such as tanh, the $R_{1}(\bm{\theta})$ term also has the effect on facilitating condensation, which is left for future work. This is also confirmed by the experimental results conducted on the tanh NNs above. Furthermore, it is believed that the linear term $ax+b$ utilized to ensure $R_{S}(\bm{\theta})=0$ in certain cases is not a fundamental requirement. Our experiments have confirmed that neural networks without the linear term also exhibit the condensation phenomenon.

We first study the effect of dropout on model flatness and generalization. For a fair comparison of the flatness between different models, we employ the approach used in [29] as follows. To obtain a direction for a network with parameters $\bm{\theta}$, we begin by producing a random Gaussian direction vector $\bm{d}$ with dimensions compatible with $\bm{\theta}$. Then, we normalize each filter in $\bm{d}$ to have the same norm as the corresponding filter in $\bm{\theta}$. For FNNs, each layer can be regarded as a filter, and the normalization process is equivalent to normalizing the layer, while for convolutional neural networks (CNNs), each convolution kernel may have multiple filters, and each filter is normalized individually. Thus, we obtain a normalized direction vector $\bm{d}$ by replacing $\bm{d}_{i,j}$ with $\frac{\bm{d}_{i,j}}{\lVert\bm{d}_{i,j}\rVert}\lVert\bm{\theta}_{i,j}\rVert$, where $\bm{d}_{i,j}$ and $\bm{\theta}_{i,j}$ represent the $j$th filter of the $i$th layer of the random direction $\bm{d}$ and the network parameters $\bm{\theta}$, respectively. Here, $\lVert\cdot\rVert$ denotes the Frobenius norm. It is crucial to note that $j$ refers to the filter index. We use the function $L(\alpha)=R_{S}\left(\bm{\theta}+\alpha\bm{d}\right)$ to characterize the loss landscape around the minima obtained with and without dropout layers.

For all network structures shown in Fig. 11, dropout improves the generalization of the network and finds flatter minima. In Fig. 11(a, b), for both networks trained with and without dropout layers, the training loss values are all close to zero, but their flatness and generalization are still different. In Fig. 11(c, d), due to the complexity of the dataset, i.e., CIFAR-100 and Multi30k, and network structures, i.e., ResNet-20 and transformer, networks with dropout do not achieve zero training error but the ones with dropout find flatter minima with much better generalization. The accuracy of different network structures is shown in Table 1.

In this subsection, we study the effect of $R_{1}(\bm{\theta})$ on flatness under the two-layer ReLU NN setting. Different from the flatness described above by loss interpolation, we define the flatness of the minimum as the sum of the eigenvalues of the Hessian matrix $H$ in this section, i.e., $\mathrm{Tr}(H)$. Note that when $R_{S}(\bm{\theta})=0$, we have,

thus the definition of flatness above is equivalent to $\frac{1}{n}\sum_{i=1}^{n}\lVert\nabla_{\bm{\theta}}f_{\bm{\theta}}(\bm{x}_{i})\rVert_{2}^{2}$.

The regularization effect of $R_{2}(\bm{\theta})$ also has a positive effect on flatness by constraining the norm of the gradient. In the next subsection, we compare the effect of these two regularization terms on generalization and flatness.

Although the modified gradient flow is noise-free during training, the model trained with the modified gradient flow can also find a flat minimum that generalizes well, due to the effect of $R_{1}(\bm{\theta})$ and $R_{2}(\bm{\theta})$. However, the magnitude of their impact on flatness is not yet fully understood. In this subsection, we study the effect of each regularization term through training networks by the following four loss functions:

where $\tilde{R}_{2}(\bm{\theta},\bm{\eta})$ is defined as $(\varepsilon/4)\left\|\nabla_{\bm{\bm{\theta}}}R_{S}^{\mathrm{drop}}\left(\bm{\bm{\theta}},\bm{\bm{\eta}}\right)\right\|^{2}$ for convenience, and we have $\mathbb{E}_{\bm{\eta}}\tilde{R}_{2}(\bm{\theta},\bm{\eta})=R_{2}(\bm{\theta})$. For each $L_{i},i\in[4]$, we explicitly add or subtract the penalty term of either $R_{1}(\bm{\theta})$ or $\tilde{R}_{2}(\bm{\theta},\bm{\eta})$ to study their effect on dropout regularization. Therefore, $L_{1}(\bm{\theta})$ and $L_{3}(\bm{\theta},\bm{\eta})$ are used to study the effect of $R_{1}(\bm{\theta})$, while $L_{2}(\bm{\theta},\bm{\eta})$ and $L_{4}(\bm{\theta},\bm{\eta})$ are for $R_{2}(\bm{\theta})$.

We first study the effect of two regularization terms on the generalization of NNs. As shown in Fig. 12, we compare the test accuracy obtained by training with the above four distinct loss functions under different dropout rates and utilize the results of $R_{S}(\bm{\bm{\theta}})$ and $R_{S}^{\mathrm{drop}}(\bm{\bm{\theta}},\bm{\bm{\eta}})$ as reference benchmarks. Two different learning rates are considered, with the solid and dashed lines corresponding to $\varepsilon=0.05$ and $\varepsilon=0.005$, respectively. As shown in Fig. 12(a), both approaches show that the training with the $R_{1}(\bm{\theta})$ regularization term finds a solution that almost has the same test accuracy as the training with dropout. For $\tilde{R}_{2}(\bm{\theta},\bm{\eta})$, as shown in Fig. 12(b), the effect of $\tilde{R}_{2}(\bm{\theta},\bm{\eta})$ only marginally improves the generalization ability of full-batch gradient descent training in comparison to the utilization of $R_{1}(\bm{\theta})$.

Then we study the effect of two regularization terms on flatness. To this end, we show a one-dimensional cross-section of the loss $R_{S}(\bm{\theta})$ by the interpolation between two minima found by the training of two different loss functions. For either $R_{1}(\bm{\theta})$ or $\tilde{R}_{2}(\bm{\theta},\bm{\eta})$, we use addition or subtraction to study its effect. As shown in Fig. 13(a), for $R_{1}(\bm{\theta})$, the loss value of the interpolation between the minima found by the addition approach ($L_{1}$) and the subtraction approach ($L_{3}$) stays near zero, which is similar for $\tilde{R}_{2}(\bm{\theta},\bm{\eta})$ in Fig. 13(b), showing that the higher-order terms of the learning rate $\varepsilon$ in the modified equation have less influence on the training process. We then compare the flatness of minima found by the training with $R_{1}(\bm{\theta})$ and $\tilde{R}_{2}(\bm{\theta},\bm{\eta})$ as illustrated in Fig. 13(c-f). The results indicate that the minima obtained by the training with $R_{1}(\bm{\theta})$ exhibit greater flatness than those obtained by training with $\tilde{R}_{2}(\bm{\theta},\bm{\eta})$.

The experiments in this section show that, compared with SGD, the unique implicit regularization of dropout, $R_{1}(\bm{\theta})$, plays a significant role in improving the generalization and finding flat minima.