On the effect of normalization layers on Differentially Private training of deep Neural networks

Training deep neural networks typically requires large and representative data collections to achieve high performance. However, depending on the application domain, some datasets may contain sensitive information such as medical records of patients or personal financial data. This has motivated the development of dedicated training methods in order to address privacy concerns (see e. g. [30]).

Differential Privacy (DP) [6] provides a concrete cryptography-inspired notion of privacy. In practice, DP algorithms are obtained from non-private algorithms by means of appropriate randomization [8]. Differential Privacy has been integrated into deep learning [25, 1] where privacy issues can arise when the trained model permits the reconstruction of sensitive information that exists in the training data. The proposed method in [1] is based on clipping gradients and adding random noise to them in each iteration of stochastic gradient descent (SGD). Combined with a moments accountant method for tracing the privacy loss, this differentially private SGD (DPSGD) technique has enabled deep neural networks to be trained under a modest privacy budget at the cost of a manageable reduction in the model’s test accuracy. However, for low privacy budgets (i. e. a small $\varepsilon$, see Section 3), which corresponds to a large privacy guarantee, this accuracy drops significantly under DPSGD.

The central role of input noise in the application of DPSGD deserves a more detailed analysis. While adding a small amount of noise during training can benefit the generalization capability of the model, too much noise leads to inferior performance because of the high sensitivity of the output to perturbation in the parameters. In spite of this, we show that neural networks augmented with batch/layer normalization layers are strongly robust against random noise injection in their weights.

In this work, we investigate the impact of batch normalization [11] and layer normalization [2] on the performance of training under privacy constraints. Normalization layers such as batch normalization [11] and layer normalization [2] are indispensable components of nearly all state-of-the-art deep neural networks. These techniques are very essential to robustly train deep neural networks without carefully custom non-linearities [13] or choosing a specific initialization scheme [26]. They also improve generalization by preventing overfitting when training very deep networks [31].

One important property of normalization methods is the invariance of the model to weight matrix re-scaling [2]. We argue that this invariance suggests robustness against noise injection and confirm this hypothesis empirically. In particular, we show that batch-normalization can be integrated with DPSGD without any additional loss in privacy during the training process. We compare our proposal with the current state-of-the-art methods by conducting a series of experiments on Computer Vision and Natural Language Processing tasks. In summary, our contributions are as follows:

• •

  We demonstrate that normalization layers have a substantial impact on the performance of models with noisy parameters and should be considered essential ingredients in robust differentially private training.

• •

  We propose an efficient method for using batch normalization layers without incurring an additional privacy loss in the training procedure. To the best of our knowledge, our work is the first to apply a DP mechanism in the presence of batch normalization.

• •

  We establish new accuracy records for differentially private trained deep networks under DPSGD on the MNIST and CIFAR10 datasets.

The rest of the paper is organized as follows. We study the effect of noise on the performance of models in Section 2. Section 3 introduces our approach for differentially private training deep networks with batch normalization. We compare our method with the existing approaches in Section 4. Finally, in section 5, we discuss the related works on differential privacy.

In this section, we investigate how random Gaussian noise affects a network’s performance in the presence of normalization layers. More specifically, we sample the weights from a Gaussian distribution $\mathcal{N}(\mu,\sigma^{2})$ with learnable mean parameters $\mu$ and constant variance $\sigma^{2}$. Backpropagation is performed by making use of the standard reparametrization trick [24]. This way of training is very similar to variational Bayesian learning of neural networks [3], where weights are represented by probability distributions rather than having a fixed value. Unlike the Bayesian approach though, where the goal is learning the true posterior distribution of the weights given the training data, here the noise is introduced via an ad-hoc distribution function.

Batch normalization [11] and layer normalization [2] are introduced to speed up deep neural network training by regularizing neuron dynamics via mean and variance statistics and reducing variance in the input to each node. Normalization techniques in combination with other architecture innovations like residual connections [10] make training of very deep networks feasible.

Both batch and layer normalization ensure zero mean and unit variance in the output of a layer but using different statistics. Batch normalization (BN) calculates the mean and variance statistics across samples in a mini-batch for each neuron independently, while layer normalization (LN) standardizes each summed input to a node utilizing the statistics over all hidden units.

More precisely, if we denote the weighted summed inputs to the $l$-th layer by $\bm{z}^{l}=\bm{w}^{\bm{T}}\,\bm{a}^{l-1}$ where $a^{l-1}$ is the activation in layer $l-1$ and $\bm{w}$ is the weight matrix, then the normalization operators rescale and shift $z$ according to:

where $\gamma_{i}\in\mathbb{R}$ and $\beta_{i}\in\mathbb{R}$ are learnable parameters which are set to one and zero respectively at the beginning of the training. The parameters $\sigma_{i}^{l}$ and $\mu_{i}^{l}$ are estimated as follows for batch normalization:

and as follows for layer normalization:

where $n$ is the number of hidden units in the layer. The expectations are estimated using samples from the training mini-batches. In the case of batch normalization, at test time, these statistics are replaced with an exponential running average of corresponding mean and variance computed during the training phase.

In this work, we focus on the scale invariance property of normalization methods. It is well known that both batch and layer normalization are invariant under scaling of the weight matrix (matrices) $\bm{\theta}\rightarrow\lambda\,\bm{\theta}$ with arbitrary $\lambda>0$ [2]. In order to demonstrate the effect of this symmetry, consider a deep neural network $f(\bm{\theta}^{*},.)$ which characterizes the relationship from input to output with trained parameters $\bm{\theta}^{*}$. The optimal weights $\bm{\theta}^{*}$ are usually learnt by minimizing a non-convex objective function, $\mathcal{L}(\bm{\theta},.)$, over the training dataset using a variant of stochastic gradient descent (SGD). The training procedure also includes tuning hyperparameters, like learning rate and number of epochs. This tuning is usually done by maximizing the performance of the network on a validation dataset. The training procedure always leaves some small uncertainty of order $\delta\ll 1$ on the final values of weights. Consequently, the performance of the model with parameters $\bm{\theta}^{*}$ and $\bm{\theta}^{*}+\mathcal{O}(\delta)$ will be essentially identical. For example, we may change the number of iterations or the learning rate very slightly without harming the performance of the network.

We can make $f(\bm{\theta}^{*},.)$ invariant with respect to scaling of the weight matrix by augmenting it with batch/layer normalization operators after each learnable layer, $f^{\text{BN/LN}}(\bm{\theta}^{*},.)$. The re-scaling invariance implies that if the weight uncertainty in the original network is of order $\mathcal{O}(\delta)$ then in the augmented network it can be of order $\mathcal{O}(\lambda\delta)$ without affecting the overall performance. This suggests that neural networks with batch/layer normalization layers should be robust against the noise in their weights. In the rest of this section, we empirically confirm this hypothesis by injecting noise into the weights of augmented networks during the training and testing procedure.

To test our hypothesis empirically, we train standard fully-connected as well as convolutional neural networks with noise injected into their weights on MNIST [16] and CIFAR-10 [14]. More specifically, we sample the weights from a Gaussian distribution $\mathcal{N}(\mu,\sigma^{2})$ with learnable mean parameters $\mu$ and constant variance $\sigma^{2}$. Backpropagation is performed by making use of the standard reparametrization trick [24].

In particular, we investigate thoroughly LeNet-300-100 and LeNet-5 [15] and variants of ResNet [10] and VGG [27] models. The structure of the models is outlined in Table 1. For each model, we construct a normalized augmented version by adding batch or layer normalization after each trainable layer. All models are implemented in PyTorch [21] and trained with the Adam optimizer [12].

Table 2 shows the accuracy of the augmented models on the MNIST test dataset, averaged over ten runs against unnormalized baselines. The BN/LN prefixes in this table denote models that are obtained by adding batch normalization or layer normalization layers to the original architectures, respectively. It is evident from this experiment that all augmented models are tolerant to noise while the baselines are not. Indeed, their accuracy does not change at all for a large range of noise levels within the statistical error as promised by the scale invariance property of the networks. On the other hand, the baseline models are very sensitive to small weight perturbations. Notably, disturbing the weights of the baseline models by a small noise of order $\sigma=0.3$ results in non-converging training.

Experiments on CIFAR-10 with ResNet and VGG networks show similar trends (see Table 3). Unlike LeNet models, the original ResNet and VGG networks already contain BN layers after all except the last trainable layer. This leads to some degree of protection against noise as demonstrated in Table 3. For example, the accuracy of ResNet-18 with a noise level of $1$ reduces only to $87\%$ instead of to random prediction.

To illustrate the role of normalization layers further, we also present results on ResNet-18-mod and VGG-16-mod which are obtained by removing the last normalization layer from the original architectures. As shown by the performance degradation with increasing noise levels in Table 3, these models are more vulnerable to the added noise.

Next, we extended our experiments to a more complex task, viz. natural language text classification, where we observe a similar effect. For this, we trained a BiLSTM model with one linear/dense layer (DL) with/without layer normalization on the AG News Corpus¹¹1http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html, a popular text classification dataset with four categories of news: World, Sports, Business and Sci/Tech. Each class has 30,000 training examples and 1,900 test examples. In total, the dataset consists of 120,000 training examples and 7,600 test examples. We further split the training examples into training/validation set where we use 96,000 examples for training the models and 24,000 for validation (e.g early stopping and for tuning adaptive learning rate). We trained the models with different noise levels $\sigma$ for 25 epochs. We find that the higher the noise level, the more epochs are needed to maintain the accuracy of the baseline model. Our results on the language data in Table 4 suggest that layer normalization makes the BiLSTM model robust to noise with minimal drops in accuracy ($1-7\%$). Our findings make us conclude that the LSTM and CNN architectures are robust to noise when equipped with layer and batch normalization.

It is worth mentioning that achieving the same accuracy in the presence of noise does not come for free as it affects the training time: the larger the noise, the slower the training. Figure 1 illustrates the evolution of the accuracy of models on the validation set, for different levels of noise. As it is evident from these plots, models with different values of noise converge to the same accuracy, albeit with different rates. For example, increasing the noise level from 1 to 10 slows down training by a factor of order 7 for LeNet and ResNet models. More details on this can be found in Appendix 2.

Differential privacy (DP) is a systematic approach to quantifying a privacy guarantee while querying a dataset. quantifying the leakage of information due to query a dataset. DP protects privacy by bounding the influence of any sample on the outcome of queries. It provides a provable guarantee for individuals. We proceed by briefly recalling some preliminaries on differential privacy and then propose our approach for training deep neural networks in a differentially private way.

Let us denote the domain of data points by $\chi$. We call two datasets $D_{1},D_{2}\in\chi$ neighboring if they differ are the same except one data point, i. e., $d(D_{1},D_{2})=1$, where $d(.,.)$ is the Hamming distance.

Intuitively a $(\varepsilon,\delta)$ differential private mechanism guarantees that the absolute value of privacy leakage will be bounded by $\varepsilon$ with probability at least $1-\delta$ for adjacent datasets. The higher the value of $\varepsilon$, the more the chance of data re-identification and so information leakage.

A standard approach for achieving differential privacy is to add some random noise $r$ to the output of queries, $q(D)+r$, and to tune the noise $r$ by the sensitivity of the query. $L_{p}$ sensitivity is defined as the maximum change in the outcome of a query for two neighboring datasets and measures the maximum influence that a single data point can have on the result of the query:

The special case where $r$ is calibrated with $S_{2}$ sensitivity and sampled according to the normal distribution is of special importance and is termed Gaussian mechanism:

Here $\mathcal{N}(0,S_{2}^{2}\sigma^{2})$ is the normal distribution with mean zero and standard deviation $S_{2}\sigma$. It can be shown that this mechanism satisfies $(\varepsilon,\delta)$ differential privacy provided that $\sigma\geq\frac{\sqrt{2\ln{\frac{1.25}{\delta}}}}{\varepsilon}$ [8].

Differential privacy has been integrated into deep learning in [25] and subsequently in [1] for the setting where an adversary has access to the network architecture and learned weights, $f(\bm{\theta}^{*},.)$. In particular, the method in [1] preserves privacy by adding noise to the SGD updates:

where $\mathbf{g}_{t}$ is the averaged gradient, $\eta$ is the learning rate and $r$ is sampled from the Gaussian distribution $\mathcal{N}(0,\sigma^{2})$. To control the influence of training samples on the parameters, the gradients are clipped by the $L_{2}$-norm.

where $g_{i}$ is the gradient corresponding to the $i$-th sample and $C$ is the clipping factor. It has been shown in [1] that each step of DPSGD is $(\varepsilon,\delta)$-differential private once we tune the noise as $\sigma=C\,z$ with $z=\frac{\sqrt{2\ln{\frac{1.25}{\delta}}}}{\varepsilon}$.

It is known that batch normalization is not consistent with DP training. Indeed, in a non-private setting, one usually keeps track of the running averages of mean and variance statistics (Eq. 2) during the training procedure and reuses this collected information at test time to normalize the inputs to neurons. More specifically, the update rule for running averages at each iteration is as follows:

where ($\bm{\mu}^{\text{Batch}}$, $\bm{\sigma}^{2,\text{Batch}}$) are estimated mean and variance statistics, ($\bm{\mu}^{\text{Batch}}_{t}$, $\bm{\sigma}^{2,\text{Batch}}_{t}$) are new observed values on iteration $t$ of training according to Eq. 2 and $\alpha$ is the momentum of moving averages.

Since these running averages are also a part of the model’s outputs, in a private training, we have to add noise also to these statistics at each iteration and distribute the privacy budgets among the weights and moving averages to make the overall procedure differentially private. Additionally, we need to truncate the summed inputs of neurons to bound the sensitivity of means and variances, which are given by [28]:

where $C^{\prime}$ is the clipping threshold for neurons activations and $L$ is the batch size. Empirically, we have found that if we tune the noise with the worst-case scenario according to the above sensitivities, the performance of the model drops drastically. Therefore we employ a more sophisticated approach to deal with batch normalization situation as shown in algorithm 1.

First of all, we do not track the running averages during the training phase and instead computed fresh statistics from the current batch are used to normalize the neurons. But to be able to deal with batch normalization at test time we concatenate a fixed amount of data points $\hat{X}$ with size $M$ taken from a public dataset, disjoint from the training data, to the input of the network, both in the training and the test phase. These samples only contribute to the statistics and not to the cost function directly.

Therefore, in the training phase, the cost is computed via $\mathcal{L}(\bm{\theta}_{t},X\cup\hat{X})[:L]$, where $X$ is a batch of size $L$ from the training data, and $[:L]$ denotes the slice of the first $L$ elements. At test time, when iterating over the dataset, the same public data points $\hat{X}$ are also concatenated to each test sample, $x$, and the output of network is computed as $f(\bm{\theta}_{T},x\cup\hat{X})[0]$. This leads to privacy preserving batch normalization and allows us to compute the normalization statistics over a batch of size $1+M$ without any reference to the training data.

In the next section, we present the impact of normalization layers on DP training using two image recognition benchmarks, i.e. MNIST and CIFAR-10, as well as text classification task in natural language processing using AG News Corpus.

In this section, we report some results of applying our method and compare them with existing DP mechanisms. The purpose of these experiments is two-fold: we show that (1) similar to non-private training (section 2), normalization layers improve the performance of models trained with DPSGD; and (2) that training very deep networks is feasible with our private version of batch normalization.

All models, as well as DPSGD, have been implemented in PyTorch [20]. To track the privacy loss over the whole training procedure, we employ the Rényi-DP technique [17]. It provides a tighter bound on the privacy loss compared to the strong composition theorem [9]. We use the open-source implementation of the Rényi DP accountant from the TensorFlow Privacy package [29] ²²2https://github.com/tensorflow/privacy. The total privacy loss $\varepsilon$ is computed as a function of the noise multiplier $z$, size of dataset $N$, size of lot $L$, the number of iterations $T$ and $\delta$.

Table 5 depicts the accuracy of LeNet-5 model on the MNIST test set for $\varepsilon$ ranging from high to very low privacy budgets. We have trained LeNet-5 with and without normalization layers, using DPSGD. For training the model augmented with batch normalization, we employed 128 images of KMNIST [5] as the public dataset. The probability $\delta$ parameter is set to $10^{-5}$ in all our experiments.

The results illustrate that the use of normalization layers consistently improves the performance of DPSGD for all finite values of privacy loss. Further, we observe that the effect of batch normalization is greater than that of layer normalization. Remarkably, we gain around $7\%$ and $10\%$ for very low privacy budgets of $\varepsilon=0.1$ and $\varepsilon=0.05$ with our private batch normalization technique.

We now turn our attention to the CIFAR-10 dataset. Table 6 summarizes the results of DPSGD on the TensorFlow tutorial model considered in [1]. We follow the same experimental setting as in [1], i.e. we fine tune the linear layers of a pretrained model trained on the CIFAR-100 dataset. For training the model with batch normalization we also use 128 images of CIFAR-100, which has completely different image examples and classes from those of CIFAR-10, as our public dataset.

As Table 6 illustrates, using batch normalization results in better accuracies than layer normalization as well as raw TensorFlow tutorial model. We also show the results of training a light weight VGG model in this table (See appendix A for the details of architecture). The non-private accuracy of this model is comparable with the TF-tutorial model but it leads to a much lower gap for finite privacy budgets. It should be mentioned that it is not feasible to train such models without making use of the batch normalization as the training is extremely unstable. Therefore our private friendly batch normalization technique allows training much more deeper and complex networks and establishing new scores for the performance of differentially private models.

Next, we extended our experiments to a more complex task, viz. natural language text classification, where we observe a similar effect. For this, we trained a BiLSTM model with one linear/dense layer (DL) with/without layer normalization (LN) on the AG News Corpus ³³3http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html, a popular text classification dataset with 4 categories of news: World, Sports, Business and Sci/Tech. Table 7 shows the result of our experiment on text classification. As before, layer normalization has a large impact on the performance of the model.

A number of different methods have been developed to preserve privacy in machine learning models. [25] proposed a distributed multiparty learning mechanism for a network without sharing input datasets, however, the obtained privacy guarantee was very loose. [1] developed an efficient differentially private SGD for training networks with a large number of parameters. Also, one may employ tighter bounds provided by the Rényi differential privacy [18] in conjunction with DPSGD. A method for adding less noise to the weights of neural networks is proposed in [23] by using adaptive clipping of the gradients. [19] shows that learning with DPSGD requires optimization of the model architectures and initializations. Perturbing the objective function as an alternative way to protect privacy has been suggested in [22]. A model based on the local differentially private mechanism is proposed in [4] to train deep convolutional networks in a way that a data owner can add a randomization layer before data leave data owners’ device. A comprehensive list of works in this area can be found in [32].

In this paper, we proposed a novel method for integrating batch normalization with differentially private stochastic gradient descent. Our method makes training very deep neural networks, such as ResNet and VGG, feasible under very strong privacy guarantees. We also have demonstrated that normalization layers are essential ingredients for a robust private training.

We would like to thank Marius Mosbach and Xiaoyu Shen for proof-reading and valuable comments. The presented research has been funded by the European Union’s Horizon 2020 research and innovation programme project COMPRISE (http://www.compriseh2020.eu/) under grant agreement No. 3081705.