Gradient Leakage Attack Resilient Deep Learning

This definition states that given $\delta$, a smaller $\epsilon$ would indicate a smaller difference between the output of $\mathcal{M}(A)$ and the output of $\mathcal{M}(A^{\prime})$. By Lemma 3.17 of [23], $(\epsilon,\delta)$-differential privacy ensures that for any adjacent $A,A^{\prime}$, the absolute value of the privacy loss will be bounded by $\epsilon$ with probability of at least $1-\delta$. When $0\leq\delta<1$, this definition implies that it is most likely that the observed output for $A^{\prime}$ will be similar to the output for its neighboring input $A$, whereas when $\delta=0$, it indicates that the output observed under $A^{\prime}$ is highly likely to be observed under $A$. Since $\delta$ is the upper bound probability of $\mathcal{M}(A)$ for breaking $\epsilon$-differential privacy, a smaller $\delta$ is desired. Following the literature [24, 27, 36, 26], $\delta$ is set to $1e-5$ in our experiments such that $\epsilon$-differential privacy would hold with probability of at least 0.9999.

According to this Lemma, noise scale $\sigma$ and privacy loss $\epsilon$ have an inverse correlation given a fixed $\delta$, i.e., a large noise scale indicates a small $\epsilon$, and conversely, a small noise scale implies the spending of a large privacy budget $\epsilon$.

Based on the DP composition theory [23], post processing theory [23], and privacy amplification [42], the privacy budget spending can be tracked throughout the iterations of DNN training. There are four representative privacy accounting methods: moments accountant (MA) [24], zCDP [27], advanced composition (AdvC) [43] and optimal composition (OptC) [44].

By Theorem 2, a randomized function $\mathcal{M}$ that consists of a sequence of $n$ differentially private mechanisms is differentially private, and its privacy guarantee is determined by the sum of the $n$ individual privacy losses, defined by $\sum\nolimits_{i}^{n}\epsilon_{i}$. In our experiments, we consider five privacy accounting rules: base composition, advanced composition, optimal composition, Moments Accountant, and zCDP.

Advanced composition (AdvC [43]): Let $\epsilon,\delta^{\prime}\geq 0$. The class of $\epsilon_{t}$-differentially private mechanisms satisfies ($\epsilon_{t},\delta^{\prime}$)-differential privacy under $T$-fold composition for:

Optimal composition (OptC [44]): Let $\epsilon,\delta^{\prime}\geq 0$. The class of $\epsilon_{t}$-differentially private mechanisms satisfies ($\epsilon_{t},\delta^{\prime}$)-differential privacy under $T$-fold composition for:

Moments accountant (MA [24]): For a mechanism $\mathcal{M}$ with Gaussian noise $\mathcal{N}(0,\sigma^{2}S^{2})$ added at each iteration where $S$ denotes the sensitivity and $\sigma$ is the noise scale, given dataset $\mathcal{D}$ with a total of $N$ data points, the random sampling with replacement and the sampling rate $q=n/N$ where $n$ is the sample size, and the number of iterations $T$, if $q<\frac{1}{16\sigma}$, there exist constants $c1$ and $c2$ so that for any $\epsilon<c1q^{2}T$, $\mathcal{M}$ is $(\epsilon^{\prime},\delta)$-differentially private for $\delta>0$ and

Note that Rényi differential privacy [45, 46] is proposed on top of moments accountant to keep track of the $\epsilon$ privacy spending of a sequence of randomized mechanisms with elegant composition rules. Rényi differential privacy can be transformed to the standard differential privacy definition. We will use Moments accountant for fair comparison with other privacy accounting methods.

zCDP [27]: For a mechanism $\mathcal{M}$ with Gaussian noise $\mathcal{N}(0,\sigma_{t}^{2}S^{2})$ added at each iteration where $S$ denotes the sensitivity and $\sigma_{t}$ is the per-iteration noise scale, given dataset $\mathcal{D}$ with a total of $N$ data points, given the random sampling with replacement and the sampling rate $q=n/N$ where $n$ is the sample size, and the number of iterations $T$, if $q<\frac{1}{16\sigma}$, $\mathcal{M}$ is $(\epsilon^{\prime},\delta)$-differentially private for $\delta>0$ and

Due to the random sampling involved, both advanced composition and optimal composition $(\epsilon_{t},\delta_{t})$ consider an amplified per-iteration privacy spending and assume a fixed $\delta$ to track $\epsilon$ using the following privacy amplification theorem. Note that for Moments accountant, we follow the implementation provided by the Tensorflow Privacy module [47] as no closed-form expression in terms of the noise scale nor the heterogeneous per-step $\epsilon_{t}$ is given to compute the accumulated $\epsilon$.

The goal of deep learning with differential privacy is to train a $(\epsilon,\delta)$-differentially private model over $T$ iterations. By composition theorem, we need to ensure that the per-step SGD, denoted by $f_{t}$, is $(\epsilon_{t},\delta_{t})$-differentially private ($1\leq t\leq T$) with $\epsilon=\sum_{t}^{T}\epsilon_{t}$ and $\delta=\sum_{t}^{T}\delta_{t}$. Hence, for each iteration $t$, we inject differential privacy controlled noise to the gradients before performing per-step SGD. Given a DNN of $M$ layers, the baseline implementation for ensuring that $f_{t}$ is $(\epsilon_{t},\delta_{t})$-differentially private injects noise to all layers of the model during each training iteration.

Fixed privacy parameters. Most of existing approaches to deep learning with differential privacy [24, 27, 36, 48], including Tensorflow privacy module [30] and Pytorch Opacus privacy module [31], all employ fixed privacy parameter strategies, such as the constant clipping method with pre-defined clipping bound (e.g., $C=4$), the fixed pre-defined noise scale (e.g., $\sigma=6$), and the fixed sensitivity $S$ defined using the constant clipping bound $C$. As a result, in addition to distributing the privacy budget $\epsilon$ uniformly across the total $T$ training iterations, the noise variance is fixed during the training, resulting in injecting constant noise to the gradients in each training iteration.

Constant clipping method with fixed clipping bound. Given a training example $i$, let $\nabla W(t)_{im}$ denote the layer-wise per-example gradient vector for the $m^{th}$ layer ($1\leq m\leq M$). The clipping method is used prior to noise injection to address the problem of gradient explosion [49]. With a constant clipping method using a fixed clipping bound $C$, the layer-wise per-example gradient vector $\nabla W(t)_{im}$ is preserved if its $l_{2}$ norm satisfies $||\nabla W(t)_{im}||_{2}\leq C$. Otherwise, $||\nabla W(t)_{im}||_{2}>C$ holds, and the gradient vector $\nabla W(t)_{im}$ needs to be brought down so that its $l_{2}$ norm is capped by $C$. This is done by multiplying every coordinate of the gradients with a scaling factor: $C/||\nabla W(t)_{im}||_{2}$. Such per-example gradient clipping will be performed on all $M$ layers for each example $i$ in the batch $B$ of the given iteration. Let $\overline{\nabla}W(t)_{im}$ denote the clipped per-example gradient for layer $m$ of training example $i$ at iteration $t$ ($m\in\{1,2,...M\}$). The clipped per-example gradients are then gathered for batch averaging: $\overline{\nabla}W(t)=\frac{1}{B}\sum\nolimits_{i}^{B}\overline{\nabla}W(t)_{i}$. Algorithm 1 present pseudo code for baseline clipping function using the constant clipping method initialized with a preset clipping bound $C$ [24].

Noise injection with fixed sensitivity $S$ and fixed $\sigma$. Next, the $(\epsilon_{t},\delta_{t})$-differential privacy controlled Gaussian noise $\mathcal{N}(0,\sigma^{2}S^{2})$ is injected to each layer of the batch gradients: $\widetilde{\nabla}W(t)=\overline{\nabla}W(t)+\mathcal{N}(0,\sigma^{2}S^{2})$. The per-step SGD function $f_{t}$ performs the gradient descent at iteration $t$ using the Gaussian noise perturbed gradients $\widetilde{\nabla}W(t)$, such that $W(t+1)=W(t)-\eta\widetilde{\nabla}W(t)$. This process of sampling, computing gradients, clipping, and noise injection repeats until reaching the $T$ total iterations.

Limitation with using Fixed Privacy Parameters. First, using a fixed clipping bound to define the sensitivity of gradient changes for all iterations can be problematic, especially for the later iterations of training. This is because the $l_{2}$ norm of the layer-wise per-example gradient vector $\nabla W(t)_{im}$ satisfies $||\nabla W(t)_{im}||_{2}\leq C$ in most cases as the training approaching the end. Second, with fixed sensitivity $S$ defined by the fixed clipping bound $C$, the constant noise computed based on $S$ and fixed $\sigma$ can be much larger than $C$ for $\sigma>1$. Third, injecting such large constant noise to gradients in each iteration of the training may have a detrimental effect on the accuracy performance and slow down the convergence of training and sadly it does not gain any additional privacy protection, because the accumulated privacy spending $\epsilon$ is only inversely correlated with $\sigma$.

Furthermore, algorithms with fixed privacy parameters may be more vulnerable to gradient leakage attacks. Any inappropriate setting of fixed privacy parameters will bring in vulnerability to gradient leakage attacks. Figure 3 provides four visual examples on four datasets: MNIST, Fashion-MNIST, LFW, and CIFAR10 under three scenarios: non-private, DP-baseline with fixed parameter setting of clipping bound $C=S=1$ and noise scale $\sigma=1$ as in [50, 29] and DP-baseline with clipping bound $C=S=4$ and noise scale $\sigma=6$ as in [24, 27]. It is observed that an adequate amount of noise is necessary to mask the gradients from malicious or curious inference attackers using the reconstruction learning on the leaked gradients.

Given that gradients at early training iterations tend to leak more information than gradients in the later stage of the training [5], an obvious approach is to design a differential privacy algorithm that can inject larger noise at the early stage of training and resort to smaller noise injection as the training is close to the end. Given that the noise variance $\varsigma$ is the product of sensitivity $S$ and noise scale $\sigma$, several possible strategies can be promising, such as having the sensitivity calibrated to the $l_{2}$-norm of the gradients, or having a smoothly decaying noise scale such that the noise variance follows the trend of gradient updates across the $T$ training iterations.

We describe three different strategies for implementing dynamic sensitivity, aiming to derive declining noise variance as the training progresses in iterations: $l_{2}$-norm based sensitivity, denoted by DP-dynS[$l_{2}$-max], dynamic decaying clipping method, denoted by DP-dynS[$C_{decay}$], and the combination of both, denoted by DP-dynS.

DP-dynS[$l_{2}$-max]. Given that the clipping with a fixed bound $C$ is performed on the $l_{2}$-norm of the layer-wise gradients of per-example in a batch, one way to accommodate the noise variance calibrated to the $l_{2}$ norm of the gradient is to use the max $l_{2}$ norm measured on per-example gradients in a batch $B$ as the sensitivity of the training function $f_{t}$ for iteration $t$ (recall Definition 2). Consider two scenarios: (1) When any of the per-example gradients in a batch is larger than the clipping bound $C$, the sensitivity is set to $C$, and however, (2) when $l_{2}$ norm of all per-example gradients in a batch is smaller than the pre-defined clipping bound $C$, the clipping bound $C$ is unfortunately a loose estimation of the true sensitivity of function $f_{t}$ at iteration $t$. If we instead define the sensitivity of $f_{t}$ by the max $l_{2}$ norm among these per-example gradients in the batch, we will correct the problems in the above scenario (2). In summary, the $l_{2}$-max sensitivity will take the smallest of the max $l_{2}$ norm and the clipping bound $C$. Given that the $l_{2}$-norm of the gradients closely follows the trend of gradient changes as the training progresses in iterations, our dynamic $l_{2}$ sensitivity approach will have adaptive sensitivity $S$, and hence adaptive noise variance when a fixed $\sigma$ is used, and consequently inject larger noise at the early stage of training and smaller differential privacy noise at the later stage of training.

Note that unlike mean estimation [51, 52], knowing the $l_{2}$ norm-based sensitivity of the multi-dimension gradient vector on per-example gradients in deep learning training cannot help disclosing the gradient value at each coordinate. Meanwhile, the $l_{2}$-max norm actually tracks the sensitivity of the training function at each iteration. Yet composition theorem only requires the individual training function component to be differentially private, with each training function component taking care of its own sensitivity. This makes the $l_{2}$-max sensitivity a possible choice for sensitivity optimization.

DP-dynS[C_decay]. The second dynamic sensitivity strategy is to use dynamic decaying clipping. Given that clipping is used to regulate the largest change of gradients during the training, as the training progresses, the maximum changes of the gradients decrease, and gradually approach zero when the training starts to converge. Hence, one may use dynamic decaying clipping method to estimate the maximum changes of the gradients and define the dynamic sensitivity accordingly. Motivated by the dynamic learning rate [53], we propose a set of decaying policies $DECAY_{C}(C_{0},\gamma,t)$ for implementing the decay-clipping based dynamic sensitivity with $\gamma_{1},\gamma_{2},\gamma_{3}$, and $\gamma_{4}$ being the control parameters for the decaying rate.

Linear decay: The clipping decays in a linear trend, defined as $C_{t}=C_{0}(1-\gamma_{1}t)$ where $\gamma_{1}>0$ is the smooth controlling term for clipping at iteration $t$.

Exponential decay: The clipping decays in an exponential trend defined by an exponential function: $C_{t}=C_{0}e^{-\gamma_{2}t}$ where $\gamma_{2}$ is the control term for exponential trending.

Cyclic decay: The clipping decay follows a decaying triangular cyclic policy originated from the cyclic learning rate [54]. With step size $V$, $C_{t}=C_{0}^{*}(1-\gamma_{3}t)$ for odd $V$ and $C_{t}=C_{0}^{*}(1+\gamma_{3}t)$ for even $V$. $C_{0}^{*}$ itself decays every cyclic triangle: $C_{0}^{*}=C_{0}(1-2*\gamma_{4}V)$.

DP-dynS. The third dynamic sensitivity strategy is to combine $l_{2}$-max sensitivity and clipping decay sensitivity $C_{decay}$ such that we can take the advantage of the best side of both worlds. Concretely, although $l_{2}$-max sensitivity is a tighter estimation of the maximum changes in gradients, it may still benefit from additional improvements in some situations, in which the $l_{2}$-max sensitivity happens to be defined by the clipping bound $C$. In such situations, by integrating the decay clipping sensitivity with the $l_{2}$-max sensitivity, the decaying clipping $C$ will be used instead of the initial large clipping bound. By Lemma 1, both the decay clipping-based dynamic sensitivity and $l_{2}$-max sensitivity do not have a direct impact on the differential privacy guarantee. However, with an accuracy target, the model with combined sensitivity optimizations may reach the accuracy and terminate the training earlier, resulting in smaller accumulated privacy spending.

Dynamic noise scale with a decaying policy is an alternative approach to supporting dynamic Gaussian noise variance over the number of $T$ training iterations, denoted by DP-dyn$\sigma$. Recall that the per-example gradients in early iterations are more informative and thus more vulnerable against gradient leakage attacks. With a dynamic decaying noise scale, we can ensure a larger noise scale and thus larger noise variance in the early stage of training and smoothly decay the scale $\sigma$ as training progresses, such that the per-example gradients are perturbed with larger noise in early iterations and relatively less exploitable by the attacker, even when a fixed sensitivity defined by the fixed clipping bound $C$ is used. In our prototype implementation, we implement the dynamic decaying noise scale policies with a starting noise scale $\sigma_{0}$, by employing the same set of three decay policies, denoted by $DECAY_{\sigma}(\sigma_{0},\gamma,t)$, used in designing our decay-clipping dynamic sensitivity.

When combining dynamic $\sigma$ with dynamic sensitivity based on both $l_{2}$-max and decay clipping, denoted by DP-dyn[S,$\sigma$], we expect the best dynamic parameter optimization for differentially private deep learning. Algorithm 2 provides a sketch of the pseudo-code of DP-dyn[S,$\sigma$], which modifies DP-baseline by adding dynamic decay clipping (line 3), dynamic decay noise scale (line 4), and combining with dynamic $l_{2}$-max sensitivity (line 11) for each iteration of the training. Table I provides a summary of comparison for the five different dynamic privacy parameter optimizations and the DP-baseline with respect to the three privacy parameters: clipping $C$, sensitivity $S$ and noise scale $\sigma$. Figure 4 measures the noise variance $\varsigma=S\sigma$ on MNIST with three different settings: (i) fixed $S=C=4$ and fixed $\sigma=6$ (DP-baseline), (ii) $l_{2}$-max sensitivity and fixed $\sigma$ (DP-dynS) and (iii) DP-dyn[$S$, $\sigma$] with $l_{2}$-max sensitivity and decaying noise scale starting from $\sigma_{0}=15$. Compared to the baseline approach with fixed privacy parameters and our methods with dynamic $l_{2}$-max or dynamic clipping, the method of DP-dyn[$S$, $\sigma$] with both dynamic sensitivity S and dynamic noise scale $\sigma$ offers more resilience and better accuracy because it can add larger noise in the early stage of training and small noise as the training progresses towards convergence.

The first set of experiments measures and compares the resilience of alternative DP algorithms with fixed-parameter strategies, existing adaptive clipping approaches, and dynamic parameter strategies against gradient leakage attacks. The following three attack metrics are used to evaluate the adverse effect and cost of gradient leakage attacks, which in turn can be used to measure the resilience of different DP approaches in the presence of attacks. They are attack success rate (ASR), #attack iterations (attack iter) to succeed the inference with 300 as the default, and attack reconstruction distance in MSE (mean square error). The successful attack reconstruction is defined by MSE smaller than 0.70. We report the attack iterations and attack reconstruction distance only for those successful attack examples. For attacks with complete failure, we record their reconstruction distance at the pre-defined maximum reconstruction/inference iterations of 300. Table III reports the attack results under non-private models, the DP-baseline, DP-dyn[S,$\sigma$], and two representative adaptive clipping methods: quantile-based clipping [55], Adaclip [56] and for the four image datasets. AdaClip performs the clipping bound estimation based on the coordinates and adaptively add different noise levels to different dimensions of the gradients, whereas the quantile clipping estimates the clipping bound using the quantile of the unclipped gradient norm. The default setting of $C=4$, $\sigma=6$ is used for DP-baseline algorithms (fixed parameters) and also used as the initial value for quantile-based clipping [55] and DP-dyn[S,$\sigma$]. The attack is performed on the gradients from the first training iteration as gradients at early training iterations tend to leak more information than gradients in the later stage of the training [5]. Figure 5 illustrates the reconstruction results. We make three observations: (1) For all four datasets, the gradients in non-private models are vulnerable to the gradient leakage attack. (2) The quantile-based clipping [55] and Adaclip [56] method can slightly improve the resilience against the gradient leakage attack compared to non-private training. The two adaptive clipping approaches focus mainly on reducing the noise variance rather than gradient attack resilience. Therefore, the DP-baseline, with large and constant noise, is more effective with high resilience against the gradient leakage threats. (3) Compared to both DP-baseline and existing adaptive clipping approaches, DP-dyn[S,$\sigma$] offers the highest resilience against gradient leakage attacks. This is because DP-dyn[S,$\sigma$] adds larger noise at the early iterations of the training thanks to the combined effect of dynamic sensitivity and dynamic noise scale, which effectively creates difficulty for the gradient leakage attack to succeed. The high attack reconstruction distance in Table III shows strong attack resilience against gradient leakage threats. The visualization in Figure 5 also provides an intuitive illustration of this effect. For the two existing approaches, the adaptive clipping brings down the differential privacy noise compared to DP-baseline and thus shows less improvement on the resilience of DP training against gradient leakage attacks, compared to DP-baseline. However, differential privacy noise injection combined with dynamic sensitivity and dynamic noise scale will offer the best defense against gradient leakage threats.

In the rest of the experiments, we provide empirical results and analysis to further illustrate the benefits of the proposed gradient leakage resilient deep learning with dynamic differential privacy parameter optimizations. We first demonstrate how the proposed DP-dyn[$S$,$\sigma$] achieves higher accuracy compared to the DP-baseline at the same ($\epsilon,\delta$) differential privacy level. Then we show how DP-dyn[$S$,$\sigma$] obtains a stronger differential privacy guarantee in terms of smaller $\epsilon$ spending at a target accuracy. Finally, we show that in addition to high resilience against gradient leakage, the proposed dynamic privacy parameter optimizations have an acceptable time cost compared to the conventional DP deep learning baseline. Furthermore, our approach consistently offers high test accuracy on all benchmark datasets, compared to both the conventional DP deep learning baseline and existing adaptive clipping proposals.

This set of experiments compares the DP-baseline with three dynamic sensitivity methods on six datasets: DP-dynS[$l_{2}$-max], DP-dynS[C_decay], and DP-dynS, which denotes the dynamic sensitivity defined by combining $l_{2}$-max sensitivity with dynamic clipping. Given that both DP-baseline and DP-dynS[$l_{2}$-max] use a constant clipping method with fixed pre-defined clipping bound, for a fair comparison, we vary the setting of clipping bound from 1 to 32 by the interval of the power of 2 and set $\sigma=6$. Table IV reports the results. We make three observations. First, DP-dynS[$l_{2}$-max] consistently outperforms DP-baseline under all settings of clipping bounds on all six datasets. This is because by Lemma 1, DP-baseline uses constant noise variance, and injects constant noise in each iteration of the training, regardless of the decreasing trend of gradients in the later stage of the training. Also, DP-baseline is highly sensitive to the settings of clipping bound for all six datasets. It results in lower accuracy when the clipping bound is set too large (e.g. C=16, 32) or too small bound (e.g. C=1). In contrast, DP-dynS[$l_{2}$-max] uses dynamic $l_{2}$-max sensitivity, which changes from iteration to iteration and closely aligns with the declining trend of gradients throughout the training. Second, DP-dynS[$l_{2}$-max] consistently outperforms DP-dynS[C_decay], showing that noise variance $\varsigma$ defined by $l_{2}$-max sensitivity and noise scale $\sigma$ offers tighter alignment than dynamic sensitivity defined by decaying clipping bound $C_{decay}$. A simple policy of linearly decaying from the initial clipping bound $C$ to $C/2$ is used in this experiment. Third, DP-dynS takes the best from both $l_{2}$-max sensitivity and dynamic clipping and consistently outperforms all other alternative approaches on all six datasets. Based on our empirical observations, it is hard to provide a scalable and stable performance for deep learning with DP property when using dynamic clipping to approximate the sensitivity S for two reasons. (1) Even with dynamic decaying clipping like the DP-dynS[C_decay] or AdaClip or Quantile Clip (see Table XI), the dynamic clipping relies on the initial setting of the clipping upper bound and the minimum clipping bound as the training progresses toward convergence to ensure the approximation of the sensitivity is correct. Such max and min clipping bound is to some extent dataset and training task dependent, as shown in Table IV. In comparison, our proposed DP-dynS[$l_{2}$-max] is a scalable and much stable dynamic DP parameter optimization in terms of attack resilience, accuracy, convergence, and privacy spending.

Privacy under five accounting methods. The first set of experiments for privacy analysis is designed to measure privacy spending using five privacy accounting methods for DP-baseline and three alternative optimizations of dynamic sensitivity: DP-dynS[$l_{2}$-max], DP-dynS[C_decay] and DP-dynS. Table V reports the results. We make two observations. First, all five accounting methods show consistent performance for all four DP algorithms on the six benchmark datasets, with Moment Accountant as the most efficient tracking of privacy spending followed by zCDP. Second, DP-dynS consistently outperforms both DP-baseline and the other two dynamic sensitivity algorithms with high accuracy performance, showing that the combo of $l_{2}$ sensitivity with decay clipping method $C_{decay}$ is more effective in enabling the sensitivity $S$ to be aligned with the trend of gradient updates during the training.

Privacy under a target accuracy and a fixed $\sigma$. In this second set of experiments, we analyze and compare DP-baseline with three alternative dynamic sensitivity optimizations in terms of their privacy spending under a target accuracy and a fixed noise scale $\sigma$. We use the accuracy achieved by DP-baseline at $T$ iterations as the target accuracy for each of the six datasets (recall Table 2). Table VI reports the results. DP-dynS is the winner consistently across all six datasets because it is the first to achieve the target accuracy with the smallest privacy spending and hence it is the first to terminate the training. Concretely, it takes 5137, 5532, 6336, 3645, 3690, and 3995 iterations for DP-dynS to achieve the target accuracy for MNIST, Fashion-MNIST, CIFAR10, LFW, Purchase-10, and Purchase-50 respectively. As a result, its accumulated privacy spending is much less than the other three approaches.

Privacy under a target accuracy and a fixed $\varsigma$. The third set of experiments for privacy analysis measures the privacy spending under a target accuracy and a fixed noise variance $\varsigma$. We compare DP-baseline and the three alternative DP algorithms with three different dynamic sensitivity strategies DP-dynS[$l_{2}$], DP-dynS[$C_{decay}$], and DP-dynS on MNIST and CIFAR10. The accuracy of DP-baseline is used as the target accuracy for both MNIST and CIFAR10. We set $\sigma=6$, $C=4$ and $C=8$ and conduct the experiments with two fixed noise variance settings: $\varsigma=24$ with $C=4$ and $\varsigma=48$ with $C=8$. Based on Theorem 2 and Lemma 1, when the noise variance $\varsigma$ is fixed, with dynamic sensitivity $S$, the noise scale $\sigma$ will follow the trend of $S$ in a reverse trend: as $S$ continues to decline with the training progresses in $\#$iterations, the noise scale $\sigma$ will increase to keep the noise scale $\sigma$ constant. As a result, larger sensitivity $S$ in the early stage of the training will lead to a smaller noise scale and larger privacy spending to protect informative gradients. As the training is approaching the end, the sensitivity $S$ will become smaller, resulting in larger $\sigma$ and smaller privacy spending. Table VII shows the results. Consider the first four scenarios, in which the target accuracy is set to the accuracy of DP-baseline for each dataset. We make three observations: (1) DP-dynS consistently outperforms the other three alternatives for the two noise variance settings on both datasets when given a target accuracy and a fixed noise variance $\varsigma$. (2) DP-dynS[$l_{2}$-max] consistently ranked as the second winner with significantly smaller privacy spending than DP-dynS[$C_{decay}$] for all four scenarios under all five privacy accounting methods. This further demonstrates that using adaptive clipping with decay function is still a loose estimation compared to using the $l_{2}$-max sensitivity for differentially private deep learning. Hence, DP-dynS[$C_{decay}$] consumes more privacy budget under both fixed noise variances for both datasets, compared to DP-dynS[$l_{2}$-max] and DP-dynS. (3) When comparing the privacy spending on the same dataset with two different fixed noise variances, or when comparing two different datasets on the same fixed noise variance, it is interesting to note that DP-baseline has the same privacy spending under both variance settings on both MNIST and CIFAR10. This is because the privacy spending $\epsilon$ is anti-correlated to noise scale $\sigma$ (recall Lemma 1) and not sensitive to the different settings of clipping bound for the constant clipping method. However, for DP algorithms with dynamic sensitivity, the privacy spending is also training data dependent. For each of the two fixed noise variances, both with $\sigma=6$, all three dynamic sensitivity algorithms on CIFAR10 will incur higher privacy spending while their respective privacy spending on MNIST will be smaller. This confirms the common knowledge that the gradient update trend during training is model-dependent and dataset-dependent.

The last scenario is included for privacy analysis under two different target accuracy. For MNIST with $C=4$, we set the target accuracy to the accuracy of DP-dynS[$l_{2}$-max] (0.9773) instead of 0.9596 from DP-baseline. We make two interesting observations. (1) DP-dynS remains the winner with strong privacy guarantee at the smallest privacy spending $\epsilon$, followed by DP-dynS[$C_{decay}$]. (2) To achieve the target accuracy of 0.9773, DP-baseline has to enforce a smaller $\sigma$ to maintain the fixed $\varsigma$, resulting in much higher spending of privacy budget according to all five privacy accounting methods. For MA, DP-baseline results in $\epsilon=21.174$ for target accuracy of 0.9773 compared to $\epsilon=0.823$ for the target accuracy of 0.9596 obtained at $T=10000$. For zCDP and base composition, DP-baseline spent $4\times$ of the privacy budget for achieving the target accuracy of 0.9773, compared to the privacy spent for achieving the target accuracy of 0.9596 at $T=10000$.

In this section, we evaluate the effectiveness of incorporating dynamic noise scale into DP-baseline and the three alternative dynamic sensitivity optimized DP algorithms.

Accuracy analysis under a target privacy budget $\epsilon$. The first set of experiments evaluate and compare these eight alternative DP algorithms on MNIST under a given target privacy budget $\epsilon$ such that each algorithm will terminate when its target privacy budget is exhausted. Unless otherwise stated, $\delta=1e-5$, $C=4$ and $T=10000$ for MNIST. Table VIII reports the results. We make three observations. (1) For DP-baseline and the three dynamic sensitivity algorithms under fixed $\sigma=6$, we measure and compare their accuracy since they have the same privacy spending as shown in Table V when reaching $T$ iterations. DP-dynS and DP-dynS[$l_{2}$-max] are clear winners with DP-dynS slightly higher in accuracy. (2) By Lemma 1, the decaying noise scale will lead to increased per-iteration $\epsilon$ spending for early iterations in the training. As a result, the training is terminated early when the accumulated privacy loss reaches the target privacy budget under each accounting method. Different privacy composition methods accumulate heterogeneous privacy spending differently and result in different ending iterations. (3) By integrating dynamic noise scale optimization, one can further improve accuracy performance under all three noise scale decay policies compared to their corresponding algorithms under the fixed $\sigma$. DP-dyn[S,$\sigma$] is consistently the best performing algorithm under all three $\sigma$ decaying policies. Although DP-dyn$\sigma$ with only dynamic noise scale slightly outperforms DP-baseline under the three decaying $\sigma$ policies, dynamic sensitivity approaches consistently outperform DP-dyn$\sigma$ in accuracy performance, showing the critical role of sensitivity in achieving high training utility of DP algorithms. (4) Empirically, noise scale decay with the exponential trend has the best accuracy performance.

Privacy analysis under a target accuracy. In the next set of experiments, we measure and compare the accumulated privacy spending under a given target accuracy to evaluate the effectiveness of the best dynamic parameter optimization DP-dyn[S,$\sigma$] by comparing it with DP-baseline and three alternative dynamic sensitivity algorithms with fixed $\sigma$ for MNIST: DP-dynS[$l_{2}$-max], DP-dynS[C_decay], DP-dynS and DP-dyn[S,$\sigma$]. The target accuracy is set to 0.9596, the accuracy of DP-baseline as provided in Table V with $C=4$ and $\sigma=6$. Table IX reports the results. We make two observations: (1) DP-dyn[S,$\sigma$] with exponential $\sigma$ decay provides the best differential privacy guarantee at the smallest privacy spending $\epsilon$ under all five privacy accounting methods. (2) Both DP-dynS and DP-dynS[$l_{2}$-max] terminate slightly earlier than DP-dyn[S,$\sigma$] under the same target accuracy. This is likely because the large early noise in the dynamic noise scale algorithm may have some marginal effect on the convergence, leading to taking a few iterations more than the DP-dynS in reaching the target accuracy. However, DP-dyn[S,$\sigma$] spent the smallest privacy budget to achieve the target accuracy even with a slightly longer training time. This also shows that early termination does not always guarantee smaller privacy spending. The dynamic sensitivity and dynamic noise scale ultimately control the right amount of noise to be added for achieving the best privacy under a target accuracy.