DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass

We propose DP-Forward, a radically different approach that perturbs forward-pass signals: Users can locally inject noise into the embeddings of (labeled) sequences before sharing them for training, in contrast to perturbing gradients in back-propagation (possibly by an untrusted party). It is meant for provable local DP (LDP) guarantees, thus protecting against stronger adversaries than DP-SGD.

Our approach also naturally fits the federated learning (FL) setting that does not gather users’ data but with substantial differences – FL typically shares noiseless local model updates. Note that any subsequent computation (e.g., gradient computation) on noisy embeddings incurs no extra privacy loss due to the free post-processing of LDP. One might force DP-SGD to offer LDP by adding “enough” noise to the orders-of-magnitude larger per-example gradient from a user, but it may yield unusable models at a similar privacy level.

DP-Forward also extends its applicability to inference via adding noise to users’ test-sequence embeddings, ensuring LDP as in training. As a “side” benefit, it can effectively mitigate emerging embedding-based privacy risks (Pan et al., 2020; Song and Raghunathan, 2020) beyond MIAs.

It is evident that the design goals of DP-Forward naturally align in tandem with our overarching objectives: LDP (vs. CDP), more direct protection of raw data (vs. gradients) against new threats (Pan et al., 2020; Song and Raghunathan, 2020), and can be as efficient as regular non-private training (allowing batch processing of noisy embeddings). The foundation supporting these desiderata, unfortunately, was unavailable. A dedicated mechanism to perturb the forward-pass signals is indispensable.

Specifically, we need to derive noises for embeddings of training/inference text sequences obtained through the forward pass of LM-based pipelines as a real- and matrix-valued function. One might adopt the classical Gaussian mechanism (GM) (Dwork and Roth, 2014) to add i.i.d. noise drawn from a univariate Gaussian distribution. Yet, GM calibrates its noise variance based solely on a sufficient condition for DP, and its variance formula is not applicable to a low privacy regime (Balle and Wang, 2018). Another candidate is the matrix-variate Gaussian (MVG) mechanism (Chanyaswad et al., 2018), tailored for matrix-valued data: It exploits possibly non-i.i.d. noise from a matrix Gaussian distribution to perturb more important rows/columns less. Although it may show better utility over GM (Chanyaswad et al., 2018), it is still sub-optimal due to the sufficient condition.

To optimize MVG, we propose an analytic matrix Gaussian mechanism (aMGM) by integrating a necessary and sufficient condition from the analytic GM (aGM) (Balle and Wang, 2018) for non-i.i.d. noise calibration. Our challenge lies in manipulating the two covariance matrices instead of a single variance. We deduce a constraint only on the two smallest singular values (Section 4.2), indicating that i.i.d. noise (as in aGM) may already be optimal for general applications like DP-Forward.²²2 With extra assumptions, dedicated allocation of other singular values by optimizing/maximizing utility functions specific to applications could help.

A transformer-based pipeline contains an input embedding layer, encoders, and task layers. All these layers prominently manipulate embeddings of text inputs in training and subsequent inference. We investigate adding aMGM noise to embeddings output by any hidden (sub-)layer before task layers (Figure 1). To ensure sequence-level LDP, we need to estimate the $L_{2}$-sensitivity (Dwork and Roth, 2014) of “pre-noise” functions for any two sequences. It is non-trivial since the functions can include different (sub-)layers that may not even be Lipschitz (Kim et al., 2021). Our strategy is to normalize the function outputs to have a fixed Frobenius (or $L_{2}$) norm, similar to gradient clipping (Abadi et al., 2016). It works especially well for deeper sub-layers, achieving comparable task accuracy to the non-private baseline (Section 5). For the first few (sub-)layers, we also make two specializations in relaxing LDP to the token level, elaborated in Appendix A.2, to improve accuracy.

Motivated by prevailing privacy concerns in LM fine-tuning and inference and inherent shortcomings of DP-SGD, we initiate a formal study of an intuitive but rarely studied approach and explore its integration with a transformer-based NLP pipeline. Specifically:

In short, DP-Forward is a better alternative to DP-SGD in training (and testing) deep-learning models, e.g., gigantic LM-based ones.

Modern transformer-based LMs, including BERT (Devlin et al., 2019) and GPT (Radford et al., 2018), are first pre-trained on enormous unannotated (public) corpora to learn contextualized text representations. Later, they can be fine-tuned for various downstream NLP tasks (e.g., sentiment analysis, question answering) using much smaller, task-specific datasets.

We consider BERT (Figure 1), which comprises a stack of $L$ identical layers (i.e., bidirectional transformer encoders (Vaswani et al., 2017)). Each layer has two sub-layers: the dot-product multi-head attention (MHA) (Vaswani et al., 2017) with $h$ heads and a feed-forward network (FFN). Each sub-layer has an extra residual connection, followed by layer normalization (Ba et al., 2016).

Let $X=\langle x_{i}\rangle^{n}_{i=1}$ be an input sequence of $n$ tokens (e.g., characters, words, sub-words, q-grams), where $x_{i}$ is from a vocabulary $\mathcal{V}$. The input embedding layer first maps each $x_{i}$ to its representation in $\mathbb{R}^{d}$, which is the sum of the token, segment, and position embeddings. We re-use $X$ to represent the hidden embedding matrix in $\mathbb{R}^{n\times d}$. For each of $h$ attentions $\mathsf{Att}_{i}$ in the MHA layer, we derive the query, key, and value matrices $Q,K,V\in\mathbb{R}^{n\times d/h}$ ($h$ divides $d$) by multiplying $X$ with head-specific weights $W^{Q},W^{K},W^{V}\in\mathbb{R}^{d\times d/h}$. Its output is

The input to $\mathsf{softmax}(\cdot)$ is an $n\times n$ matrix of pairwise dot products. Finally, MHA concatenates (denoted by $||$) all the head outputs into a matrix in $\mathbb{R}^{n\times d}$, right multiplied by a projection matrix $W^{O}\in\mathbb{R}^{d\times d}$:

FFN is composed of two linear mappings with a ReLU activation in between. It separately and identically operates on each $x_{i\in[1,n]}$,

where $W_{1}$, $W_{2}$, $b_{1}$, and $b_{2}$ are trainable matrix/vector-valued parameters. Its output on $X$ is $\mathsf{FFN}(X)=[\mathsf{FFN}(x_{1})^{\top}||\cdots||\mathsf{FFN}(x_{n})^{\top}]$. The residual connection for sub-layers is $X+\mathsf{MHA}(X)/\mathsf{FFN}(X)$. The layer normalization $\mathsf{LN}(x_{i})$ normalizes all $x_{i}$ entries to have zero mean and unit variance using an extra scale-then-shift step.

At the output of the final encoder, the hidden embedding matrix is reduced to a sequence feature in $\mathbb{R}^{1\times d}$. Standard reduction methods include mean pooling (Reimers and Gurevych, 2019) (computing $\sum^{n}_{i=1}x_{i}/n$) or taking the last embedding of a special token [CLS] for classification (Devlin et al., 2019).

The pre-training of BERT is based on two self-supervised tasks: masked language model (MLM) and next sentence prediction (Devlin et al., 2019). We adopt MLM: It randomly masks out some tokens, indexed by $\mathcal{I}$, in an input sequence $X$. The objective is to predict those masked tokens using their context by minimizing the cross-entropy loss

where $\theta$ denotes all the parameters of BERT transformer encoders.

DP (Dwork et al., 2006) is a rigorous, quantifiable privacy notion. It has two popular models, central and local. In central DP, a trusted data curator accesses the set $\mathcal{X}$ of all individuals’ raw data and processes $\mathcal{X}$ by a randomized mechanism $\mathcal{M}$ with some random noise. Formally:

The neighboring notion is application-dependent (to be discussed in Section 3.1). Typically, it involves the “replace-one” relation: $\mathcal{X}^{\prime}$ can be obtained from $\mathcal{X}$ by replacing a single individual’s data point (e.g., a sequence-label pair). CDP offers plausible deniability to any individual in a dataset. In contrast, local DP (LDP) (Kasiviswanathan et al., 2008) removes the trusted curator, allowing individuals to locally perturb their data using $\mathcal{M}$ before being sent to an untrusted aggregator for analytics.

Embeddings $f(X)$ encode semantic information of input sequences $X$, each of which has $n$ tokens (Section 2.1). Fine-tuning (or subsequent inference of) NLP pipelines essentially processes $f(X)$. DP-Forward fine-tuning protects every $X$ by an output perturbation mechanism $\mathcal{M}$ over $f(X)$, in contrast to DP-SGD, which perturbs aggregates of gradients $f^{\prime}(X,y)$ over $X$ and label $y$. Simply put, our $(\epsilon,\delta)$-LDP holds for $X$ while DP-SGD provides CDP for $(X,y)$.

Sequence-only protection is meaningful since sequences often convey (implicit) sensitive information (e.g., authorship), whereas labels (e.g., a single bit denoting positive/negative) can be public. We defer to Section 3.4 for achieving “full” LDP over $(X,y)$. To bridge the gap between theoretical guarantees of DP-SGD and DP-Forward, we first define sequence DP⁴⁴4One could generalize it to “feature” (or “input”) DP, as DP-Forward also allows other types of features beyond embeddings (and its essence is input-only privacy). To keep our focus on NLP, we use “sequence” here. (PixelDP (Lécuyer et al., 2019) treats pixels as image features.) (SeqDP) in the central setting.

DP-Forward in our paper (except Appendix A) applies the general normalization approach to any $f(\cdot)$ for sequence-level (Seq)LDP.

Let $f(\cdot)$ be an arbitrarily deep forward pass, ranging from the first (input embedding) layer itself to all but the last (task) layer in a BERT-based pipeline (Figure 1). Correspondingly, let $p(\cdot)$ be the remaining layers, ranging from the last task layers themselves to all but the first (input embedding) layer. Every sequence $X$ becomes an embedding matrix $f(X)\in\mathbb{R}^{n\times d}$ at the output of layers in encoders or $\mathbb{R}^{1\times d}$ before task layers (Section 2.1). To offer $(\epsilon,\delta)$-SeqLDP, we adopt a suitable output perturbation mechanism $\mathcal{M}$, such as GM, considering that a dataset has only one labeled sequence.

Since $\mathcal{M}$ can work on the output of any hidden layer, estimating $S_{2}(f)$ is non-trivial. Specifically, $\mathsf{MHA}$ itself, let alone more layers included, is not Lipschitz continuous, meaning its outputs can change arbitrarily for even slight input variation (Kim et al., 2021). To address this, our approach is to normalize or clip the function outputs:

as in DP-SGD (Abadi et al., 2016), where $C$ is a tunable parameter. We then have $S_{2}(f)=2C$. Such normalization makes task utility less “sensitive” to the choice of $C$ since signal and noise increase proportionally with $C$, whereas the signal may be unchanged when $f(\cdot)$ is not clipped. It also has many other benefits, such as stabilizing training, avoiding overfitting, and accelerating convergence (Aboagye et al., 2022). Hence, we resort to normalization in our experiments. One can then calibrate Gaussian noise $Z$ and derive $f(X)+Z$ for the post-noise layers $p(\cdot)$.

Note that we remove the residual connection when adding noise to the output of the first MHA layer to avoid $p(\cdot)$ reaccessing $X$ (dashed arrow, Figure 1) to maintain free post-processing. This may lead to instability (e.g., gradient vanishing) (Yu et al., 2021), but it can be mitigated by pre-training new BERT without such a residual connection to keep consistent with later fine-tuning/inference.

DP-Forward using GM suffers from the “curse of dimensionality” when $d$ is large (e.g., $768$ for BERT-Base). To alleviate the curse, we can append two linear maps, $M_{1},M_{2}\in\mathbb{R}^{d\times d^{\prime}}$ with $d^{\prime}\ll d$, such that $f(\cdot)$ and $p(\cdot)$ respectively have $M_{1}$ and $M_{2}$. Both maps are randomly initialized and updated like other weights using gradients. The raw embedding matrix is first right multiplied by $M_{1}$, leading to $\mathbb{R}^{n\times d^{\prime}}$ or $\mathbb{R}^{1\times d^{\prime}}$, before being normalized. Our privacy guarantee will not be affected since $S_{2}(f)$ remains the same. We then use $M_{2}$ to restore the dimensionality to be compatible with the raw pipeline; $M_{2}$ incurs no extra privacy loss due to the free post-processing. Nevertheless, it needs dedicated efforts to modify the pipeline; dimension-reduced embedding matrices may also lose useful information, degrading task utility. We thus make $M_{1}$ and $M_{2}$ optional (see Section 5.2).

Suppose we use a raw, public BERT checkpoint⁶⁶6Using noisy BERT for fine-tuning (and subsequent inference) is deferred to Section 3.6. for fine-tuning. In the forward pass of the $i$-th ($i\geq 1$) step, it offers the latest $f^{(i-1)}(\cdot)$ to a batch of users, mimicking the regular mini-batch SGD. $f^{(0)}$ is from the raw checkpoint. Users are randomly chosen (without replacement), and their number is a fixed parameter. Users in the batch individually compute their noisy embeddings $f^{(i-1)}(X)+Z$ to ensure SeqLDP (Theorem 3). They then send them with unperturbed labels $y$ to the service provider, who runs $p^{(i-1)}(\cdot)$ over $(f^{(i-1)}(X)+Z,y)$ to compute the batch loss; any post-processing of embeddings under SeqLDP incurs no extra privacy degradation on $X$. $p^{(0)}$ here includes the rest raw BERT part and randomly initialized task layers.

During the back-propagation, the service provider can update $p^{(i-1)}(\cdot)$ to $p^{(i)}(\cdot)$ via the gradient (derived from the loss and noisy embeddings) of the post-noise layers. To avoid accessing users’ raw $X$, it needs to freeze the pre-noise layers $f^{(i-1)}(\cdot)$ as $f^{(0)}$. Parameter freezing is compatible with the more recent zero-shot or in-context learning paradigm (Min et al., 2022). It is useful when models are gigantic and full fine-tuning is expensive. However, the more layers are frozen, the worse the utility might be (even in non-private settings).

There are two general ways to update $f^{(i-1)}(\cdot)$ securely: i) We can assume an extra trusted party (as in DP-SGD), but it becomes central DP. ii) Users can first derive the gradients for the layers inside $f^{(i-1)}(\cdot)$ locally on their $X$ and then resort to secure aggregation (Bonawitz et al., 2017) for global updates at the service provider. However, it is costly. For better utility, we update $f^{(i-1)}(\cdot)$ in experiments, requiring us to consider privacy degradation across different epochs due to the composability (as detailed below). Dedicated approaches (that balance efficiency, privacy, and utility) are left as future work.

The proof follows that of GM (Dwork and Roth, 2014). The crux is that $S_{2}(f),\forall X,X^{\prime}$ is given by the output normalization, independent of the inputs.

DP-Forward ensures SeqLDP for fine-tuning, while DP-SGD offers central DP (for sequence-label pairs). To facilitate a fair comparison (on privacy-utility tradeoffs), we make two changes. First, we also perturb the labels with a suitable mechanism for the standard LDP, i.e., extending the protection from sequence to sequence-label pairs. Second, we use shuffling (Erlingsson et al., 2019) to “translate” our (label-protected) DP-Forward with LDP to claim (example-level) CDP as DP-SGD.

When $|\mathbf{y}|$ is large, we can use prior to “prune” $\mathbf{y}$ to smaller $\mathbf{y}^{\prime}$ (Ghazi et al., 2021). The prior can be publicly available (e.g., auxiliary corpora similar to the users’ data) or progressively refined from a uniform distribution via the multi-stage training (Ghazi et al., 2021). One can then estimate an optimal $|\mathbf{y}^{\prime}|$ by maximizing the probability that the output is correct, i.e., $\Pr[y=\hat{y}]$. With (prior-aided) RR (Ghazi et al., 2021), we can achieve full LDP.

The proof follows from the basic composition theorem (Dwork and Roth, 2014).

Suppose we train for an epoch, and the normalization factor is $C$. For DP-SGD, the batch size is $b$; the subsampling probability and the number of training steps are respectively $b/N$ and $N/b$. If each step is $(\epsilon,\delta)$-DP, the overall privacy loss is $(O(\epsilon\sqrt{b/N}),\delta)$-DP using the strong composition and privacy amplification by subsampling (Abadi et al., 2016).

DP-Forward with shuffling can also be seen as composing $N$ subsamplings, each a fraction of size $1$ (Steinke, 2022). It is $(O(\epsilon\sqrt{1/N}),\delta)$-DP, which is “amplified” from $(\epsilon,\delta)$-LDP. For an easier analysis of SNR, we omit $\epsilon_{2}$ of RR since the overall $\epsilon$ is dominated by composing subsampled Gaussian. So, our Gaussian noise variance is $b\times$ smaller than DP-SGD’s in each step; the SNR of each entry in embeddings vs. the aggregation of $b$ gradients can be estimated as $O(C/\sqrt{nd})$ for DP-Forward vs. $O(C/\sqrt{d^{\prime}})$ for DP-SGD, where $d^{\prime}$ is the gradient dimension and is much larger than $nd$, the embedding-matrix size.

Given only fine-tuned pipeline parts $f(\cdot)$, users can derive the noisy embedding matrices of their test sequences for inferences at the service provider while ensuring $(\epsilon,\delta)$-LDP. Inference using noise aligned to the noisy fine-tuning is also beneficial for task accuracy.

Local inference (as in DP-SGD) without noise forces the service provider to reveal its entire pipeline, losing its intellectual property and incurring more time and storage costs for both $f(\cdot)$ and $p(\cdot)$.

The proof is inherited from GM (Dwork and Roth, 2014). Different from DP-Forward fine-tuning, LDP holds for test sequences since the labels are absent.

Directly using the raw BERT might not “match” DP-Forward fine-tuning/inference, degrading task utility. Pre-training BERT with DP-Forward on publicly available text (e.g., Wikipedia), besides the private user-shared data, can make future operations “adaptive” to noise. It requires us to modify the raw MLM objective in Eq. (1):

where $\theta^{*}$ denotes the parameters of “noisy” BERT. This endows the noisy BERT with some “de-noising” ability since the objective is to predict the raw masked tokens from noisy embeddings $\mathcal{M}(f(\hat{X}))$. It does not really breach privacy due to the free post-processing; LDP is ensured for each sequence, as the pre-training is self-supervised (without labels). Such noisy pre-training can also be outsourced to dedicated GPU clusters, enabling “de-noising BERT as a service.”

De-noising as post-processing is not new, but most prior arts need prior knowledge, e.g., Bayesian prior. aGM formulates it as an unusual estimation problem since a single noisy output is observed for each input, which can then be solved by appropriate estimators, e.g., the Bayesian one (Balle and Wang, 2018). Another attempt (Lécuyer et al., 2019) trains a separate noisy auto-encoder, which learns the identity function $f(X)=X$ stacked before an image classification network, to de-noise the noisy input. It has limited applications for only noisy input embeddings and incurs extra changes when migrating it to an NLP pipeline.

In contrast to the classical GM, MVG adopts possibly non-i.i.d. noise $Z\in\mathbb{R}^{n\times d}$ drawn from the zero-mean matrix Gaussian distribution $\mathcal{MN}_{n,d}(0,\Sigma,\Psi)$, where $\Sigma\in\mathbb{R}^{n\times n}$ and $\Psi\in\mathbb{R}^{d\times d}$ are the row- and column-wise covariance matrices. Intuitively, it adds less noise to more “important” rows or columns for possible better utility.

The definition is equivalent to the conventional form given by the matrix trace. It generalizes the univariate Gaussian used in GM; $Z$ becomes i.i.d. when $\Sigma,\Psi$ are diagonal and equal-valued. Below recites the main theorem of the MVG mechanism for $(\epsilon,\delta)$-DP.

To illustrate how the MVG mechanism works, we quote an example (Chanyaswad et al., 2018): performing regression using an identity query on a liver disorders dataset (McDermott and Forsyth, 2016) with $6$ features and $248$ samples (i.e., $f\in\mathbb{R}^{248\times 6}$). MVG treats ‘ALT’ and a teacher label as the two most indicative features based on some prior, thus added with less noise (Chanyaswad et al., 2018). To report the best empirical results, it tries different precision budget (or noise variance) allocation strategies so that the total budget (Theorem 2) is not overspent. For example, it evenly allocates $\tau>50\%$ (a tunable parameter) of the budget to the two important features and the rest to the other four. Compared to GM using i.i.d. Gaussian noise, MVG can improve root mean square error (RMSE) by up to $0.003$ at the same privacy level (Chanyaswad et al., 2018).

At least two slacks caused the sub-optimality. The first and foremost is due to a sufficient condition for $(\epsilon,\delta)$-DP (Dwork and Roth, 2014): $\Pr[\mathcal{L}_{\mathcal{M},\mathcal{X},\mathcal{X}^{\prime}}\geq\epsilon]\leq\delta$, which is also used in the classical GM. With the Laurent-Massart Theorem (Laurent and Massart, 2000), MVG further transforms it to $\Pr[\mathcal{L}_{\mathcal{M},\mathcal{X},\mathcal{X}^{\prime}}\leq\epsilon]=1$ for a subset of all the possible outputs. The second lies in a loose matrix-trace-based privacy analysis; a follow-up (Yang et al., 2023) derives a tighter bound from Definition 1 and a matrix-norm inequality.

To enhance MVG while still adding possibly non-i.i.d. noise $Z\sim\mathcal{MN}_{n,d}(0,\Sigma,\Psi)$, we put forth the analytic matrix Gaussian mechanism (aMGM) by exploiting a necessary and sufficient condition for $(\epsilon,\delta)$-DP, which is formulated using two PLRVs by the analytic GM (aGM) (Balle and Wang, 2018). It is non-trivial⁷⁷7A recent pre-print (Yang et al., 2021) also studied using matrix Gaussian distribution. The proof of (Yang et al., 2021, Lemma $4$), pivotal for our Theorem 6, is problematic. We prove it in Appendix C. since we now need to work with two covariance matrices $\Sigma$ and $\Psi$ instead of a single variance $\sigma^{2}$ in aGM.

It directly implies the sufficient condition due to $\Pr[\mathcal{L}_{\mathcal{M},\mathcal{X}^{\prime},\mathcal{X}}\leq-\epsilon]\geq 0$. We next show that $\mathcal{L}_{\mathcal{M},\mathcal{X},\mathcal{X}^{\prime}}$ or $\mathcal{L}_{\mathcal{M},\mathcal{X}^{\prime},\mathcal{X}}$ of aMGM is also Gaussian, a similar result has been proven in aGM (Balle and Wang, 2018, Lemma 3).

With Lemma 4, we can then specialize the left-hand side of Eq. (3). Particularly, we use the Gaussian cumulative density function (CDF)

to explicitly express the two probabilities (see Lemma 5) instead of approximating them by the tail bounds of a Gaussian distribution.

We can further re-write the left-hand side of Eq. (3) as $g(||\Delta^{\prime}||_{F})$:

a function of $\Delta$ and $(\Sigma,\Psi)$; it is defined w.r.t. $\Delta$ and $\sigma^{2}$ for aGM (Balle and Wang, 2018). To satisfy Theorem 3, we require $g(||\Delta^{\prime}||_{F})\leq\delta,\forall\mathcal{X}\simeq\mathcal{X}^{\prime}$. Since $g(\cdot)$ is monotonically increasing (Balle and Wang, 2018, Lemma 7), we first find the upper bound $\mathcal{B}$ of $||\Delta^{\prime}||_{F}$ as the “solution” to $g(||\Delta^{\prime}||_{F})=\delta$ and then determine $U,V$ (hence $\Sigma,\Psi$) based on $\mathcal{B}$ and $\Delta$ with $S_{2}(f)=\sup_{\mathcal{X}\simeq\mathcal{X}^{\prime}}||\Delta||_{F}$.

We use three typical datasets/tasks that are widely used in NLP/DP literature (Yu et al., 2021; Li et al., 2022b; Yu et al., 2022; Yue et al., 2021) and GLUE benchmark (Wang et al., 2019): i) Stanford sentiment treebank (SST-2), ii) Internet movie database (IMDb) (Maas et al., 2011) for binary sentiment classification of single- and multi-sentence movie reviews, and iii) Quora question pairs (QQP) for semantic equivalence test over question pairs on Quora.com. Their test sets do not have any labels; we use the original dev sets as the test sets. Table 2 summarizes their characteristics. They all carry privacy risks; e.g., stylistic features of posts may leak the author’s identity. We use task accuracy (w.r.t. the ground truth labels) as the utility metric.

By default, we report the accuracy of DP-Forward inferences on tasks fine-tuned using DP-Forward (with ${\sim}2$pp gains compared to the case of “DP-Forward fine-tuning + non-private inference”). We also realize DP-SGD fine-tuning with the latest Opacus (Yousefpour et al., 2021) but do not add any noise to its inference. Another baseline is non-private (in both fine-tuning and inference).

For the hyperparameters used throughout our experiments, we set the number of training epochs $k=3$, learning rate $\eta=2\mathrm{e}{-5}$, batch size $b=32$, and normalization/clipping factor $C=1$. We keep others (e.g., no weight decay, no learning rate decay) default as literature (Wang et al., 2019). The privacy parameter $\delta$ is fixed as $1\mathrm{e}{-5}$ (Yu et al., 2022).

The sequence length $n$ is variable. While the hidden dimensionality $d$ is tied as $768$ for BERT-Base, we can resort to two linear maps for “mediating” it (see Section 3.2). Since we normalize embedding matrices of size $n\times d$ to have a fixed norm $C$, each entry’s signal magnitude relies on $(n,d)$. In contrast, the noise variance is the same given $C$ and fixed privacy parameters. The signal-to-noise ratios (SNRs) affecting accuracy can be configured based on $(n,d)$.

Figure 2 shows the evaluation accuracy of SST-2 fine-tuned using DP-Forward with $n$ tuning from $16$ to $256$. We study adding aMGM noise at five hidden layers’ outputs. The results indicate that the best accuracy is often achieved at $n=64$ or $128$, so we opt for $n=128$ (which is sufficient for most sequences) in subsequent experiments.