Private Language Model Adaptation for Speech Recognition

FL distributes the model training process across a large number of client devices. Each device trains on private data and computes model updates independently. Those updates are then uploaded to a central server for aggregation and the updated model is deployed to each client afterwards. We describe our approach on FL-based NNLMs adaptation for ASR as follows.

Pre-training. We train an initial NNLM using a large general corpus and if available, any “close-in-domain” proxy data on the server side. The model then is delivered to each local device along with an ASR model;

Client-side update. As soon as a user has input sufficient volume of utterances, local personal transcribed data is used to fine-tune all parameters of the current NNLM on the device. Then the client model update is sent back to the server if the device is selected to join the cohort;

Server-side update. Once adequate client model updates are received by the server, global model update is conducted and an updated server model is deployed to each local device.

The client-side and server-side updates above are repeated until model convergence. The procedure is outlined in Algorithm 1, with more details in subsections 2.2, 2.3, and 2.4.

Upon the initial deployment, on-device ASR model with the pre-trained NNLM for second-pass rescoring runs on each local client to transcribe user’s utterances. The decoded hypotheses are then stored on the local device and serve as in-domain data for on-device NNLM adaptation. In this work, only the 1-best hypothesis of each utterance is stored and used for on-device training.

Suppose at round $t$ of FL training, each selected client downloads the NNLM $\theta_{t}$ from server and performs secure local training on their own device, that is, fine-tuning $\theta_{t}$ using private data. Mini-batch stochastic gradient descent (SGD) is used as the local optimizer. Specifically, in the $k$th training epoch, transcripts data on client $i$ is split into multiple batches. For the $b$th batch, let us denote $\theta_{i,k,b}^{t}$ as the current client model and $g_{i,k,b}^{t}$ as the computed gradients after back-propagation. Then the client model is updated as

where $\eta_{l}$ is a local learning rate. After $K$ epochs of training, the client uploads its model update (i.e. difference of model parameters; see subsection 2.4) to the central server over a secure connection.

In the next subsection, we describe how to compute $g_{i,k,b}^{t}$ in speech LM task, and how to utilize token-level confidence scores to address the potential quality degradation due to mis-transcribed text as training data.

Cross entropy loss is usually used for NNLM training. The following shows this function for the $b$th batch of $k$th local training epoch on client $i$

where $n_{b}$ denotes the batch size, $T_{j}$ refers to the sequence length, $v_{j,s}^{*}$ represents the $s$th word of the $j$th transcript, and $\hat{p}_{j,s,v_{j,s}^{*}}$ indicates the predicted probability of observing $v_{j,s}^{*}$ over the entire vocabulary.

Adapting NNLMs using ASR transcribed data has the challenge of dealing with potential transcription errors. In this work, we leverage external confidence classifier models to mitigate this issue. Specifically, we modify the NNLM training objective using confidence scores. Let $\hat{c}_{j,s}$ be the estimated confidence score on the word of $v_{j,s}^{*}$ in the $j$th transcript, and

be the estimated utterance-level confidence score, which is the average of token-level confidence scores. We propose the following three modified loss functions for NNLM training.

Hard thresholding. We adopt utterance-level confidence scores for training data selection, which amounts to exclude the set of utterances

from training, where $c$ is some fixed constant. Notice that this method is equivalent to include an indicator function as a multiplier on each utterance in the loss function.

Utterance-level weighting. The utterance-level confidence scores are leveraged for loss weighting

Token-level weighting. We utilize token-level confidence scores for weighting in the loss function

Suppose that at round $t$, the server has the model $\theta_{t}$ and samples a set $\mathcal{I}_{t}$ of clients. Let $\theta_{i}^{t}$ denotes the model of each client $i\in\mathcal{I}_{t}$ after local training, and $\Delta_{i}^{t}:=\theta_{t}-\theta_{i}^{t}$ be the model difference on client $i$ which is sent back to the server. Let

be the averaged model difference or “pseudo-gradient” which is used in general server optimizer updates. Here, $w_{i}^{t}$ refers to the weight being assigned to the model difference from client $i$ in the aggregation, i.e. number of words in the training data for adapting the client NNLM in round $t$.

We use the FedAdam optimizer for updating the global model [31]. Specifically, let $\eta_{g}$ be the learning rate and hyper-parameters $\beta_{1}$, $\beta_{2}\in[0,1)$, then

where $m_{0}=0,v_{0}=0$, and $\epsilon_{g}$ is a small positive number.

A differentially private mechanism enables the public release of model parameters with a strong privacy protection [29, 30].

Here $d$ and $d^{\prime}$ are defined to be adjacent if $d^{\prime}$ can be formed by adding or removing a single training example from $d$. It is worth noting that the definition of adjacent datasets in Definition 2.1 depends on the application. Most prior work on DP deals with example level (or utterance level in our case). For our task, a better definition is user-level adjacency for protecting whole user histories in the training set [32], since a sensitive word may be uttered several times by an individual user. Note that given some target $\delta$, a smaller $\epsilon$ leads to stronger privacy protection, but often, can degrade the model accuracy.

Two modifications to the FL-based NNLM adaptation are needed to ensure an differentially private algorithm. First, clip the gradient computed on any client per each round to bound a user’s impact on model parameters. Second, add randomly sampled Gaussian noise to the clipped gradient.

In subsection 3.4, we perform privacy analysis of the proposed NNLM adaptation approach.

In our experiments, the ASR model is trained using in-house video dataset (14K hours), which is sampled from public social media videos and de-identified before transcription; both transcribers and researchers do not have access to any user-identifiable information (UII).

The following two ASR applications are considered. The first is conversation speech and the second is short voice command for smart devices. For both use cases, in-domain speech datasets are collected using mobile devices through crowd-sourcing from a data supplier for ASR; the data is properly anonymized and no UII is contained in the datasets.

Table 1 shows a summary of the two datasets which contain the adaptation split for NNLMs adaptation, and the test split for model evaluation purpose. It is worth mentioning that the word error rates (WERs) on the adaptation split using the video ASR model with a pre-trained NNLM for second-pass rescoring are 9.7% and 12.2% for conversation and voice command applications, respectively.

We would like to simulate the real-world scenarios after deploying the ASR and pre-trained NNLM models to clients. Voice data from the adaptation split is streamed to each device and decoded by the on-device speech models. Then the transcripts are used to fine-tune the NNLMs for domain adaptation.

For the ASR model, we use connectionist temporal classification (CTC) [8] criterion to learn an acoustic model and it is further composed with a 5-gram LM in a standard weighted finite-state transducers framework. Here, we adopt a 6-layer latency-control bi-directional LSTM encoder with hidden dimension 1,000. The NNLM is used for second-pass 5-best rescoring, where we utilize a LSTM based model with character embeddings [33] dimension 100, and 2 layers of 512 hidden units. For each hypothesis among the 5-best list, its NNLM score is linearly interpolated with the score from the 5-gram LM. The interpolation weight is chosen to give the lowest WER. The confidence classifier model is trained on video dataset using feed-forward networks and handcrafted input features from decoding results; isotonic regression is used for calibrating the model.

To simulate the environment of FL-based approaches, for each utterance, we generate a random device label from a Zipf distribution. Thus utterances with a common device label are considered being received by the same device. This results in approximately 8K devices for each application. Figure 1 shows the histogram of number of training examples (i.e. utterances) on each device in the simulation, where we can see the empirical distribution is highly skewed.

Regarding the hyper-parameters of FL training, we set the number of selected users per round $|\mathcal{I}_{t}|=100$; learning rate $\eta_{g}=0.001$ in the global FedAdam optimizer and $\eta_{l}=1.0$ for the client SGD optimizer. Locally, we only train 1 epoch with batch size 8 for any selected client per each FL round. We use 10 epochs for FL training, where each epoch corresponds to 100 rounds.

The baseline model in our experiments is the one where we use the on-server pre-trained NNLM for rescoring, without domain adaptation. We compare it with the fine-tuned NNLM using in-domain unsupervised text from the adaptation split (transcribed by the ASR model) in the FL frameworks. Note that such in-domain data never leaves physical devices and is thus not accessible from servers due to privacy restrictions. Multiple methods in handling potential transcription errors in the adaptation data are measured, including using all transcripts, hard thresholding, utterance-level and token-level weighting. For comparison purposes, we also include the result without NNLM rescoring.

Table 2 and Table 3 show the perplexity (PPL) and WER results on the conversation and voice command evaluation datasets. Compared with the baseline model, FL-based domain adapted NNLMs (using all transcripts) obtain relative 2.1% and 8.4% WER gains on the two use cases. In addition, models leveraging confidence scoring always outperform the one using all transcripts as the training data, and obtain up to relative 0.5% and 2.4% WER reductions on the two applications, respectively. For short voice command evaluation set, hard thresholding leads to the best result. For longer conversation utterances, token-level weighting performs the best.

We also evaluate the performance of NNLM adaptation without pre-training, that is, training from scratch using the in-domain data in the FL settings. From the results in Table 4, we can see that training from scratch has around relative 1% WER degradation comparing to fine-tuned models in the FL settings. Thus it is beneficial to start with some pre-trained NNLM before on-device adaptation.

Our privacy analysis is performed in the framework of Rényi DP [34], which is a natural relaxation of DP based on the Rényi divergence. It is well-suited for expressing guarantees of privacy-preserving approaches and for composition of heterogeneous mechanisms.

In this analysis, the $L_{2}$ norm clip is set to 0.5, and the noise multiplier, which is the ratio of the standard deviation of Gaussian noise to the $L_{2}$ sensitivity, is set to 0.2, 0.5, and 1.5 in our experiments. We set the target $\delta$ as 1e-5 and the value of $\epsilon$ is calculated accordingly.

Table 5 displays the privacy analysis results on FL-based adapted NNLMs using all or token-level weighted transcripts. It is expected that the lower the value of $\epsilon$, the larger the PPL and WER. It is worth noting that these resulting models still perform better than the server-trained NNLM without domain adaptation, although the margins of gains become smaller than the one without strong privacy guarantees.