Adversarial Attacks and Defenses in Speaker Recognition Systems: A Survey¹¹footnotemark: 1

Fig. 1 shows an overview of a typical SRS. The SRS includes three modules, i.e., preprocessing module, feature extraction module, and model inference module. Meanwhile, the lifecycle of an SRS involves three stages, i.e., training stage, enrollment stage, and recognition stage.

When audio is input, the preprocessing module first filters out background noise and high-frequency signals beyond the frequency range of human voices. Feature extraction algorithms are then used to generate a feature vector that reduces dimensions of the audio by capturing its most important features and characteristics. Various feature extraction algorithms have been proposed, such as Mel-Frequency Cepstral Coefficients (MFCC) [19], Spectral Subband Centroid (SSC) [20], and Perceptual Linear Predictive (PLP) [21]. Among them, MFCC is the most popular one in SRSs due to its ability to expose important acoustic features, similar to human ears. After that, the feature vector is passed to a model for either training or inferencing.

In the training stage, corpora are used to train the SRS and adjust system parameters to obtain a capable SRS. After that, multiple speakers enroll in the SRS. All enrolled speakers form a speaker group. The SRS calculates and stores a feature vector for every enrolled speaker, which is used in the recognition stage. In the recognition stage, the SRS is responsible for recognizing the identity of unknown input audio.

According to the difference of recognition tasks, SRSs can be divided into two categories: Speaker Verification and Speaker Identification. For an arbitrary input audio $x$, a Speaker Identification System (SIS) determines which enrolled speaker utters $x$. Regarding a Speaker Verification System (SVS), an unknown speaker not only needs to input his/her audio $y$ but also needs to claim his/her identity. The SVS determines whether $y$ is uttered by the claimed speaker, and the verification result is “yes” or “no”.

Adversarial attack is a kind of attack method that an adversary generates adversarial examples to make a machine learning model misbehave. An adversarial example refers to specifically crafted input designed to look normal to humans but causes misbehaviors of a machine learning model [8]. Given an input $x$ with its corresponding label $y$, and a well-trained machine learning model $f(\cdot)$, an adversarial example $x^{\prime}$ can be constructed as:

Here, $\delta$ is called adversarial perturbation, which is the noise that is added to a clean sample to make it an adversarial example [8]. $\theta$ is the parameter of the machine learning model $f(\cdot)$. The hyperparameter $\epsilon$ is used to control the maximum perturbation generated. Suppose $L(\cdot)$ is the loss function and $y^{\prime}$ is the target label, the adversarial perturbation $\delta$ can be calculated by

To solve the above formula, many attack methods have been proposed. We first introduce a classic and well-known method, FGSM [9]. The adversarial perturbation can be calculated by the following formula.

Here, $\eta$ is the magnitude of the perturbation. The adversarial example $x^{\prime}$ is calculated as: $x^{\prime}=x+\delta$. Kurakin et al. [10] proposed BIM which iterates FGSM for multiple rounds. The adversarial example is generated in multiple iterations.

Here, $Clip(\cdot)$ is a function which limits the change of the generated adversarial example in each iterations.

In this subsection, we introduce threat models of attack methods and defense methods, respectively.

In this subsection, we put forward evaluation criteria for attack methods from three aspects: practicability, imperceptibility, and effectiveness.

From the perspective of practicability and effectiveness, we propose a set of criteria to evaluate the performance of defense methods in SRSs.

Adversarial attacks against SRSs can be categorized as optimization-based attacks and signal processing-based attacks. The optimization-based attacks generate adversarial examples by solving an optimization problem that is obtained by formalizing the purpose of adversarial attacks. In recent years, several optimization-based adversarial example generation algorithms have been proposed, such as FGSM [9] and BIM [10]. The signal processing-based attacks use signal processing techniques to generate adversarial examples. Although these attacks do not directly target machine learning models embedded in SRSs, they can still force machine learning models to misbehave.

Inspired by the success of adversarial attacks in computer vision and speech recognition, Kreuk et al. [15] tried to fool a DNN-based SVS by adversarial examples for the first time. Adversarial examples were generated by FGSM, which is $l_{\infty}$ perturbation norm. They verified the effectiveness of adversarial attacks in SVSs by a white-box attack. Then they deployed two black-box attacks to explore the transferability of adversarial examples they generated. Experimental results show that the adversarial examples are transferable in both cross-feature and cross-dataset settings. Finally, through human perception experiments, they found that human listeners cannot distinguish between clean samples and adversarial examples, which reflected the imperceptibility of adversarial perturbations generated by FGSM. However, since the purpose of this work was to verify that SVSs are susceptible to adversarial examples, the authors did not make efforts to pursue a better attack effect and did not attack commercial SRSs. In addition, they ignored generation time and over-air attacks, which are more practical than over-line attacks.

Li et al. [24] found that [15] has two limitations. On the one hand, [15] studied the impact of adversarial attacks on DNN-based SVSs, but ignored GMM-based SVSs. On the other hand, cross-model transferability was not discussed. To overcome the two limitations, they deployed a white-box attack to verify that GMM-based SVSs are also subject to adversarial examples. They then implemented three black-box experiments to study the transferability of adversarial examples generated by FGSM. The results show that adversarial examples are transferable in cross-feature, cross-dataset, and cross-model settings. Finally, they demonstrated the imperceptibility of adversarial examples they generated to human listeners by human perception experiments. Although [24] solved two limitations of [15], the attack performance of [24] was not good enough since it generated adversarial examples by simple FGSM. In addition, they did not discuss generation time and over-air attacks.

The aforementioned work cannot achieve real-time attacks since they generated different adversarial perturbations for different audio, which usually costs a lot of time. Real-time attacks are inseparable from universal adversarial perturbations since universal adversarial perturbations can be added to any samples to generate effective adversarial examples. If an adversary obtains universal adversarial perturbations in advance, it only needs one addition operation to generate an adversarial example, which is time-efficient. Li et al. [25] tried to generate universal adversarial perturbations for SincNet [26], a state-of-the-art speaker identification model, with a generative network, which can learn the mapping from a low-dimensional normal distribution to a universal adversarial perturbation subspace. They conducted both untargeted attacks and target attacks against SincNet [26] in grey-box settings. The results show the existence of universal adversarial perturbations. In addition, they studied the imperceptibility of universal adversarial perturbations by SNR. However, they did not deploy over-air attack experiments and did not try to attack commercial SRSs.

Xie et al. [27] almost simultaneously conducted similar work. While Li et al. [25] generated universal perturbations with a generative network, Xie et al. [27] generated universal perturbations with a conventional optimization-based approach. They attacked an SIS in white-box settings successfully. After that, they tried to attack the SIS in the real world, i.e., over-air attacks. However, they failed because of the attenuation and multi-path effects of sound in the propagation process. They then assumed that the adversary has the knowledge of the room’s layout and took Room Impulse Response (RIR) into consideration to enhance the robustness of adversarial examples. The results show that considering RIR can greatly increase the ASR in over-air attacks. However, the high attack success rate came at the cost of low SNR. That means they need to make a trade-off between the attack effect and the imperceptibility of the adversarial perturbations.

Although [25, 27] generated universal adversarial perturbations successfully, Li et al. [28] found that they only focused on static-speech attack scenarios, but ignored streaming-speech attack scenarios. In static-speech attack scenarios, an adversary should obtain universal adversarial perturbations and clean samples to generate adversarial examples before feeding adversarial examples into an SRS. On the contrary, in streaming-speech scenarios, an SRS takes streaming audio inputs (e.g., live human speech) and an adversary can fool the SRS by playing universal adversarial perturbations through a nearby loudspeaker. There is no doubt that streaming-speech scenarios are closer to the real world than static-speech scenarios. Therefore, Li et al. [28] designed AdvPulse, a method to generate a subsecond-level adversarial perturbation that can be added at any point of a streaming audio input to launch targeted adversarial attacks. In other words, they generated universal adversarial perturbations against SRSs in streaming-speech attack scenarios. In addition, when a loudspeaker and an SRS are less than three meters apart, adversarial examples they generated can successfully deceive the SRS. This work is advanced. Unfortunately, they only considered white-box settings but ignored black-box settings.

Chen et al. [29] proposed an attack method named Fakebob, which formalizes the generation of adversarial examples into an optimization problem. In this optimization problem, a score threshold and the strength of adversarial perturbations are considered. To solve the optimization problem, they proposed an approach that utilizes a novel algorithm to estimate the threshold, a natural evolution strategy (NES) to estimate gradient, and finally the BIM method is applied to generate adversarial examples. This work is currently the most comprehensive one. It has four main contributions. Firstly, they effectively implemented targeted attacks on SRSs in black-box settings and the ASR can reach 99%. Secondly, they considered all possible SRSs based on different tasks, including SVSs and SISs. Thirdly, they did many experiments, including over-line and over-air experiments, on both open source systems and commercial systems, which prove the transferability and practicability of Fakebob. Fourthly, they tried four defense methods to defend against Fakebob. Experimental results show that Fakebob still affects the performance of victim systems, which demonstrates the robustness of Fakebob. Meanwhile, they employed human perception experiments to explore the imperceptibility of adversarial perturbations. However, FakeBob took several minutes to generate an adversarial example, which limits its wide use in the real world.

Abdullah et al. [30] noticed that different audio samples may have the same feature vector when being transformed by acoustic feature extraction algorithms (e.g., MFCC) and nearly all SRSs appear to rely on several feature extraction algorithms. Based on this knowledge, they successfully attacked a commercial SVS and a commercial SIS in black-box settings. They first obtained desired audio (e.g., OK, Google uttered by Alice). Then they designed four methods to obfuscate the desired audio as much as possible to generate obfuscated audio, i.e., adversarial examples. The four methods include Time Domain Inversion (TDI), Random Phase Generation (RPG), High Frequency Addition (HFA), and Time Scaling (TS). Although this work achieved efficient attacks since an adversarial example could be generated in a few seconds, adversarial examples it generated were noise in human perception which were easy to be noticed by human listeners.

Abdullah et al. [31] designed an attack method to circumvent surveillance in telephone networks. They assumed that SISs rely on the components of audio that are non-essential for human comprehension. In order to find out the non-essential components, they first split audio into phonemes. Then, signal decomposition algorithms were used to decompose the phoneme into individual components and corresponding intensities. Since the low-intensity components are less perceptible to humans, the low-intensity components are likely what is looked for. Therefore, adversarial examples are generated by filtering out low-intensity components of every phoneme. They employed an untargeted attack in black-box settings to prove the effectiveness of the attack method they proposed. The results show that when only one phoneme in an utterance is perturbed, the ASR of the utterance can reach 10%-20% for most phonemes. An adversary can perturb multiple phonemes in an utterance to achieve a high ASR. They further demonstrated that the attack method is transferable in a cross-model setting. Although they achieved untargeted attacks and over-telephone-network attacks, most adversaries cannot obtain the right to monitor the telephone network. Targeted attacks and over-air attacks against intelligent voice assistants embedded in smart devices are the mainstream of adversarial attacks against SRSs now and even in the future.

While [31] successfully achieved untargeted attacks by removing low-intensity components of clean samples, Wang et al. [32] tried to achieve target attacks by adding additional components to clean samples. They also aimed to generate inaudible adversarial perturbations, instead of maintaining a slight noise to the clean sample. Inspired by previous work [33, 34] on speech recognition systems, they first obtained the desired audio (e.g., OK, Google) as adversarial perturbations. Then, they leveraged frequency masking, which refers to the phenomenon that one faint but audible sound becomes inaudible in the presence of another louder audible sound, to hide adversarial perturbations in normal audio, such as birdsong and white noise. They successfully attacked a DNN-based SIS by inaudible adversarial audio in human perception. Unfortunately, they deployed experiments in white-box settings rather than more practical black-box settings. In addition, they did not explore the transferability of adversarial examples they generated and did not explore over-air attacks.

In Section 4, we comprehensively review adversarial attack methods in SRSs and compare them in Table 1. Based on Table 1, we conclude our review as below.

Among all the works we reviewed in this section [15, 24, 25, 27, 28, 29, 30, 31, 32], the methods based on optimization [15, 24, 25, 27, 28, 29] account for two-third, while the methods based on signal processing [30, 31, 32] account for one-third. The first type of method can be transferred from computer vision directly since they generate adversarial examples from digital vectors rather than raw audio or images. For example, Goodfellow et al. [9] proposed FGSM to deceive an image classification model, while Kreuk et al. [15] and Li et al. [24] also used FGSM to fool SRSs successfully. The methods based on signal processing cannot be transferred in different domains directly since they leverage acoustic signal processing techniques to generate adversarial examples. For example, Abdullah et al. [31] generated adversarial examples by filtering out low-intensity components in raw audio. Obviously, this type of method cannot be used to deceive image processing tasks.

We observe that researchers not only focus on simple white-box attacks but also pay attention to black-box attacks. More than half of reviewed works [15, 24, 29, 30, 31] explored the effectiveness of attack methods in black-box settings. In addition, all reviewed works except [31] achieved targeted attacks, which are more difficult than untargeted attacks.

Three of all reviewed works [25, 27, 28] generated universal adversarial perturbations, which help adversaries to achieve practical real-time attacks. More than half of reviewed methods [15, 24, 29, 30, 31] proved that adversarial examples they generated are transferable. Transferable adversarial examples can not only deceive the SRS that generates them, but also deceive other SRSs. In other words, transferable adversarial examples can deceive multiple SRSs, while non-transferable adversarial examples can only deceive the SRS that generates them. Therefore, transferable adversarial examples are more practical in the real world. In addition, four of all reviewed studies [27, 28, 29, 30] considered over-air attacks and deployed experiments to explore over-air attacks. Among them, the effective attack distance of adversarial examples generated by [30] is only 0.3 meters, while the effective attack distance of adversarial examples generated by [28] can reach 3 meters. More than half of reviewed attack methods [28, 29, 30, 31, 32] can successfully attack commercial SRSs.

Adversarial examples generated by the optimization-based attack methods [15, 24, 25, 27, 28, 29] sound clean to humans. Adversarial examples generated by the three attack methods based on signal processing [30, 31, 32] are clean, noisy, and inaudible to humans, respectively. Additionally, more than half of reviewed works [15, 24, 29, 31, 32] deployed human perception experiments to explore the imperceptibility of adversarial perturbations. At the same time, four reviewed works [25, 27, 28, 29, 32] measured SNR of adversarial examples to quantitatively represent the relationship between audio and perturbations in adversarial examples.

Three of all reviewed works [25, 27, 28] achieve real-time attacks since they generate universal adversarial perturbations. The optimization-based attack method proposed in [29] takes several minutes to generate an adversarial example, while the signal processing-based attack methods proposed in [30, 31] only take several seconds. This indicates that the methods based on signal processing are more time-efficient than the optimization-based attack methods.

There are two types of defense methods against adversarial attacks: 1) proactive defenses, 2) passive defenses. Proactive defense methods employ adversarial data augmentation to retrain original models such that they can be robust to adversarial examples. Passive defense methods defend against adversarial attacks by adding new components rather than modifying original models. According to the function of new components, passive defense methods can be divided into detection methods and purification methods. When an adversarial example is identified, a detection method aims to refuse it to enter systems, while a purification method aims to feed it to systems after removing adversarial perturbations.

Wang et al. [35] proposed adversarial regularization based on adversarial examples to defend against adversarial attacks. Adversarial regularization aims to seek the worst sample around an input sample and then use the worst sample to optimize an SRS. They used adversarial examples generated by FGSM and virtual adversarial training based on local distributional smoothness (LDS) to attack a DNN-based SVS. It is worth noting that virtual adversarial training can calculate adversarial perturbations for unlabeled samples. To the best of our knowledge, this is the first work to apply virtual adversarial training into SVSs. After that, they leveraged FGSM and LDS to regularize the SVS, respectively. They showed that adversarial regularization is a medium-general defense method through several experiments. However, the performance of adversarial regularization is not good enough, since the EER of the regularized SVS only decreased slightly. In addition, this work did not consider adaptive attacks and did not mention defense time.

Many previous works have explored defense methods to resist spoofing attacks, including synthesis, convert and replay attacks, for SVSs. However, spoofing countermeasure models are still vulnerable to adversarial attacks. To address this issue, Wu et al. [36] proposed two defense methods, one is proactive, i.e., adversarial training, and the other is passive, i.e., spatial smoothing, to improve the robustness of SVS spoofing countermeasure models. We will introduce spatial smoothing in Subsection 5.3. Adversarial training refers to a defense method that uses adversarial data augmentation to retrain the model to enhance the robustness of the model. They retrained a DNN-based SVS by adversarial data augmentation which was generated by Projected Gradient Descent (PGD) method. They proved that the performance of adversarial training is better than spatial smoothing through several experiments, which is because adversarial training is model-specific. However, adversarial training is low-generality. In addition, defense time and adaptive attacks were not mentioned in the paper.

Passive defense methods utilize new components to detect or purify adversarial examples. In this subsection, we first review detection methods, followed by a review of purification methods.

In Section 5, we comprehensively review the existing works about defense methods [35, 36, 39, 40, 41, 43, 44] for adversarial attacks in SRSs. Meanwhile, we compare all the defense methods reviewed in this section in Table 2. Based on Table 2, we summarize our review as below.

Among all the studies reviewed in this section, passive defense methods [36, 39, 40, 41, 43, 44] are distinctly overwhelming with three-quarters of all reviewed papers. Passive defense methods are so popular since they can be deployed in any SRSs to defend against adversarial attacks. However, proactive defense methods [35, 36] cannot be transferred between different SRSs since they are model-specific, which makes them receive less attention than passive defense methods.

We observe that all reviewed works deployed experiments to defend against non-adaptive attacks, while only two works [40, 44] try to defend against adaptive attacks that are more threatening than non-adaptive attacks.

Half of reviewed defense methods [36, 40, 41, 44] are highly general. Highly general defense methods are favored by researchers since they can defend against any attack methods theoretically. In addition, all reviewed works considered and deployed over-line attacks. Over-line ensures lossless transmission of adversarial examples. Therefore, defense methods that can defend against over-line attacks can also defend against over-air and over-telephone-network attacks.

All reviewed works employ experiments to show the effectiveness of defense against conventional attack methods, such as FGSM, BIM, and PGD. However, they do not defend against some advanced attacks, such as AdvPluse [28] and FakeBob [29]. At last, defense time, an important evaluation criterion of practicability, is missed in discussion in all reviewed works..

By reviewing and comparing the above literature with our proposed criteria, we figure out several open issues for adversarial attacks and defenses in SRSs.

First, it is difficult to enhance the robustness of SRSs. Adversarial training, which utilizes adversarial data augmentation to retrain SRSs, is an effective way to enhance the robustness of SRSs. However, obtaining adversarial data augmentation, i.e., adversarial example and its true label pairs, is time-consuming since we need to manually label each adversarial example. Therefore, how to enhance the robustness of SRSs efficiently is still an open and tough issue.

Second, it is not convenient to directly compare the performance of attack methods or defense methods proposed in different works. This is caused by the differences in experimental settings and evaluation metrics applied in different works. For example, the defense methods proposed in [40] and [41] were used to defend against adversarial examples generated by BIM. However, we cannot directly compare the performance of them since [40] and [41] used different evaluation metrics, i.e., [40] used FPR and FAR, and [41] used EER and DA. Similarly, we cannot compare the performance of defense methods proposed in [43] and [44] since they were used to defend against adversarial examples generated by different attack methods. All in all, the differences in experimental settings and evaluation metrics among different works hinder researchers from comparing the performance of attacks methods and defense methods in a direct way. Uniform evaluation metrics or criteria should be defined and adopted.

Third, signal processing-based attacks receive little attention. On one hand, although signal processing-based attacks are more efficient than optimization-based attacks as discussed in Subsection 5.4, to the best of our knowledge, only three articles [30, 31, 32] raise signal processing-based attacks in SRSs, which are far less than optimization-based attacks. On the other hand, all defense methods are proposed to defend against adversarial examples generated by optimization-based attacks, such as FGSM and BIM, while ignoring signal processing-based attacks. We cannot judge if current defense methods can defend against signal processing-based attacks. In short, signal processing-based attacks have not been paid sufficient attention in the current literature.

Fourth, the literature still lacks research on poisoning attacks against SRSs. The poisoning attacks refer to adding malicious data into training data, resulting in a biased model. Previous work has shown that poisoning attacks can seriously threaten the security and privacy of ML models in the image field [45, 46, 47]. However, as the structure of SRSs is more complex than image processing systems, little work pays attention to poisoning attacks against SRSs.

We suggest several future research directions motivated by the above open issues as below.

Firstly, applying virtual adversarial training [48] to enhance the robustness of SRSs is worthy of deep-insight research. Virtual adversarial training can relieve the pressure of labeling adversarial examples since it retrains SRSs in semi-supervised settings. In addition, virtual adversarial training has low computational costs and a small number of hyperparameters. Therefore, virtual adversarial training in SRSs may be an interesting attempt to enhance the robustness of SRSs.

Secondly, unified adversarial attack and defense evaluation frameworks should be established. Specifically, the attack evaluation framework should clarify target SRSs, which include both open source systems and commercial systems, and attack evaluation metrics as shown in Subsection 3.1.3. The defense evaluation framework should include defendable attacks, which are used to generate adversarial examples for evaluating the performance of defense methods, and defense evaluation metrics as shown in Subsection 3.2.3. To comprehensively evaluate the performance of defense methods, both optimization-based attacks and signal processing-based attacks should be included into defendable attacks. In short, establishing unified frameworks for both attack and defense is an effective way to help researchers compare the performance of different methods of adversarial attack and defense to stimulate mutual development.

Thirdly, the research on the security of preprocessing module and feature extraction module should be strengthened. As shown in Fig.1, the SRS includes three modules, i.e., preprocessing module, feature extraction module and model inference module. The structure of SRSs is more complex than image recognition systems due to the addition of preprocessing and feature extraction. Each module introduces a surface of attacks, causing exploitable vulnerabilities from the perspective of an adversary. Researchers in the audio field proposed some signal processing-based attacks that utilize vulnerabilities of the preprocessing module or the feature extraction module, such as those mentioned in [30, 31, 32] for SRSs and in [14, 49] for speech recognition systems. However, there are still many unknown vulnerabilities. Therefore, we recommend strengthening the vulnerability mining of preprocessing module and feature extraction module. Correspondingly, defense methods should also be studied to enhance the robustness of these two modules. We believe that such an arms race will promote the security of SRSs.

Finally, we suggest studying poisoning attacks against SRSs and corresponding defense methods. On one hand, state-of-the-art SRSs require a huge amount of training data and it is common to collect these data from potentially untrustworthy sources (e.g., edge devices). Therefore, it is easy to poison the training dataset of SRSs. On the other hand, federated learning is a popular method to train ML models in a somehow privacy-preserving way, including ML models used in SRSs. However, it is difficult to guarantee that each party participating in federated learning is honest and trustworthy. A malicious party may deliberately use poisoned data for model training resulting in security and privacy threats. Therefore, it is interesting and promising to research poisoning attacks against SRSs and corresponding defense methods, especially in the context of federated learning, as well as other learning models.