Audio-visual Speaker Recognition with a Cross-modal Discriminative Network

Speaker recognition has enabled many real-world applications [1, 2, 3, 4]. These systems are expected to perform effectively under adverse conditions. The NIST speaker recognition evaluation (SRE)s are organized to benchmark systems in different such scenarios [5]. Various robust systems are developed in the past that perform effectively and provides state-of-the-art [6, 7]. Unlike previous SREs, the 2019 NIST SRE investigated a new direction on audio-visual (AV) SRE [8]. The evaluation task deals with verifying the claimed identity of a person for a given pair of enrollment and test videos. In other words, it advocates the use of audio-visual cues for improved speaker recognition in real-world scenarios.

The significance of processing multimedia in other fields has increased in the recent years [9]. The latest audio-visual SRE can be viewed as one such outcome following this trend. Some of the other tasks considering multimedia instead of single modality using speech are automatic speech recognition (ASR) [10], speech separation [11] and speech diarization [12]. The studies in these works exploited the association of audio and visual cues adequately. For instance, in audio-visual ASR, lip language recognition is used to support ASR systems; in audio-visual speech separation, the movement of mouth can assist detecting who is speaking when.

While coming to audio-visual SRE, the simplest way to perform multimedia based speaker recognition is to have separate systems for audio and visual inputs, then combine the results of speaker and face recognition systems [8, 13]. We note separating audio-visual SRE into two sub-tasks is a straight forward approach to simplify the problem. However, the two subsystems are disjoint and one does not consider the knowledge from other. It is further worth emphasizing that the motivation to process multimedia for SRE is not only to add another visual system but also to explore the relationship between the audio and video. Therefore, disregarding the association between different modalities may result in the loss of some information.

The studies in neuroscience show that humans associate the voice and the face of a person in the memory [14]. While listening to a voice of an individual, one can select the right static face corresponding to the same person between two static faces at a higher than chance level and vice versa [15, 16, 17, 18]. In computer science, there has been study on cross-modal biometric matching [19]. Further, various works use pair-wise loss-based methods to improve the cross-modal system performance [20, 21]. The learned associations between audio and visual cues are general identity features (such as gender, age and ethnicity) and appearance features (such as big nose, chubby and double chin) [22, 23]. The existing works utilize both joint and disjoint general information between two modalities to train the cross-modal verification network to determine if the given face and speech segment belongs to the same identity for verification tasks [24, 25, 26]. We believe that the general cross-modal discriminative features provide additional information in audio-visual speaker recognition.

In this work, we propose a cross-modal discriminative network, that is called voice-face network (VFNet), to learn the association between voice and face. VFNet is trained using speaker and face embeddings collectively that are extracted from separate systems. We consider x-vector and InsightFace based systems for extracting the speaker and face embeddings, respectively [27, 28]. Further, these two systems are used for SRE using audio as well as visual input based single systems, followed by their fusion for a baseline audio-visual system. The output of VFNet is used to represent the general association between voice and face. The speaker recognition studies are conducted on 2019 NIST SRE corpus. The contributions of this paper include the novel idea of cross-modal discriminative network, and its use in audio-visual speaker recognition study.

The rest of the paper is organized as follows. Section 2 describes the proposed VFNet based cross-modal verification system. In Section 3, we present the audio-visual speaker recognition with cross-modal verification. Section 4 and Section 5 reports the experiments and their results, respectively. Finally, Section 6 concludes the work.

This section describes the proposed VFNet for cross-modal verification. Figure 1 shows the architecture of VFNet that considers two inputs: a voice waveform and a human face. The output of the network is a confidence score to describe whether the voice and the face come from the same person. We now discuss the detailed pipeline of VFNet.

First, the speaker and the face embeddings are extracted by x-vector and InsightFace models, respectively [27, 28]. As both these embeddings represent information from different modality, they are fed to a 256-D fully connected layer (FC1) with the rectified linear unit (ReLU) activation, then followed by another 128-D fully connected layer (FC2) without the ReLU. These layers are introduced to lead the speaker and face embeddings for learning the cross-modal identity information from each other. Further, they help to project the embeddings from both modalities into a new domain, where their relation can be established.

For a given pair of speaker embedding $e_{v}$ and a face embedding $e_{f}$, their transformed embeddings $T_{v}(e_{v})$ and $T_{f}(e_{f})$ are derived from VFNet, followed by the cosine similarity scoring $S(T_{v}(e_{v}),T_{f}(e_{f}))$ between them. We also have $1-S(T_{v}(e_{v}),T_{f}(e_{f}))$ to represent the negative voice-face pair. By using softmax function based on these two scores, the output of VFNet is obtained as

$$p_{1}=\frac{e^{S(T_{v}(e_{v}),T_{f}(e_{f}))}}{e^{S(T_{v}(e_{v}),T_{f}(e_{f}))}+e^{1-S(T_{v}(e_{v}),T_{f}(e_{f}))}}$$

(1)

$$p_{2}=\frac{e^{1-S(T_{v}(e_{v}),T_{f}(e_{f}))}}{e^{S(T_{v}(e_{v}),T_{f}(e_{f}))}+e^{1-S(T_{v}(e_{v}),T_{f}(e_{f}))}}$$

(2)

where final output $p_{1}$ is the score to describe the probability that the voice and the face belong to the same person, $p_{2}$ being the score depicting the probability that the voice and the face do not belong to the same person. Finally, we feed our predictions $p$ and the ground truth verification labels $\hat{p}$ to optimize the cross-entropy loss $\mathcal{L}_{\hat{p}}(p)$ as follows

$$\mathcal{L}_{\hat{p}}(p)=-\sum_{i}{\hat{p_{i}}\log(p_{i})}$$

(3)

In audio-visual SRE, an enrollment video provides the target individual’s biometric information (voice and face) and the assignment asks the model to automatically determine whether the target person is present in a given test video [8].

Figure 2 shows the proposed audio-visual speaker recognition framework with VFNet on the left panel, and the baseline, a voice-face score level fusion system, on the right panel. The given voice segments of the target speakers from the enrollment utterances and the entire test utterances are considered for extracting the speaker embeddings using x-vector system [27]. Similarly, the InsightFace system extracts the face embeddings for given faces of the target speakers from the enrollment videos and all detected faces from the test videos [28].

On the left panel, the VFNet system provides an association score between the target speaker voice in the enrollment and the detected faces from the test video. Matching pairs between voice and face will give rise to high association, while mismatches, such as age, gender, and ethnicity discrepancy, will do otherwise.

On the right panel, the audio and visual systems run in parallel to verify the claimed identity by computing match between the enrollment and the test embeddings. We note that the speaker recognition system considers probabilistic linear discriminant (PLDA) based likelihood scores, whereas the face recognition system computes cosine similarity scores. We consider the baseline system as a score level fusion between the two parallel systems.

We propose to fuse the VFNet score and the baseline audio-visual system as shown in Figure 2 for a final decision. The score level fusion is performed using logistic regression for various systems discussed in this work. We report performance of the VFNet, baseline systems, and the overall system separately in the experiments.

In this section, the details of the audio and visual systems developed in this work are mentioned. The database, embedding extraction and audio-visual speaker recognition systems are described in the following subsections.

In this work, we propose a novel framework for audio-visual speaker recognition with cross-modal discrimination network. The VFNet based cross-modal discrimination network finds the relation between a given pair of human voice and face to generate a confidence score if they correspond to the same person. While VFNet can perform comparable to the existing state-of-the-art cross-modal verification systems, the proposed framework of audio-visual speaker recognition with VFNet outperforms the baseline audio-visual system. This highlights the importance of cross-modal verification, in other words, the relation between audio and visual cues for audio-visual speaker recognition.

This research work is partially supported by Programmatic Grant No. A1687b0033 from the Singapore Government’s Research, Innovation and Enterprise 2020 plan (Advanced Manufacturing and Engineering domain), and in part by Human-Robot Interaction Phase 1 (Grant No. 192 25 00054) from the National Research Foundation, Prime Minister’s Office, Singapore under the National Robotics Programme.