Speaker Recognition in Realistic Scenario Using Multimodal Data

Sandra et al. [15] laid the groundwork for speaker recognition systems attempting to find a similarity measure between two speech signals by using filter banks and digital spectrograms. We provided a brief overview of speaker recognition methods as clustered in two main categorizes: Traditional and neural network based methods.
Traditional methods. There have been many advancements in the speaker recognition task due to the availability of data and computing resources. However, noisy environment presents a challenging scenario. For several years, the standard speaker recognition task relied on ways that are dependant on features that required manual intervention and domain knowledge. This includes features extracted from low dimensional short term representation of the speech signals such as MEL Frequency Cepstrum Coefficients [16]. The performance of these systems degrade in real world conditions [17, 18]. These systems are dependent on Human ability to extract useful features which is a limitation for the system. Joint Factor Analysis captures both speaker-specific and session-specific variability in speech signals by decomposing the speech signal into a set of latent factors [19]. Support Vector Machine (SVM) classifier has been very successful for robust recognition tasks. However, such methods are very slow, complex and prone to degradation when applied to various real world scenarios. Despite of these advancements, the performance of traditional approaches drop in the presence of noise. Moreover, the performance degraded as the size of data increases. In real world applications there is often no knowledge of environmental noise, transmission channel used and number of speakers in the background. In such cases the traditional methods may degrade in performance.
Deep Learning Methods. Over the last few years, advances in computing resources and neural networks has led to more efficient methods. With these advancement, CNN are extensively used in tasks such as speaker recognition. For example, the work in [2, 3] propose a CNN based method to transform speech segments into spectrograms for speaker recognition task. With this advancement, speaker recognition task is moved from manually extracted features to data driven methods. Specifically, the work in [2] train a modified VGG-M on spectrogram extracted directly from speech segment.

Recently, an association between faces and voices of speakers are established by leveraging cross-modal verification and matching tasks [7, 10, 6, 9, 20, 21]. The work in [7] used a triplet network to learn joint representation for face-voice association task. Similarly, the work in [22] used a triplet network [23] to minimize the distance between faces and voices by extracting features from face subnetwork [24] and voice subnetwork [25]. Nawaz et. al [6] learns shared latent space by taking advantage from class centers with a single stream network which eliminate the need of pair or triplet samples. On similar grounds, Saeed et. al [20, 21] proposed a light-weight, plug-and-play mechanism that exploits the complementary cues from faces and voices to form enriched fused embeddings and clusters them based on their identity labels via orthogonality constraints.

In contrast to existing methods, our goal is to extract robust features from a multimodal system trained on faces and voices for speaker recognition task.

We extracted $1024$-D features of VoxCeleb$1$ dataset with VGGVox subnetwork to establish a baseline. SVM classifier is trained on these features for speaker recognition task. Decision function shape is set to one vs one for multi class classification using SVM, kernel parameter was set to poly while degree of polynomial kernel function was set to 3. After training SVM on the features extracted using VGGVox, accuracy of identification is 91%.

Our proposed method consists of training a multimodal system using a two-branch network with face and voice information. Afterwards, the multimodal is used to extract features to train a classifier for speaker recognition task. Face and audio features are extracted from VGGFace [26] and VGGVox [2] subnetworks respectively. Afterwards, face and voice features are input to two independent branches, with each modality specific branch respectively. Features from both subnetworks are fused after passing from fully connected and normalization layers. Fig.2 shows the proposed framework.

We extracted features from face and voice information. These features are then fused and passed to fully connected layer to learn joint representations from both face and voices signals. After fusion a softmax layer is used to learn for the output classes. Softmax function is used as the activation function to predict a multinomial probability distribution where probability is required for multi class classification problems. Features extracted from this two branch network are then used to train a classifier.

We want the fused feature to capture the semantics of the speaker or identity. In other words, these features should be able to predict the identity labels with good accuracy. It is possible if the samples belonging to the same class are placed nearby whereas the ones with different classes are far away. A popular choice to achieve this is softmax cross entropy (CE) loss, which also allows stable and efficient training. Now, the loss with fused embeddings is computed as

Categorical cross entropy is a very good measure of how distinguishable two discrete probability distributions are from each other [27]. Adam was used as optimizer with learning rate ranging from $0.01$ to $0.13$. Network was trained using batch size of $512$, $1024$, $2048$ and $4096$. Maximum results were achieved with $0.04$ as learning rate and $2048$ as batch size.

We extracted the feature from VGGVox subsnetwork and trained a classifier to establish a baseline, resulting in 91% identification performance. Fig. 3(a) shows confusion matrix of the baseline results. Moreover, stochastic neighbor embedding for a sample test set in Fig. 4(a) shows that the network has distributed the features for multiple classes far apart which has reduced the accuracy of classifier on those features.

Table II shows results of a classifer trained on features extracted from the two-branch network. Confusion matrix of two branch fused features can be seen in Fig.3(b). Moreover, T-SNE plot for a sample test set in Fig.4(b) shows that the network has distributed the features more efficiently where same class features are close to each other that has resulted in better learning of SVM for speaker recognition task.

Experiments show that when speech signals are aided by faces during feature extraction, speaker recognition is improved significantly. Without facial information, the system is likely to be effected with noise. When we have face information aiding the voice information degraded in one mode can be recovered by the other.