Adaptive re-calibration of channel-wise features for Adversarial Audio Classification

Feature engineering is an essential component of learning algorithms, the performance of the ML models is heavily dependent on how we represent the feature vector. As a result, a significant time and effort is spent in designing preprocessing pipelines and data transformations. In the audio domain, audio files usually exist in the form of digital files with wav, .mp3, .wma, .aac, .flac etc. as the common formats. The major audio features extracted from them can either be timbral texture features or rhythmic content features. In this work our focus has been only on using timbral texture features which specifically include MFCC and LFCC. A common practice while modelling deep learning frameworks in the audio domain is to convert the audio into spectrogram which is a concise snap of an audio wave that has undergone a Fourier transform. Mel-frequency cepstral coefficients (MFCC) is a cepstral representation of the audio which has been widely used in automatic speaker recognition and vocoder systems. Introduced in the 1980’s, they have been the state of the art ever since as they have proven to be robust in training several deep learning algorithms for high level audio domain tasks. Linear-frequency cepstral coefficients (LFCC) is another alternative to MFCC which has been also used as a go to feature for training models to obtain learnable parameters, the difference between LFCC and MFCC is based on the filter banks that they use for transforming the audio, where MFCC uses a Mel filter bank, LFCC uses a linear filter bank. Several studies have shown both features to be comparable although MFCC is dominantly used in speakers as well speech recognition still.

Many times, a single feature representation obtained from the data is insufficient to convey the necessary details of the underlying distribution of the natural process, thus a common approach in many feature engineering methods is fusing together features obtained from the single data source through different methods. This combination of features can either be a simple concatenation or can be an informed mathematical combination function. In the image domain fusing features obtained from different layers is not a novel idea, image pyramids are one such example of this attempt. However, while combining such cross-layer features a common problem we run into is of scale variance. In our framework we decided not to rely on simple concatenation or summation of features but rather include scale invariant fusion mechanism that has been recently introduced in the image domain. Attentional feature fusion (AFF) [1] [Fig 3] is a trainable mechanism that has the ability to fuse together features obtained across long and short skip connections without running into issues of scale variance. The attention module MS-CAM [Fig 2] in AFF aggregates the contextual information along the channel dimension from different receptive fields which makes the fusion process more dynamic and informed.

A common theme across deep learning research has been obtaining powerful representations from the data that capture specific properties that are most useful for the task at hand. Squeeze Excitation block [2] [Fig 4] is an architectural unit that has been introduced specifically for this purpose. But instead of combining spatial correlations the SE block focuses on modeling the interdependencies across the channels of the features obtained from CNNs. As a result of such modelling and re-calibration, the more useful features are emphasized, and less useful ones get suppressed. From Fig we can see the structure of the block which can be understood in two separate operations viz., squeeze operation and excitation operation. The squeeze operation as the name suggests collects and squeezes together all the feature maps to generate a descriptor. The excitation operation then takes this descriptor embedding to generate weights for each channel. Once we obtain the weights, they are applied to the input feature map to generate the output for this block.

From extensive literature study on the problem of identifying spoofed audio and classification of deep fakes, we came to the conclusion that most of the existing approaches in this area either rely on classical ML algorithms that depend on probabilistic modeling or exploiting the temporal information of the audios or there existed a wide range of complex deep learning models with millions of parameters consisting of gating operations which were significantly bulkier. Also due to the rapid advances in the generative algorithms that can create indiscernible synthetic audios the performance of these methods needed to be tested on these novel synthesizing algorithms.

One major baseline for our work was the End2End model [3] [Fig 5] specifically the Time-domain Synthetic Speech Detection Net (TSSDNet) based on a Inception CNN. The underlying theme of this work is to show that there is no need for any hand crafted features for detection of synthetic speech; rather a model could be trained directly on raw audio inputs to classify the natural samples from synthetic ones. The Inception style TSSDNet solely relies on identifying the forensic artifacts formed due to the synthesizing process. Since such artifacts have a greater chance of being observed in the features obtained in the upper layers the model therefore the model is shallower. This work tested its framework on the ASVspoof dataset [9] and demonstrated that the TSSDNet significantly reduced the cross dataset Equal Error Rate. Another thrust in this problem was through the introduction of Res2Net architecture [5] as shown in the which utilizes a residual connection across different channel groups split from the input feature map. The underlying of this approach lies within creating multiple scaled feature fields. This work experimented with several features derived from the ASV dataset [9] but found CQT as the most informative feature set.

In this section, we describe two types of datasets. FoR dataset [6] that is used for training and testing phases and ASVSpoof2019 Logical Access (LA) dataset [9] to evaluate the effectiveness and generalization power of fake speech detection models. We later introduce the metrics to evaluate the performance.

The proposed models in our project follow the traditional architecture which is front-end feature engineering and back-end classification [Fig 5]. The front-end acoustic features are pretty important for efficiently detecting fake speech. Two simple and commonly used acoustic features 1. Linear frequency cepstral coefficients (LFCC) and 2. Mel frequency cepstral coefficients (MFCC) are evaluated in our experiments. All raw audio files are converted into multiple 1 sec samples and for each second we generate MFCC frames (20 x 44) hop-length = 512, and FFT = 2048 using librosa library and LFCC frames (136 x 13) window-hop = 0.01, FFT = 512 using spafe library.

LFCC and MFCC are largely identical in terms of coefficient extraction scheme. The only difference is in the filter bank, since the LFCC filter bank coefficients equally cover all speech frequency ranges and consider them of equal importance while MFCC considers only low-frequency ranges. So, the advantage of using LFCC is it improves the distinguishability of speaker-specific characteristics available in the higher frequency zone. How we use this advantage in training our features is described in the next section.

The acoustic features generated above are fused together and are passed through our classifiers for prediction. For fusion, we have used AFF (Attentional feature Fusion) technique [1]. To pass through the AFF, the features must be of the same size but the above obtained MFCC and LFCC samples are of different sizes. MFCC - (20 x 44) and LFCC (136 x 13). So, we padded both sample types with zeros on both ends and created a unique sample size of (136 x 44). These padded MFCC and LFCC data are passed through a single 2D Convoluted layer with a kernel of size 2x2, stride = 1 to extract features from them. After obtaining the convoluted features of both MFCC and LFCC, we fuse them by passing through the AFF block 3 that is described in the background section.