Joint Speech Activity and Overlap Detection with Multi-Exit Architecture

Our system takes raw speech waveform as the input. With motivation to build an adaptive front-end, the extractor module is constructed by SincNet [5], four 2D-convolution layers, two average pooling layers and squeeze-and-excitation (SE) layers [23]. Here, instead of using traditional handcrafted speech features, we use the SincNet features with cut-off frequencies learned from the raw waveform. The feature dimension is further reduced through the subsequent layers by exploring temporal pattern among features.

The feature outputs from extractor are further mapped by three cascaded Conv2d modules. Each module includes two convolution layers with different configuration of filter number and kernel size. Note that the output dimension of each Conv2d module is consist to that of the input, to ease attachment of multiple exits with shared LSTM parameters.

Inspired by the architecture of CRNN network [24], LSTM sub-module is placed right after those CNN modules to aggregate feature sequence. Each sub-module has one average pooling layer and one Bi-LSTM layer. It is important to note that, to reduce the capacity of the model, these LSTM sub-modules share their parameters across exits.

Each MLP sub-module includes two fully connected layers. The former is employed for dimension reduction while the latter noted as classifier for classification. For objective of joint VAD and OSD, this module predicts the probability of occurrence of three classes, labeled as {0: non-speech, 1: single speaker speech, 2: overlapped speech}.

Last of all, we point out that above proposed system architecture is general and can be easily extended to handle other multi-class prediction problems (say, joint activity detection and concurrent speaker counting).

The motivation behind the scheme is to introduce a possibility for different exits to learn from each other. As widely known, KD is a popular technique for knowledge transfer from a pre-trained large-sized network (teacher) to a small-sized one (student). For case of training MEA, we assume that an ensemble representation, as a summary statistic of individual representations from multiple exits, can play a role of teacher. It triggers us to adopt a KD-based method as in [25], which transfers knowledge by encouraging every exit to mimic the probabilistic outputs of the teacher exit.

Guided by the idea, we introduce two kinds of KD-losses with respective teacher-student options (as illustrated in Fig.3). In particular, for student option, we consider representations before or after each exit classifier, denoted by $F_{i}$ (latent features) and $Z_{i}$ (un-normalized probability outputs) for the $i$th-exit respectively. As for teacher $F$ and $Z$, we adopt the ensemble information from all exits, which can be simplified as $Z=\frac{{1}}{{M}}\sum^{M}_{i=1}Z_{i}$ and $F=\frac{{1}}{{M}}\sum^{M}_{i=1}F_{i}$.

With the KD-based loss, the overall training loss $L_{joint}$ can be expressed as a joint loss. It combines an aggregated conventional classification loss $L_{C}$ with a probability-based distillation loss $L_{Z}$ and a feature-based distillation loss $L_{F}$, weighted by coefficients $\alpha$ and $\beta$. That is,

Note that the additional two KD-loss items do not require access to the actual ground-truth $y$.

Alternatively, another training strategy is to enforce the network to explicitly control the flow of information via feature fusion. In particular we adopt the DC scheme [21], which uses dense inter-connections to concatenate previous features from early backbone stages to features at later stages. Note that different from the skip connection scheme, DC scheme ensures feature reusability, where output feature of one layer is concatenated to the input to the next layers rather than a summation.

To accommodate this training scheme, the original Conv2d module introduced earlier is accordingly modified to a dense block (as illustrated in Fig.4). The yielded new module is called as Conv2d-DC.

Recall that our main objective is twofold: to achieve overall decent classification performance and to verify the possibility of dynamic inference with computation-accuracy tradeoffs. To this end, we propose two inference modes to access each objective individually.

The first inference mode, called normal mode, is to treat our proposed system as a conventional classification system, by only examine system outputs at the final exit. The other mode is to evaluate all exit outputs under a given threshold $\gamma$. That is, for a test sample, its prediction probabilities at all exits are collected in order. If any of them is higher than the given threshold, the corresponding class label is assigned to the sample, otherwise, the class label at the final exit is used instead. This inference strategy is termed exiting mode.

Three benchmark datasets are utilized for our system training: AMI Meeting [26], DIHARD III [16] and VoxConverse dataset [27]. Apart from the officially designated training dataset, we also generate synthesized data by weighted downmixing of two random speech samples, which offers additional 40% training data. In addition, following a common practice, speech samples are mixed on-the-fly during the training process, which augments each training sample with random noise and reverberation component based on MUSAN and RIRS_NOISES [28, 29] respectively. For system evaluation, both test sets in the DIHARD III and AMI are used to examine our system on both performance and generality. In such a setting, a summary of dataset statistics is reported in Table 1, where training data comprises both official data and our synthesized data. In addition, for the test datasets, the class ratios of both VAD and OSD are provided. Note that the imbalance ratio (IR) between VAD-class and OSD-class is severe, up to 1:11.3, on the DIHARD III test set.

Our model takes 1.5s monophonic speech chunks (with sampling rate of 16kHz) as input, and outputs 50 class labels per speech chunk. This corresponds to generate classification results every 30ms.

Full implementation details of our system can be found in Table 2, along with the count (M) of parameters of core modules. The total number of parameters of the model is up to 1.5M when the DC-based scheme is applied (otherwise, it’s around 1.3M).

Our system is optimized using the training loss $L_{joint}$, defined in (2). Here we adopt the proportionally-weighted cross entropy (CE) loss for $L_{c}$ and the Kullback-Leibler (KL) divergence loss for $L_{z}$ and $L_{F}$, respectively. In addition, weights are set by $\alpha=0.5$ and $\beta=1.0$. With a batch size of 256, our system is trained for 50 epochs using Adam optimizer. The initial learning rate is 0.001 and scales with a factor of 0.6 when there is no loss decrease over 6 epochs.

Following our training configuration, our system yields up to 5 prediction labels per frame(30ms). Such multiple label options are further post-processed by applying the majority rule voting strategy. Then the one with a majority is chosen as the final predicted label.

As a follow up to label predictions, VAD and OSD segmentation can be simply inferred, where the OSD segments are formed by concatenating the frames with class {2: overlapped}; and VAD segments are obtained by combining the frames of both classes {1: single speaker, 2: overlapped speech}.

Inherited from prior studies, two sets of standard evaluation metrics are adopted in this study. One set of them includes the false alarm rate (FA) missed alarm rate (Miss) and detection error rate (ER); the other contains precision, recall and $F_{1}$-score ($F_{1}$).

To better evaluate the ability of our system for joint VAD and OSD, we expand our system evaluation by benchmarking it against several top performing prior arts. For a fair comparison, those competitive prior studies are published recently for OSD on the either AMI or DIHARD III dataset, and results reported in their original publications are quoted.

All detailed comparison results are listed in Table 3 and Table 4 for VAD and OSD, respectively, where the best performance is bold-faced.

Regarding performance on the VAD task, Table 3 shows that in terms of error rate, our model performs best on both datasets. It considerately outperforms the previous SOTA, pyannote 2.0, with relative improvement of 7.3% and 4.1% on the dataset AMI and DIHARD III, respectively. Besides, the table also reveals that: 1) the VAD system [30] has high false alarm rate; 2) pyannote 2.0 [30] outperforms its early version [31]; 3) our system shows the lowest false alarm rates and the best overall error rates. Also, it is worth noting that pyannote 2.0 [30] assigns specific hyper-parameters for different dataset, which is not realistic in practical scenario.

A similar behavior is observed for performances on the OSD task. As shown in Table 4, our proposed system achieves more prominent performance gains over the existing systems. In particular, regarding $F_{1}$ score, our system surpasses pyannote 2.0 by 5.2% relative on the AMI and 4.3% on the DIHARD III. Notably, this advantage is more pronounced when measure with error rate, with relative 11.7% improvement on the AMI and 6.7% on the DIHARD III, respectively. In addition, on the more challenging DIHARD III test set, our system is considerably advantageous by outperforming prior studies over all metrics.

To ours best knowledge, the results of pyannote 2.0 are the best in the existing studies. By outperforming the pyannote 2.0, our system provides the state-of-the-art performance. In addition, it is worth noting that our system achieves such performance with even smaller model size (around 1.4 million parameters, comparing to 1.5 million used in pyannote 2.0). Lastly, again, the pyannote 2.0 requires to determine hyper-parameters by cross-validation on a validation set, which limits its generalization. In contrast, our system use common parameters for all datasets and avoid the unrealistic dataset-specific tuning as much as possible.

To examine individual impact of each proposed scheme, we conduct ablation studies on our system by disabling one scheme at a time. The according experimental results (in terms of $F_{1}$-scores) are reported in Table 5.

From the table, we make the following key observations:

• •

  high VAD performance is maintained in all ablation experiments, which suggests our system has a very strong baseline;

• •

  disabling the DC-based scheme leads to 0.5% absolute OSD performance drop on the DIHARD III test set;

• •

  further disabling the KD-based scheme results in another 0.4% absolute OSD performance drop on the DIHARD III test set;

• •

  lastly, our baseline system by removing the MEA gets 1.8% OSD performance degradation on the AMI test set, which indicates that MEA is the most important factor for our performance gains.

In all, by breaking down the performance gains on OSD task, it is proven that all our proposed schemes are beneficial and offer complementary contributions to the system performance. Additionally, among three schemes, our proposed architecture is dominant that boosts system performance the most.