Three-class Overlapped Speech Detection using a
Convolutional Recurrent Neural Network

Conventional OSD systems using DNNs are trained to perform binary classification because the objective at the inference phase is to decide whether overlapped speech exists for each frame. It is a widely accepted straightforward approach, training the model to perform as one wants. However, by training the OSD model as a two-class classifier, both non-speech and single speaker segments are labeled as not overlapped. We assume that grouping non-speech and single speaker speech into not overlapped, regardless of the existence of human voice, may not be ideal in two aspects. First, two small clusters may be created within not overlapped class, diminishing the OSD model’s generalization. Second, training OSD as a two-class classification task intensifies class imbalance. For instance, suppose there exists a sample in the train set that has 40 non-speech frames, 100 single speaker speech frames, and 10 overlapped speech frames. Training it with conventional two-class results in 14:1 imbalance ratio, but with three-class training, it becomes 4:10:1. Although manual weight can be set for each class, it is difficult to account for the single speaker segments.

Thus, we propose to train the OSD model as a three-class classifier where the classes are non-speech, single speaker speech, and overlapped speech. This is inspired by studies [10, 11] that use hidden Markov models where single speaker speech and non-speech are separately defined as states. By dividing not overlapped class into single speaker speech and non-speech classes, we argue that not only the problem occurred by grouping different categories may decrease, but also the DNN can leverage from additional information. It has commonalities in terms of motivation with the multitask learning [18] framework where more specific and concrete information is additionally provided to help learning. Also, precise class balancing using loss weights becomes possible. The proposed three-class scheme has no additional requirements when performing inference. One can simply use the node of the output layer that indicates overlapped speech (before or after applying softmax non-linearity) as the model’s score for the given input frame.

Most preceding studies that use DNN-based models for the OSD task either utilize a CNN or a RNN [12, 5, 13, 14]. However, we propose to use a CRNN architecture with three convolution blocks and two gated recurrent unit (GRU) layers. Three motivations led us to propose and use the CRNN architecture. First, CNNs and RNNs have different aspects that the OSD model can benefit from. CNNs are known for their capability of modeling local patterns and RNNs are widely used for their capability of modeling sequential data. We argue that by using a CRNN architecture, advantages of both CNNs and RNNs can be exploited. Second, decreasing the length of sequence via pooling layers included in CNN will facilitate the training of GRU layers. In addition, decreasing the sequence length will give a smoothing effect on the model’s output. For example, according to our configuration when 150 frames are input, the CRNN model will output 25 frames where each output frame represents six input frames. Third, a CRNN architecture shows state-of-the-art performance in sound event detection [16] which is also a sequence labeling task that predicts onsets and offsets of various predefined sound events.

Table 1 denotes the proposed CRNN architecture used for OSD. Three convolution blocks are used where the length of a sequence is divided by 6, followed by an average on the mel bin dimension. A squeeze-excitation [19] is applied to each block. Details on how the length of the sequence and the number of mel bins are decreased are addressed in Section 4.2. The output of CNN is input to two bidirectional GRU layers that output sequences. Each frame is then classified among the three classes and lastly duplicated to match the length of the input sequence.

We adopt two data augmentation techniques on top of widely used noise and reverberation augmentation [20, 21]. First, we generate augmented overlap speech samples using two single speaker segments alike the method introduced in Bullock et al. [14]. It diminishes the class imbalance problem and generates sufficient overlapped speech samples; most datasets that can be used for training an OSD model have less than 10% overlapped speech regions. Second, we augment training samples by downsampling an audio clip and then upsampling back. We use this augmentation to cope with telephone conversation audio clips that have been upsampled in the test phase. Through experiments, we empirically find that this augmentation mitigates performance degradation on 8kHz sampling rate audio clips that exist in the evaluation dataset.

We use four datasets to train the proposed OSD model: AMI Meeting corpus [26], DIHARD I development set[27], DIHARD II development set [28], and VoxConverse dataset [29]. The AMI corpus includes 100 hours of meeting conversations. DIHARD I and II development sets include 19 and 23 hours of data from diverse domains respectively. The VoxConverse dataset consists of approximately 20 hours of conversation from YouTube videos. For all datasets, multi-channel audio clips were modified to single-channel and the sample rate of all audio clips was adjusted to 16kHz. The DIHARD II evaluation set and the DIHARD III development set [30] are used to report the performance of the proposed OSD model. OSD model’s performance is reported using precision and recall. Speaker diarization result is reported in DER, the summation of false alarm, miss, and confusion.

We use a 128-dimensional log mel-spectrograms (i.e., filterbank energies) as input to the DNN with a window size of 25ms and 10ms shift. Pre-emphasis is applied before applying Fourier transforms and mean normalization is applied to the extracted log mel-spectrograms.

Table 2 describes the comparison between the existing model and the proposed OSD model applying both three-class training and CRNN architecture. The OSD model proposed in Bullock et al. [14], which demonstrated state-of-the-art performance in AMI and DIHARD II evaluation set is used as the baseline. Experimental results show that our proposed OSD model establishes a new state-of-the-art performance on the DIHARD II evaluation set. As the DIHARD III challenge has been held recently, we further report the proposed model’s performance on its development set. Note that this model is similar to, but performs slightly better than the OSD model we used when participating the DIHARD III challenge [25]. Hereafter, we report all performances using the DIHARD III development set.

Table 3 addresses comparison experiments with ablation studies to show the effectiveness of training the OSD model as a three-class classifier and using a CRNN architecture. Using the best performing system in the first row as a pivot, we first remove the individual proposed approach. Then, ablations experiments on the number of GRU layers and the extent of sequence length compression are dealt. All performances are reported using a threshold that shows a precision of 0.9.

Results from the second to the fourth rows show that both three-class training and CRNN architecture successfully improves the OSD model’s performance. Thee-class training relatively improved recall by 12%. CRNN architecture was also proven effective where removing GRU layers degraded the performance more. Through the fifth and sixth rows, we conclude that the model benefits from multiple GRU layers where using one GRU layer decreased performance the most. The last three rows show that compressing the sequence length using pooling layers, before GRU, is also effective for OSD. Reducing training sequence length was effective than not reducing at all, but significantly decreasing also decreased the performance.

Table 4 reports speaker diarization performance using the proposed OSD model on the DIHARD III Track 1 development set. Performance is reported using DER, equal to the summation of false alarm, miss, and confusion. The speaker diarization system refers to that of the Naver Clova team’s submission for the DIHARD III challenge [25]. OSD model threshold was set to a value that shows a precision of 0.95.

Experimental results show that further segmentation and allocating additional speaker labels using the developed OSD model successfully lowers the DER of speaker diarization, decreasing the DER from 14.29% to 13.74% on the DIHARD III development set. Miss, which the OSD model should reduce, decreased to 9.40% showing relative improvement of 8.73%, at the cost of an increase in false alarm by 0.09 which was caused by the OSD model predicting single speaker regions as overlapped. Our system shows 32% lower DER than the challenge baseline and are comparable to the challenge winning system.