M³V: A multi-modal multi-view approach for Device-Directed Speech Detection

In multi-modal learning, we use text-audio pairs as the input of the model. Let the processed audio be $X_{a}$ and the text be represented by $X_{t}$. Given an audio-text pair of $N$ in a batch $\{X^{(i)}_{a},X^{(i)}_{t}\}$ where $i\in\{1,2,...,N\}$, the problem is a binary classification problem to classify whether it’s device-directed or non-device-directed.

From the pairs, the audio and text are passed through an audio encoder and a text encoder, respectively. We choose a Wav2vec2 as the audio encoder which is proposed in [8] and a GPT2 [7] as the text encoder which is a transformer-based language model. Let ${f_{a}}$ represent the audio encoder and ${f_{t}}$ represent the text encoder. For an audio-text pair in a batch:

By combining text representation and audio representation, we get multi-modal representation $M_{i}$.

Then we input the text representation ${T_{i}}$, audio representation ${A_{i}}$, multi-modal representations ${M_{i}}$ into their respective feed forward network (FFN), and output multi-modal view result $V_{m}$ and unimodal view results $V_{a}$, ${V_{t}}$ after softmax:

In the previous section, we introduced the implementation of M³V from a multi-modal view, but in practical applications, we found that it is limited. Specifically, due to the unavoidable errors of ASR, these models often produce incorrect predictions for unaligned input pairs. In order to solve this problem, we frame the problem as a multi-view learning task that introduces unimodal views and a text-audio alignment view in the network besides the multi-modal view. As shown in Fig 1, the left of the model is a multi-modal learning network, which can produce multi-modal view and uni-modal view information. The right side is a contrastive model, which is capable of inferring whether text and audio are consistent by exploiting text and audio information that is only available during training. For a contrastive model, we want the vectors of text-audio sources to be close to the uniform utterance. For this, we follow CLAP [9] and employ InfoNCE loss [10] to measure dependencies between the audio and text modality. Consider the audio and text view of a min-batch as $A_{i}$ and $T{i}$, then the loss of contrastive model $\mathcal{L}_{c}$ can be defined as:

After the training of the above model, in the actual inference process, we obtain four probability scores, respectively from the four views. To better adapt to specific downstream tasks, we propose three applied strategies to improve the overall accuracy of device-directed tasks. For final results, $True$ means device-directed and $False$ non-device-directed.

We use real recordings of natural human interactions with in-vehicle virtual assistant NOMI for training and testing the models. The training dataset consists of 340 hours of audio data comprised of 500k utterances. The normal test data consists of 3.6 hours of audio data with 48k utterances. In addition, in order to prove that our model also has benefit compatibility in the case of weak ASR system capability, we selected a batch of ASR error data from real vehicle data, including 560 utterances with 55.60% character error rate (CER). The model performance is evaluated in terms of equal error rate (EER) and accuracy.

For comparison, we first train models with a single modality separately. We select the backbone on several encoders currently used in the text encoders including GPT2 [7], BERT [13], Roberta [14], etc and audio encoders including Wav2vec2 [8], Whisper [15], SpeechTransformer [16], etc. Finally, we choose the encoder with the highest accuracy in each modality of training data. For audio modality, we use Wav2vec2 to model the sequence of speech frames. For text modality, we employ a GPT2 model which is based on the transformer. Then, we combine text and audio by training the Multi-modal as the baseline of M³V comparison.

As shown in Table 1, by combining speech and recognized text, the multi-modal approaches significantly boost unimodal approaches both text only and audio only, achieving 95.27% accuracy and 6.59% EER. In comparison to multi-modal methods, the proposed approach outperforms the direct concatenation approaches, showing the advantage of learned alignment between speech and text.

Our multi-view experiments use utterance-level representations to calculate the contrastive loss. Due to two disjoint networks sharing their learned information through the loss function, alignment information is learned in the multi-modal fusion head, resulting in a higher accuracy of 96.27%. On this basis, the individual and aligned modal results are also output. Results from multiple perspectives will be used for downstream strategies.

In the policy experiment evaluation module, we select 560 utterances with ASR errors to evaluate the generalization of the model in a challenging scenario. In order to quantify the improvement of the method, we first performed an accuracy evaluation of human performance. We selected five annotators to label these data and finally took the average as the accuracy of human performance.

As shown in Table 2, the accuracy of manual annotation is 92.60%. By combining speech and text, the multi-modal model is improved by 0.79% compared to the manual annotation, and the accuracy of using M³V model is improved by 2.40%, reaching an accuracy of 95% in the ASR error data.

By feeding the multi-view results of the model into the policy decision module, the accuracy of the task is further significantly improved. Compared with manual annotation, the accuracy of Policy I is improved by 2.94%, while Policy II is more significantly improved by 3.11%, achieving the best effect on the ASR error dataset. The above experiments proved that the model plus strategy method could reduce the dependence on ASR error and improve model robustness.