Multi-modality Associative Bridging through Memory:
Speech Sound Recollected from Face Video

Audio-visual multi-modal learning is one of the active research areas. There are two categories of audio-visual multi-modal learning using DNNs: fusion and finding a common latent space of cross-modal representation. The fusion methods [48, 33, 10] aim to exploit the complementary information of different modalities and achieve high performance compared to the uni-modal methods. They try to find the best fusion architecture of a given task [1, 34, 36, 28]. However, as the fusion methods receive all modalities as inputs, they could not properly work if one of them is not available. The learning methods finding a common latent space from multi-modal data [35, 19, 4, 24] aim to reduce the heterogeneity gap between the two modalities. Several works [7, 8, 14] have proposed metric learning methods and adversarial learning methods to find the common representation. Other works [63, 2] have proposed to learn from superior modality for a given task using knowledge distillation [18] which guides the learned feature to resemble the superior modal feature. Although finding a shared latent space or guiding one modal representation to resemble the other has the advantage of using the common information between the two modalities with uni-modal inputs, reducing the heterogeneity gap between the multi-modal data is considered as a challenging problem [64, 37].

In this paper, we try to not only take the advantages of both methods, but also alleviate the problems of each method. We propose to handle the audio-visual data using two modality-specific memory networks connected with an associative bridge. During inference, the proposed framework can exploit both source and the recalled target modal contexts even when the input is uni-modal. Moreover, since each modality works on its corresponding modality-specific module, we can bypass the difficulty of finding a shared latent space.

Memory network is a scheme to augment neural networks using external memory [55, 46]. They have shown the effectiveness of memory network on modeling long-term dependencies in sequential data [29]. Miller et al. [32] introduce key-value paired memory structure where key memory is firstly used to address relevant memories with respect to a query, extracting addressed values from the value memory. We utilize the key-value memory network [32], where the key memory is for saving the source modal features, and the value memory is for saving the target modal features. Thus, we can access both source and target modal contexts by recalling the saved target modal feature from the value memory when only source modality is available.

The memory network is also used in multi-modal modeling. Song et al. [44] introduce a cross-modal memory network for cross-modal retrieval. Huang et al. [22] propose an aligned cross-modal memory network for few-shot image and sentence matching. Using a shared memory, they encode memory-enhanced features, which will be used for image/text matching. Distinct from the previous methods, our proposed framework uses modality-specific memory network where source-key memory saves the source modality and target-value memory saves the target modality.

Lip reading is a task that recognizes speech as text from lip movements. Chung et al. [6] propose word-level audio-visual corpus data and a baseline architecture. The performance of word-level lip reading is significantly improved with the architecture [45, 38] of a 3D convolution layer, a ResNet-34, and Bi-RNNs. Some works [54, 56] use both optical flow and video frames to capture fine-grained motion. Xu et al. [57] suggest a pseudo-3D CNN for the frontend which is more efficient compared to vanilla 3D CNN. Zhang et al. [61] show that the lip reading can be made beyond the lips by utilizing entire face as inputs. Martinez et al. [31] improve the backend by changing the Bi-RNN into multi-scale temporal CNN.

It is widely known that the audio modality has superior knowledge for speech recognition than the visual modality by showing outstanding performance. In this paper, we try to complement the lip visual information by recalling the speech audio information from the proposed multi-modal bridging framework.

Speech reconstruction from silent video aims to generate acoustic speech signal from silent talking face video. Ephrat et al. [13, 12] firstly generate speech using CNN and they improve it with a two tower CNN-based encoder-decoder architecture whose inputs are both optical flow and video frames. Akbari et al. [3] propose to pretrain an auto-encoder to reconstruct the speech, whose decoder part is used to generate the speech from a face video. Vougioukas et al. [50] propose GAN based approach which maps video directly to audio waveform. Prajwal et al. [39] attempt to learn on unconstrained single-speaker dataset. They present a model that consists of stacked 3D convolutions and an attention-based speech decoder, formulating the task as a sequence-to-sequence problem.

The speech reconstruction from silent video is considered as a challenging problem, due to the information insufficiency of face visual movement to fully represent the speech audio. We try to provide complementary information with the recalled audio representation through the proposed associative bridge with memory, and enhance its performance. With both the visual and the recalled audio contexts, we can generate high-quality speech in both speaker-dependent and speaker-independent settings.

Each memory network inside the proposed framework saves generic representations of each modality. The generic visual and audio representations are produced from the respective modality-specific deep embedding modules. The visual (i.e., source modal) representation $f_{src}\in\mathbb{R}^{T\times C}$ is extracted by using spatio-temporal CNN that captures both spatial and temporal information, and the audio (i.e., target modal) representation $f_{tgt}\in\mathbb{R}^{T\times D}$ is embedded from 2D CNN whose input is preprocessed mel-spectrogram from raw audio signal, where $T$ represents the temporal length of each representation. Since the paired audio-video inputs are synchronized in time, the two embedding modules can be designed to output the same temporal length.

Based on the modality-specific representations, we firstly introduce how the source and target addressing vectors are formulated. The addressing vector refers to the guidance that determines where to assign weights on memory slots for a given query. Suppose that the source modal representation $f_{src}$ is given as a query, then the cosine similarity with source-key memory $\mathbf{M}_{src}$ is obtained,

where $s_{src}^{i,j}$ represents the cosine similarity between $i$-th memory slot of source-key memory and source modal feature in $j$-th temporal step. Next, the relevance probability is obtained using Softmax function as follows,

where $r$ is a scaling factor for similarity. By calculating the probability over the entire memory slot, the source addressing vector for the $j$-th temporal step $A^{j}_{src}=\{\alpha^{1,j}_{src},\alpha^{2,j}_{src},\dots\alpha^{N,j}_{src}\}$ can be obtained.

The same procedure is applied for target modal representation $f_{tgt}$ and target-value memory $\mathbf{M}_{tgt}$ to produce the target addressing vector, $A^{j}_{tgt}=\{\alpha^{1,j}_{tgt},\alpha^{2,j}_{tgt},\dots\alpha^{N,j}_{tgt}\}$ of $j$-th temporal step. The addressing vectors will be utilized in recalling the saved representations inside memory and connecting the two modality-specific memories, in the following subsections.

The obtained target addressing vector $A^{j}_{tgt}$ is to correctly match the target-value memory $\mathbf{M}_{tgt}$ for reconstructing target representation $\hat{f}^{j}_{tgt}$. To do so, the target-value memory $\mathbf{M}_{tgt}$ is trained to memorize the proper target modal representation ${f}^{j}_{tgt}$. We firstly obatin the reconstructed target representation $\hat{f}^{j}_{tgt}$ as follows,

Then, we design the reconstruction loss function to guide the target-value memory $\mathbf{M}_{tgt}$ to save the proper representation. We minimize the Euclidean distance between the target representation and the reconstructed representation,

With the saving loss, the target-value memory $\mathbf{M}_{tgt}$ saves the representative features of the target modality. Therefore, the recalled target modal representation $\hat{f}^{j}_{tgt}$ from target-value memory $M_{tgt}$ is able to represent the original target modal representation $f_{tgt}$.

To recall the target modal representation from the target-value memory by using the source-key memory and source modal inputs, we construct an associative bridge between the two modality-specific memories. Specifically, the source-key memory is utilized to provide the bridge between source and target modalities in the form of the source addressing vector. That is, through the source addressing vector $A^{j}_{src}$, the corresponding saved target representation is recalled. To achieve this, the source addressing vector $A^{j}_{src}$ is guided to match to the target addressing vector $A^{j}_{tgt}$ with the following bridging loss,

where $D_{KL}(\cdot)$ represents Kullback–Leibler divergence [27]. With the bridging loss, the source-key memory saves the source modal representations in the same location, where the target-value memory saves the corresponding target modal features. Therefore, when a source modal representation is given, the source-key memory provides the location information of the corresponding saved target modal representation in the target-value memory, using the source addressing vector.

Through the associative bridge and the modality-specific memories, we can obtain the recalled target modal feature $v_{tgt}$ by using source addressing vector $A_{src}$ as follows,

Here, the target modal feature $v_{tgt}$ is recalled by querying the source-key memory $\mathbf{M}_{src}$ with the source modal representation $f_{src}$. Thus, we do not need the target modal inputs for recalling the target modal feature. Then, we can apply the recalled target modal feature for a downstream task in addition to the source modality, improving task performance by exploiting the complementary information.

The proposed framework is trainable in an end-to-end manner, including the modality-specific embedding modules, memory networks, and the downstream sub-networks. To this end, the following task loss is applied,

where $g(\cdot)$ is a loss function corresponding to the downstream task, $h(\cdot)$ is a fusion layer such as a linear layer, $y$ represents label, and $\oplus$ represents concatenation. The first term of the loss function is related to the performance on a given task that utilizes both the source and the recalled target modalities. The second term guarantees that the target modal embedding module learns the meaningful representations that will be saved into target-value memory in an end-to-end manner.

Finally, the total loss function is defined as a sum of the all loss functions,

The pseudo code for training the proposed framework is shown at Algorithm 1.

Lip reading is a task that recognizes speech by solely depending on lip movements. We apply the proposed multi-modal bridging framework to the lip reading to complement the visual context by bringing superior knowledge of the audio through the associative bridge and to enhance the performance.

Speech reconstruction from silent video is a task of inferring the speech audio signal by watching the facial video. To demonstrate the effectiveness of the proposed multi-modal bridging framework, we apply the proposed framework to the speech reconstruction from silent video task to provide the recalled audio context in an early stage of decoding for generating a high quality speech.

In this section, we visualize the addressing vectors of both lip reading and speech reconstruction model in speaker-independent setting. Fig.3 (a) shows the video clips of LRW dataset with consecutive 5 frames and the corresponding addressing vectors of lip reading model. From the addressing vectors of different speakers speaking the same pronunciation, we observe the similar tendency of the addressing vectors. For example, when the face video is saying ``sta'' in words started and start, similar memory slots are highly addressed. The same tendency can be observed in the speech reconstruction model shown in Fig.3 (b). This shows that source-key memory consistently finds the corresponding saved audio location in the target-value memory by using the talking face video clips as a query, which means the associative bridge is meaningfully constructed.

In addition, we compare addressing vectors of facial video clips with different pronunciations. Fig.4 shows the consecutive video frames with its corresponding pronunciation, and the comparison results. We can observe that the source addressing vectors of saying similar pronunciation have high similarity, while differently pronouncing videos have low similarity. For example, video clips of pronunciation ``a'' of word about and amount have 0.906 cosine similarity. In contrast, the similarity between ``ri'' of word period and ``a'' of word about is low with 0.404.

We examine that the proposed framework can bypass the difficulty of finding a common representation of different modalities while bridging them. We compare the performance of word-level lip reading with the previous multi-modal learning methods that can exploit shared information of audio-visual modalities with uni-modal inference input by finding a common latent space. We build two multi-modal adaptation methods: cross-modal adaptation method [7] and knowledge distillation method [18]. The first is pretrained to synchronize the audio-visual modalities, and then trained for lip reading. The second method is additionally trained so that the features from lip reading model resemble the features from the automatic speech recognition model.

We show the word-level lip reading accuracies on LRW dataset in Table 5. By utilizing multi-modality with visual modal inputs only, all of the methods show the performance improvements from the baseline, and the proposed framework achieves the best performance. The comparison shows the efficiency of the proposed framework, where it does not need to find a common latent space of two modalities by dealing with each modality in a modality-specific memory.

Lastly, we visualize the representations of visual modality, audio modality, and recalled audio modality from visual modality, by mapping them into 2D space. Fig.5 shows t-SNE [49] visualization of learned representations of visual and audio modalities, and the recalled audio from visual modality and the actual audio modality. Since we handle each modality with modality-specific embedding module and memory, the two modalities have separate representations in the latent space (Fig.5 (a)). However, as Fig.5 (b) shows, the recalled audio from the visual modality through the associative bridge shares a representation similar to the audio modal representation. Thus, we can utilize both audio and visual contexts while maintaining their own modal representations. This visualization demonstrates that we can effectively bridging the multi-modal representations without suffering from the cross-modal adaptation by dealing with each modality in modality-specific modules.