Incorporating Domain Knowledge Graph into Multimodal Movie Genre Classification with Self-Supervised Attention and Contrastive Learning

Movie genre classification could be divided into two categories: single-modality-based and multimodal-based. The former predicts genres by one kind of modality data, such as posters, plot summaries, movie trailers, audios, or metadata. (Simões et al., 2016; Yadav and Vishwakarma, 2020) extract features from movie trailers, while (Wi et al., 2020) and (Ertugrul and Karagoz, 2018) focus on using only poster images or plot summaries. As single-modality data is relatively simple to process, these methods have achieved excellent performance. However, for multimodal-based methods (Behrouzi et al., 2022; Ben-Ahmed and Huet, 2018; Fish et al., 2020) which combine two or more modalities , the results have not been as good due to the complexity of data features.

Self-supervised learning is a special learning method without direct supervision signal. The supervision signal is generated by using the features of the dataset itself. Nowadays, self-supervised learning is attracting more attention from researchers, such as Next Word Prediction (Iter et al., 2020; Devlin et al., 2018), Automated Text Augmentation (Meng et al., 2021; Giorgi et al., 2021) in natural language processing and Colorization (Vondrick et al., 2018; Zhang et al., 2016), Context Prediction (Noroozi and Favaro, 2016; Misra et al., 2016) in computer vision. Moreover, self-supervised learning (Misra and Maaten, 2020; Hendrycks et al., 2019; Baevski et al., 2022; Chen et al., 2022) has great potential to replace fully supervised learning in representation learning domain.

In our paper, we provide a novel insight into the self-supervised learning. We develop a paradigm to the attention module by summarizing the distribution from constructed domain knowledge graph.

Contrastive learning is a technique that trains a model to differentiate between similar and dissimilar examples. Such method can be used to learn representations of data. Initially, this approach (Park et al., 2022; Wang et al., 2022a; Han et al., 2021) was explored in the self-supervised setting. The feature embedding is learned without explicit labels by solving a pretext task. Supervised contrastive learning (Khosla et al., 2020; Gunel et al., 2021) is another form of contrastive learning that employs annotated data to generate positive pairs by selecting samples from various instances of a specific category.

In the contrastive learning paradigm, positive and negative pairs are defined by semantic similarity. Neverthless, it is hard to be applied to multi-label classification because of multiple semantics. Recently, several work attends to bridging combination of multi-label and contrastive learning. (Dao et al., 2021; Wang et al., 2022a) compute the contrastive loss by determining a single label that best matches each sample. However, such methods also serve to increase the distance between the sample and other labels. (Zhang et al., 2022b) proposes a multi-label contrastive learning framework that utilizes a hierarchical structure to leverage all available labels, but in movie genre classification there is no hierarchical structure of labels. (Hassanin et al., 2022; Bai et al., 2022) aim to utilize the label embedding space. (Hassanin et al., 2022) adopts a center loss but fails to construct negative pairs. (Bai et al., 2022) focuses on figuring out the similarity between label embedding, which may harm the discriminative ability of fused features. To alleviate the problem of defining positive pairs, our proposed G-CACL module enlarges the genre embedding space and a centroid of genres is generated as the positive anchor for each sample. Negative samples are well-designed for the effectiveness of our module.

Knowledge Graph Construction. A domain knowledge graph is constructed full-automatically by using metadata. Our knowledge graph schema is derived from the fields of metadata and we define four types of entities as the name of fields:

which correspond to directors, titles, casts and genres respectively. Moreover, we define six kinds of relations:

which represent the relations of directors and titles, casts and titles, genre and titles, directors and cats, directors and genres, casts and genres. Notably we only use metadata from $\mathcal{D}_{train}$ to avoid data leakage. We visit the fields of metadata and extract each value as an entity which has a unique matching id. Furthermore, we traverse all entity pairs in metadata, and each entity pair forms a triplet with the corresponding $R$.

Knowledge Graph Embedding. In order to capture the relations between entities, we apply a translate model for knowledge graph embedding, for instance, TransH (Wang et al., 2014). Finally we get the embedding matrix of all entities ${Mat}_{e}\in\mathbb{R}^{N_{k}\times D_{k}}$, where $N_{k}$ denotes the number of entities in knowledge graph and $D_{k}$ is the dimension of entity embedding.

Utilization of Group Relations in Metadata. The group relations in metadata can aid the prediction of movie genres as illustrated in Section 1. Since the translate model has enabled knowledge graph capture the relations between entities, for $i$-th sample we traverse all the director and cast entities in metadata. Finally we obtain the embedding of entities from ${Mat}_{e}$:

where $E^{d}$ denotes the embedding of an director entity. $N_{di}$ and $N_{ci}$ are the number of directors and casts of the $i$-th sample respectively. Finally, the feature of knowledge graph $F_{i}^{K}\in\mathbb{R}^{D_{k}}$ is defined as the sum of all embedding in $K_{i}$. Notably, in the test phase all $E^{d}$ and $E^{c}$ of many samples do not exist in ${Mat}_{e}$. It is because that ${Mat}_{e}$ is trained by the entities of $\mathcal{D}_{train}$. $F^{K}$ becomes the 0 vector at this point and it can be formulated as:

We adopt an attention module to balance the weight of each modality for each sample. It is composed of a linear function and a sigmoid function. Notably the parameters of the attention module for three modality features are shared.

As illustrated in Figure 2, for multi-modality features $F_{i}^{T}\in\mathbb{R}^{D_{t}}$, $F_{i}^{I}\in\mathbb{R}^{D_{i}}$ and $F_{i}^{K}\in\mathbb{R}^{D_{k}}$, where $D_{t}$ and $D_{i}$ are the dimension of text feature and image feature extracted from Clip model, we apply batchnorm1d and linear projection function to convert them to the same shape $D_{p}$. After alignment, these three feature maps are entered into the attention module to obtain their attention scores $A_{i}^{T}$, $A_{i}^{I}$ and $A_{i}^{K}$. Next, these multi-modality features are multiplied by their corresponding attention scores and add up as the ultimate fused feature ${F}_{i}^{fu}\in\mathbb{R}^{D_{p}}$:

where $batchnorm$ and $project$ denotes the batchnorm1d and linear projection function respectively.

To ensure the reliable attention allocation of the attention module, we introduce the Attention Teacher (AT) module based on self-supervised learning. Unlike the previous methods which mainly create pretext tasks in vision or language domain, we mine the distribution feature from the knowledge graph. It is observed that different samples contain different numbers of entities when forming $F^{K}$. Thus it is natural to consider that varying scores of attention should be assigned to $F^{K}$ of different samples. Specifically, $F^{K}$ composed of a small number of entities deserves a lower attention score. Especially in testing phase, if $F^{K}$ is 0 vector as is presented in Formula (4), the attention scores should be close to 0 at this point. Moreover, we consider the degree of entities in knowledge graph, because an entity is less important if it has very few neighbours in the knowledge graph structure.

Taking above discussions into consideration, we finally utilize the following equations to define the pseudo label $\widetilde{Y}_{i}$ of attention scores for $F_{i}^{K}$:

where $W$ denotes the number of samples in $\mathcal{D}_{train}$ and $V$ is the degree of an entity in the knowledge graph. In Formula (6), we introduce the average number of sum of directors and actors $\overline{N_{d+c}}$ and the average degree $\overline{V}$ of all samples in $\mathcal{D}_{train}$. Our intention is that if the sum of directors and actors and the degree of current sample are both beyond the average number in $\mathcal{D}_{train}$, the corresponding pseudo label $\widetilde{Y}_{i}$ should become higher, otherwise it would be lower. Formula (6) requires the range of $\widetilde{Y}_{i}$ in $(0,1)$ to fit the range of produced attention weights which are limited by a sigmoid function.

Then we design an appropriate loss function to make the self-supervised attention scores converge normally. Since $A_{i}^{K}$ is a vector while $\widetilde{Y}_{i}$ is in scalar form according to Formula (6), $A_{i}^{K}$ is averaged as scalar form either to compute loss with $\widetilde{Y}_{i}$. Since the goal of AT module is to guide $A_{i}^{K}$ close to $\widetilde{Y}_{i}$, a regression loss is adopted in this module. Moreover, it is worth noting that $A_{i}^{K}$ and $\widetilde{Y}_{i}$ are both designed to be between 0 and 1. Thus directly applying l1 or l2 regression loss would limit the loss value in range (0,1) and have the risk of underfitting. In order to ensure a large enough gradient, we apply a logarithmic function to guarantee larges gradients instead of directly adopting l1 or l2 regression loss. Considering a batch of input with batchsize $B$, the self-supervised attention loss is defined as follows:

In this way the definition domain of loss function is limited to 0 to 1 with large enough gradient and monotonic increasing trend. From Formula (7), it can be observed that only the knowledge graph attention score $A_{i}^{K}$ is supervised by its pseudo label $\widetilde{Y}_{i}$ to train the attention module, while $A_{i}^{T}$ and $A_{i}^{I}$ do not participate in the training procedure. Nevertheless, in our experiment (Section 4.5) we find that the attention module trained with Formula (7) can produce reasonable scores $A_{i}^{T}$ and $A_{i}^{I}$ for texts and images.

We propose a Genre-Centroid Anchored Contrastive Learning (G-CACL) module which facilities the genre embedding from knowledge graph to strengthen the discriminative ability of ${F}^{fu}$ . Considering that in contrastive learning, each sample is typically assigned a single semantic label and positive pairs are defined by whether they belong to the same semantic label. However, in our task each sample is annotated multi-genre. If we regard each genre embedding as a positive anchor, it is hardly achievable to push the feature of sample close to all the positive anchors synchronously. To overcome this limitation, we attempt to represent the semantics of multiple genres in single anchor.

Specifically, for a batch of fused features ${F}^{fu}$, the genre embedding set of $i$-th feature ${F}_{i}^{fu}$ is:

where $N_{gi}$ is the number of annotated genres of ${F}_{i}^{fu}$. we enlarge the genre embedding space by computing the centroid $C_{i}$ of $G_{i}$:

which is regarded as the positive anchor for ${F}_{i}^{fu}$. We could obtain the union of genre embedding of all samples in the current batch:

where $N_{gj}$ is the number of annotated genres of all samples in the current batch. We define the complement set of $G_{i}$ in ${\textstyle\bigcup_{i=1}^{B}}G_{i}$ as the negative samples of ${F}_{i}^{fu}$:

Before computing loss, the centroid and genre embedding go through the linear function to be transformed into the same shape as ${F}^{fu}$:

The loss function of G-CACL module is defined as follows:

where $\tau$ is the temperature coefficient following (Khosla et al., 2020). In this loss function, we push the ${F}_{i}^{fu}$ close to its corresponding $f_{C_{i}}$, and away from the embedding of negative samples $S_{i}^{Neg}$. $q_{i}$ denotes the sum of similarity between ${F}_{i}^{fu}$ and each embedding in $S_{i}^{Neg}$.

Furthermore, for the multi-label classification, we adopt the binary cross-entropy loss. For a batch of output after the classifier, the multi-label classification loss is:

where $p_{ij}$ denotes the predicted probability that the $i$-th sample belongs to the $j$-th genre.

Finally, we compose $\mathcal{L}_{atten}$, $\mathcal{L}_{contra}$ and $\mathcal{L}_{class}$ as the ultimate training loss for our IDKG:

Dataset. We evaluate IDKG on two datasets, MM-IMDb and MM-IMDb 2.0. MM-IMDb dataset is a multi-label movie genre classification dataset released by (Arevalo et al., 2017) which contains 25959 films. Each sample is composed of a poster, a plot summary and metadata which covers the directors, actors, publication year, etc. Following (Arevalo et al., 2017), we randomly split the dataset into train set, test set and validation set at the ratio of 0.6, 0.3 and 0.1.

To further verify our framework, we create a novel and more challenging dataset, MM-IMDb 2.0. We collect 33742 movies with their posters, plots and metadata from the IMDB website. As is illustrated in Table 1, the ratio of the quantity of Drama to the quantity of Film-Noir is nearly 65:1. Besides, we also constrain several other tail genres. Moreover, we partition our dataset the same proportion as MM-IMDb dataset.

Domain Knowledge Graph. As presented in Section 3.1, we process the metadata of $\mathcal{D}_{train}$ into a domain knowledge graph. For MM-IMDb dataset, there are 2231455 triplets and 264271 entities in its domain knowledge graph. Since MM-IMDb 2.0 dataset has much more movies, the number of triplets and entities are 3512803 and 318299 respectively.

Implementation Details. IDKG is trained on Pytorch with a 2080ti gpu. For the translate model for knowledge graph embedding, we use the toolkit released by (Han et al., 2018). We adopt the SGD optimizer with 0.5 learning rate and the embedding dimension is 200. Moreover, we train the translate model for 500 epochs with 100 batchsize. For the stage two we use the AdamW (Loshchilov and Hutter, 2019) optimizer and the learning rate is 1e-3. We train 15 epochs with 64 batchsize. To extract the image and text features, we apply the Clip (Radford et al., 2021) model and the parameters are frozen in our experiments. The unified vector space dimension is 512 which is set the same as (Arevalo et al., 2017). The parameter $\tau$ for the G-CACL module is set [0.05, 0.1, 0.3, 0.5, 0.7, 0.9] following (Khosla et al., 2020).

Evaluation Metrics. For the evaluation, we report the Micro-F1, Macro-F1, Weighted-F1 and Samples-F1 scores following (Arevalo et al., 2017; Seo et al., 2022) as our metrics.

We compare IDKG with state-of-the-art methods on two datasets and our approach achieves far superior performance. We categorize the existing methods into three types: 1) Multimodal type is a straightforward strategy to solve the problem. This type of works (Arevalo et al., 2017; Vielzeuf et al., 2018; Kiela et al., 2019; Braz et al., 2021; Sankaran et al., 2021; Xu et al., 2023; Li et al., 2022; Yu et al., 2022) mainly focus on the strategy of extracting the well-represented features from each modality respectively by adopting corresponding pre-trained models. It is noted that we also select several recent outstanding Multimodal methods (Xu et al., 2023; Li et al., 2022; Yu et al., 2022) as our competitors. 2) Graphical method is conducted to explore the potential of structural sources. MM-GATBT (Seo et al., 2022) leverages graph neural networks to learn the relational semantics of entities by using encoded images as node features. 3) Our method is a comprehensive framework which fully taking advantage of the two types above. Notably the translate model used for Table 2, 3 and 5 is RotateE (Sun et al., 2019) and the ablation study on translate models is shown in Section 4.3. Moreover, we compare our IDKG with GMU (Arevalo et al., 2017) for head and tail genres on two datasets with Macro-F1 scores as the evaluation metric.

Results on MM-IMDb Dataset. The comparison results on MM-IMDb dataset are shown in Table 2. We observe that the Graphical method MM-GATBT (Seo et al., 2022) outperforms all the Multimodal methods in each metric, which demonstrates that the semantic relations in metadata could serve a great benefits to the capacity of predicting genres. Table 2 shows that IDKG surpasses MM-GATBT by at least 15% on all evaluation metrics. One possible reason may be that in MM-GATBT the graph nodes are composed of the image embedding, where there is no additional knowledge in graph nodes at bottom. IDKG incorporates the knowledge graph embedding with other modalities by using the group relations in metadata which enriches the features for genre prediction. Moreover, two effective modules are designed to address the unreliable attention allocation and indiscriminative fused feature issues, thus boosting the performance of our model.

Results on MM-IMDb 2.0 Dataset. The overall results show that the performance of all methods on MM-IMDb 2.0 dataset is inferior to that on MM-IMDb dataset. The reason can be that MM-IMDb2 2.0 dataset is a more challenging dataset. As shown in Table 2, our proposed IDKG also successes in beating all the competitors by at least 12% which is a large margin on all metrics. The experimental result demonstrates that taking advantage of both Multimodal and Graphical methods can remarkably boost the performance.

Macro-F1 score analysis. As shown in Table 1, the distribution of genres is imbalanced in MM-IMDb dataset. Towards severer imbalance problem, we enlarge the proportion of the number of head genres and tail genres when collecting MM-IMDb 2.0 dataset as mentioned in Section 4.1. As Macro-F1 neglects the proportion for each label, it is more sensitive to the imbalance of genres distribution than other evaluation metrics. Thus we compare Macro-F1 between GMU and IDKG for sampling six head genres and six tail genres on two datasets as can be seen in Figure 3 and the genres are arranged in descending order of quantity from left to right. We could observe that for GMU the Macro-F1 of head classes is generally far larger than that of tail classes on two datasets due to the imbalance distribution. However, for IDKG the Macro-F1 of tail genres is close to that of head genres and almost Macro-F1 of all genres is around 80%, which demonstrates distinguished classification ability of IDKG.

To evaluate the effectiveness of each component of IDKG, we conduct extensive ablation experiments on two datasets. In particular, we incorporate the domain knowledge graph and the G-CACL module into GMU (Arevalo et al., 2017) one by one for more solid proof. After the ensemble of all modules, we name the new model IDKG (GMU). Notably since GMU provides a strategy of feature fusion, AT module is not conducted in its ablation experiment. Moreover, we compare the different translate models for knowledge graph embedding and show the influence on IDKG performance. Finally, we analyze the parameter $\tau$ of G-CACL module.

Results on IDKG. Table 3 illustrates that the discarding of domain knowledge graph loses the most performance by at least 10% of all modules which proves that group relations in metadata could contribute to the model performance greatly. With the removal of AT module, the performance of IDKG on all metrics declines to various degrees on two datasets. This is may be that AT module ensure the reliable attention allocation, thus improving the accuracy. The performance goes down if IDKG is not equipped with G-CACL module. The reason is perhaps that G-CACL module could improve the discriminative ability of fused feature which could boost the effectiveness of our model.

Results on IDKG (GMU). As can be seen from Table 3, the trend of performance change on IDKG (GMU) is the same as IDKG. This illustrates the effectiveness of our proposed modules. It is noted that the performance drops significantly when remove G-CACL module for IDKG (GMU). It demonstrates that GMU is weak in the discriminative ability of fused feature and our G-CACL module compensates the shortcoming.

Effect of Translate Models. We compare different translate models including TransH (Wang et al., 2014), TransR (Lin et al., 2015), TransD (Ji et al., 2015), ComplEx (Trouillon et al., 2016), ConvE (Dettmers et al., 2018) and RotateE (Sun et al., 2019) to observe the effects on IDKG performance. Moreover, we list the Hit@10 results of predicting missing links on WN18 dataset, which are reported in the toolkit (Han et al., 2018). As show in Table 4, we notice that the RotateE achieves the best performance and ComplEx performs worst. Combining the Hit@10 results, we could draw a conclusion that the performance of IDKG is correspond to the capacity of translate model due to strong embedding representation enhancing the ability of capturing relations between entities.

Analysis on parameter $\tau$. As illustrated in (Wang and Liu, 2021), smaller $\tau$ is sensitive to difficult negative samples that too small $\tau$ would destroy the embedding space due to the unproper negative samples setting. According to Figure 4, when $\tau$ is 0.1 the performance of IDKG is the best on MM-IMDb dataset. The reason may be that in our method the negative pairs are rationally constructed, thus small $\tau$ would benefit the contrastive loss.

To demonstrate the effectiveness of G-CACL module, we not only conduct extensive ablation study as illustrated in Section 4.3, but also replace it with existing multi-label classification methods which adopt contrastive learning. The competitors are MulCon (Dao et al., 2021), MCL (Hassanin et al., 2022), MLTC (Wang et al., 2022a) and C-GMVAE (Bai et al., 2022) and all of them are introduced in Section 2.3. The results demonstrated in Table 5 verify that our G-CACL module is superior to other contrastive learning methods. We observe that the performance of embedding space initialized randomly is worse than that initialized from the genre embedding of our knowledge graph. We assume that it is because the knowledge graph embedding captures the discrimination between genre semantics, thus improving the effectiveness of our module.

To test the validity of AT module, we present a case study as shown in Figure 5. We record $A_{i}^{I}$, $A_{i}^{T}$ and $A_{i}^{K}$ in testing phase and sample 3 cases, where the modalities that contribute more to the prediction are marked in red. It is observed that the attention module outputs the reliable scores for each modality. Notably despite that $A_{i}^{I}$ and $A_{i}^{T}$ are not directly supervised, they still output the rational scores. For example, the plot summary of the second and third sample should be allocated more attention weights and $A_{i}^{T}$ of them are relative high. We infer that $A_{i}^{I}$ and $A_{i}^{T}$ are soft-supervised because of the shared parameters of the attention module.