DSC IIT-ISM at SemEval-2020 Task 8: Bi-Fusion Techniques for Deep Meme Emotion Analysis

Similar to textual content on social media from other sources like tweets and comments, the textual part of a meme also contains important cues that can help extract the sentiment and emotion of the meme. Since the textual part of a meme gives it some specific context, it can sometimes be more detrimental in analysing the sentiment or emotion behind a meme. Therefore, as the first approach in our work we analyse some of the text based models to solve the three tasks.

BiLSTM: Long Short-Term Memory (LSTM), [Hochreiter and Schmidhuber, 1997] is a widely known recurrent neural network (RNN) architecture. We use Bidirectional LSTM [Schuster and Paliwal, 1997] models for our experiments. A Bidirectional LSTM (BiLSTM) layer processes the text in the forward as well as backward direction and hence is known to give better context understanding.

RoBERTa: Bidirectional Encoder Representations from Transformers, or BERT [Devlin et al., 2018] has been a revolutionary language model for a while now which is based on the self-supervised pretraining technique and has shown to generalize extremely well to downstream tasks. The recently released RoBERTa - A Robustly Optimized BERT Pretraining Approach [Liu et al., 2019], builds on BERT’s language masking strategy and modifies key hyperparameters in BERT, including removing BERT’s next-sentence pretraining objective, and training with much larger mini-batches and learning rates. RoBERTa has also been trained on a much larger text corpus than BERT, for a longer amount of time. This allows RoBERTa representations to generalize even better to downstream tasks compared to BERT. Here we use Roberta-Base to encode the textual part of the meme.

It is well known that the human brain prefers and is faster at processing visuals as compared to reading text. This is mainly because the entire image is perceived at once while text has to be processed linearly. On seeing a meme, the image is the part one perceives first and so it can provide a generic context on top of which the text part acts. We extract the visual representation of images in the form of abstract as well as specific features.

AlexNet: AlexNet [Krizhevsky et al., 2012] is a widely known convolutional neural network (CNN) which consists of convolutional and fully connected layers and is one of the first networks to use rectified linear unit as its activation function. We do not use a pretrained model and train from scratch on the dataset.

ResNet: ResNet [He et al., 2015], short for Residual Networks is a classic neural network used as a backbone for many computer vision tasks. ResNet makes use of the skip connection to add the output from an earlier layer to a later layer thereby helping it mitigate the vanishing gradient problem. For the sake of experiments reported in this paper we use ResNet with its ImageNet [Deng et al., 2009] pretrained weights.

Facial Expressions: On deeper exploratory analysis of the given dataset we found that most of the meme images contain faces of one or more people in them. The facial expression of these people might contain useful information about the overall sentiments expressed by the meme. ?) have already implemented an intuitive end-to-end model for generating rich face emotion captions from meme images using a ResNet to generate emotion features which are used by a captioning model to generate caption sequences. Therefore, as a part of our visual approach, for capturing facial expressions in our dataset we try using their pretrained ResNet block for generating predictions of the current task. However, because of poor performance of this model on the validation set we exclude it from our experiments on the test set.

Face Emotion Captions: Unlike our earlier approach of processing facial emotions in meme images, this time we try incorporating facial expressions as a third channel of information in addition to the image and meme overlay text. This is done by building our textual models on top of the text captions generated by the ?) model and then treating these textual models as a third input in our fusion approaches, described later. It is also worthwhile to mention here that for the memes that did not contain faces, random captions were generated by the model. However again due to poor performance of these models on the validation set we exclude this from our experiments on the test set.

The above methods consider both modalities individually for representation learning. However, effective computational processing of internet memes needs a combined approach. After extracting the text and image features they need to be combined to get a complete representation of a meme. Figure 1 gives a high level illustration of the modality fusion techniques that we use.

Early Fusion: This method involves simply concatenating the textual and visual feature vectors extracted by above mentioned models to produce a joint representation before passing it through a softmax classifier. We do this for the features generated by BiLSTM & AlexNet, and also for RoBERTa & ResNet.

GMU: The Gated Multimodal Unit [Ovalle et al., 2017] is a building block to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. We use GMU particularly to combine the representations produced by RoBERTa and ResNet.

Late Fusion: In this method, instead of combining text and visual models at the time of training, we fuse the models at inference time by using their individual softmax prediction scores. During our experiments we assign equal weights to text and visual models to take effectively an average of the prediction scores of the models to make the final predictions. We do this to combine the predictions of RoBERTa with ResNet.

In all the above approaches we have to train separate models for each subtask of the three tasks thereby requiring a total of 8 models to be trained per submission. Using multitask learning (MTL) enables us to train a model with multiple outputs which can be used for multiple classification tasks simultaneously. In MTL we have the hidden layers of the network shared among the tasks and also keep some individual layers for each output. MTL is known to leverage useful information contained in multiple related tasks to help improve the generalization performance of all the tasks [Zhang and Yang, 2017]. To this end, we use multitask models with early fusion of features from BiLSTM & AlexNet for Task A and the 4 subtasks of Task B together. We also use a similar model for solving Task A and the subtasks of Task C together.

Except for the MTL model, we train the following models separately for Task A and each of the subtasks of Task B & Task C. We use Keras with Tensorflow backend [Chollet and others, 2015] on Google Colab GPU to implement and train our models. Unless otherwise stated, all models use the default Adam optimizer with the Cross Entropy loss and trained till the validation loss stopped improving. We use a batch size of 32.

BiLSTM: As a preprocessing step, we remove the punctuation and symbols from the text data, convert to lowercase and tokenize to generate sequences with truncated length of 64. We train models with an input layer having embedding size 32, followed by a bidirectional LSTM layer with recurrent dropout and then a softmax dense layer. We keep the number of LSTM units to be 32 as higher values didn’t improve performance. Also, we apply a dropout of 0.5 in between the layers. To deal with the class imbalance, we use appropriate class weights for loss during training.

RoBERTa: For implementation details of RoBERTa-Base readers can refer to ?). Since transformer based models can handle long sequences we take the input sequence length for RoBERTa equal to the length of the longest text which was 199. Apart from preprocessing steps like lower casing, punctuation and symbol removal, we use oversampling to make the target class ratio 1 for handling class imbalance as the model tends to get biased towards one class on the given dataset. We fine tune the pretrained model on the given dataset using the output corresponding to the CLS token of the last layer followed by a batch normalization layer, a dense layer of hidden size 256 and finally a softmax layer. The model is trained with a learning rate of 3e-4.

AlexNet: As mentioned in the original paper we resize the images to 224 x 224 with 3 channels as input to the model. The filter sizes of the convolutional layers follow [96, 128, 128, 256, 256]. After flattening we apply 2 dense layers of size 1024 each before the final softmax layer. We apply a dropout of 0.4 after each layer and use class weights to handle imbalance.

ResNet: Before passing the meme images to the model all the images are resized to 256 x 256, rescaled and all the three RGB channels of the images are retained. We use the pretrained ImageNet weights to initialize the model and the outputs are followed by dense layers of hidden sizes [768, 256] and a softmax layer. A learning rate of 3e-4 is used to train the model.

BiLSTM+AlexNet: After using the same process and architecture as mentioned in their individual training details, we replace the softmax layer in the BiLSTM and the AlexNet model with a dense layer of 64 units to get the text and image vector representations. Then we pass both through a shared dense layer of size 64 and concatenate (Early Fusion) the outputs to form a joint vector of length 128. Then we keep another dense layer of 128 units before the softmax prediction layer. Here also we employ class weight balancing.

RoBERTa+ResNet: For the pretrained models we have explored three ways of fusion of information, as mentioned earlier, viz Early Fusion, GMU and Late Fusion. For Early Fusion (Figure 1(a)), we batch normalize the outputs of the models, concatenate them and finally pass them through a softmax layer. For GMU (Figure 1(b)), we take a bimodal GMU unit which is connected to the two models as input and has a softmax layer connected at its output. Finally, for Late Fusion (Figure 1(c)), we take the predictions of the two earlier separately trained models and average their prediction probabilities to get the final predictions. Since both the pretrained architectures are relatively deeper neural networks we use SGD optimizer while training the Early Fusion and GMU models to prevent gradient vanishing and oversampling to prevent the model from getting biased.

MTL: For the multitask learning approach we use the BiLSTM+AlexNet architecture, with slight modifications, and train from scratch. We train 2 models here, one for Task A with Task B and another for Task A with Task C. To handle the increased complexity of the model objective we add another bidirectional LSTM layer and double the vector size for each modality to 128. We also increase the size of the joint vector to 256 which is fed into separate 128 unit dense layers for each model task followed by individual softmax layers. For training the model we keep the loss contribution or weights equal for all outputs. We also try using the Task B semantic labels for auxiliary tasks with Task A as the main objective but it doesn’t give any improvements.

The evaluation metric used here is the macro F1 score for Task A. For Task B and Task C, the macro F1 scores for each subtask were averaged. Table 2 has the model test set scores from our experiments as well as the baseline results released by the competition organisers. From the table we can see that the RoBERTa+ResNet model with Early Fusion has the best performance for Task A and Task B while the same model using the Late Fusion strategy has the best score for Task C. For the BiLSTM+AlexNet model, using MTL did not give any improvements. We find that among the 3 fusion strategies, Early Fusion scores best for Task A and Task B, Late Fusion gives the best performance on Task C while GMU gives intermediate results. We chose the BiLSTM+AlexNet model for our contest submission as it gave the best results during the development phase. However, after final evaluation the RoBERTa+ResNet models were found to be better.