Multimodal Representation Learning With Text and Images

Supervised learning models are trained using a labelled dataset. And as a result, these models learn features in data while mapping it to the respective output labels. There are several drawbacks of using Supervised learning. If the input to the model is irregular or noisy, it will give drastically inaccurate results. It requires the training set to include all possible examples that belong to a class, else it will output a wrong class. To overcome these limitations, we adopt unsupervised representation learning techniques to learn abstract representations in raw data. The theory used is to reduce high-dimensional data to lower dimensions while retaining essential information so as to discover new patterns in data. This also reduces the manual task of labelling the dataset.

Visual data such as images, videos usually require a domain expert to identify and define the certain qualities algorithmically, like the transformations to be used. To make the model invariant to the input, Deep Neural Networks (DNNs) are used to learn representation of input data such as text, images. This is done using an iterative process, wherein DNNs learn to make good predictions by iteratively updating behaviour of neurons in all layers. This requires the domain experts to define the target labels and the accuracy they want from the network. The output of the current layer will serve as inputs to the subsequent layer, and the final-most layer outputs predicted labels. Once the network is performing with the desired accuracy, the iterative process can come to a halt. This network can now be fed as a pre-trained network to a different learning task, as the penultimate layer of this network outputs the abstract representations for new task, also known as transfer learning.

In domains such as healthcare, input data can come from various modalities, such as X-ray images, clinical records, clinical features etc. A patient can be described by all of these features simultaneously. As a result, to represent an entity like a patient, would need methods that can learn jointly from these varied sources of data, which also vary in modalities. Multi-modal representation learning is thus an area of interest in such domains.

As we often work on large unlabelled datasets, we use unsupervised representation learning on multi-modal data, consisting of images and text in particular, in order to efficiently reduce the problem of manual feature engineering, which is a computationally expensive process. We leverage the capabilities of autoencoders - vanilla and convolutional- along with matrix factorization methods to achieve this.

Matrix factorization methods have been used in learning representations from collections of matrices. However, they require the inputs to be in the form of matrices. In case of inputs like images, additional processing would be necessary to convert such higher dimensional tensors into matrices. In this paper, we propose a novel architecture that can take in a combination of tensors and matrices as inputs, and learn representations for the associated entities jointly, in an end-to-end fashion.

In the subsequent sections, we will first survey existing matrix factorization methods that work on an arbitrary collection of matrices, the details of our proposed architecture, the details of the dataset we use for evaluating our model, the model development and the comparison between existing methods and our model. We conclude with how this model can be enhanced further.

For multi-modal data setup, matrix factorization methods are generally preferred. Some of the common methods found to be useful in learning representations for entities from a collection of matrices are Collective Matrix Factorization[6], Data Fusion by Matrix Factorization[11] and Deep Collective Matrix Factorization [2]. These methods learn entity specific representations for the entities along the rows and columns of the matrices and use them to reconstruct the matrices. However, these methods expect the input to be a collection of matrices, and tensors are not supported. Our model overcomes this limitation by combining the reconstruction capabilities of matrix factorization and the ability of convolutional autoencoders to work on tensor inputs. Thus, with our proposed architecture, we can jointly learn representations for entities based on inputs of any dimensions(matrices or tensors).

CMF factorizes a matrix into two factors – one for the row entity and another for the column entity. If $U^{[r_{m}]}$ represents the row entity representation and $U^{[c_{m}]}$ represents the column entity representation for a matrix $X_{m}$, with $r_{m}$ as the row and $c_{m}$ as the column entity, $X_{m}=f_{m}(U^{[r_{m}]}.U^{[c_{m}]T})$, where $f_{m}$ is a matrix specific link function. However $X_{m}$ can only take scalar values in each of its cells.

DFMF performs a tri-factorization of a matrix R as $R=G_{i}S_{ij}G_{j}^{T}$ where $S_{ij}$ is a factor describing the relation between the row and column entities. $G_{i}$ and $G_{j}$ are the row and column representations for the relation matrix R. This also takes in as input a collection of matrices. Also, here R is assumed to be a binary matrix, where each cell denotes the degree of a relationship – presence or absence.

DCMF concatenates the matrices sharing the same entity, and uses autoencoders to factorize the concatenated matrices into entity representations. Here, both real and binary values are possible for the cells making up the matrices, however the collection can only comprise of matrices, due to the concatenation process. For example, if an entity E is described by both a one-dimensional vector and by a 3-dimensional image, concatenating the two views would be difficult.

The model we propose can handle the presence of two differing dimensional views of the same entity, like a book denoted by a textual representation (which can be represented using Word2Vec[3]) and by images. This finds extensive application in many domains like the healthcare sector, where patients can be described by both their clinical records as well as their X-ray records.

We propose an end-to-end architecture for multi-modal representation learning. Figure 1 shows the proposed architecture - Multi-Modal Encoder Decoder Architecture - II(MMEDA-II). To explain our architecture, we will be considering the case of two inputs - a matrix and a three-dimensional tensor.

The model accepts three input matrices: X0 (of dimensions m x n1 x k), X1 (of dimensions m x n2). A third matrix X2 ( of dimensions n2 x m) is created by transposing X1. In this case, the $i^{th}$ row in X0 and X1 correspond to the attributes of the same entity. X0 is passed through a convolutional autoencoder (AE0) and X1 and X2 are passed through a feed forward vanilla autoencoder (AE1 and AE2 respectively). The decoding output from autoencoders AE0, AE1 and AE2 are represented by X0’, X1’ and X2’ respectively.

Because we want the final representation for a data point to include information from both matrices X0 and X1, output from the encoder layer of AE0 is concatenated with output from the encoder layer of AE1 and passed through a feed forward network to obtain the final representation. This final representation is multiplied with the transpose of the output of the encoder layer of AE2 to get matrix X1”.

The dataset used in this project is the Goodreads dataset from [9] and [10], with about 2.3 million books found on the Goodreads website, with cover images and features like average ratings, reviews, genre categories and book descriptions as some of the available features.

In this dataset, we are interested in learning lower dimensional representations of books, based on the book descriptions and the cover images.

To conduct our experiment, the available dataset was divided into 80-20 train and test splits, with 4000 books in the train and 1000 books in the test splits. The complete set of 5000 books, with their Word2Vec representations and cover images were passed through the models described above to obtain lower dimensional representations, of 50 dimensions. The learnt representations for the 4000 books in the train set were used to train the downstream task, and the representations for the 1000 test books were used to evaluate the representations.

The downstream tasks were the prediction of the book average rating, modelled as a regression task, and the genre classification, modelled as a classification task. The regression was done using the Linear regression package in scikit-learn and classification using RandomForest from scikit-learn. A maximum depth 2 and random state 0 were used for the RandomForest classifier. We consider the F1 score, accuracy, precision and recall as the evaluation metrics for classification and mean square error for regression.

Having obtained the processed dataset, we first used a supervised multi layer perceptron (subsection 6.1) as a baseline to compare our models against. We started with a model that used matrix factorization (subsection 6.2), similar to the algorithm in [2], on the pre-trained one-dimensional image representations and Word2Vec representations. Since these pre-trained models are trained to perform a specific task, and our goal was to obtain image representations that are not mapped to any label, we replaced the pre-trained models with convolutional autoencoders. This lead to the model described in subsection 6.3. In the subsequent models, we modify the architecture to train both the convolutional autoencoders and the matrix factorization architecture in an end-to-end fashion (subsections 6.4 and 6.5).

The figure 2 shows the overall pipeline followed.

From our experiments, we evaluated the results of the proposed model and the models trained during the model development process, against popular matrix factorization methods like CMF. For CMF to work, we obtain input matrices by passing the image tensor object through pre-trained CNN networks like GoogleNet, InceptionNet, ResNet and VGG. We also compare the methods against a baseline multi-layer perceptron. The results are shown in Table 1. The representations learnt from the various architectures are used to perform a downstream classification and regression task. The F1 score, precision, recall are compared for classification, while the mean square error is used to compare the regression results. The results for the 4 pre-trained CNN models appear quite similar, probably because they are pre-trained on the same dataset, and towards the last layers, the representations learnt could be similar.

From the methods in Table 1, we obtain the best performing method(CMF using ResNet image representations) and compare it against the proposed method, along with the intermediate methods obtained during model development. These results are shown in Table 2. Although unsupervised, the representations learnt from our proposed architecture appears to do well. The joint learning from the different modalities, directly from the input tensors, could be one reason for this. Being an unsupervised learning method, the learnt representations are reusable and can be used for any downstream task, including clustering. The availability of fewer data points could be one explanation for the relatively poor performance of MLP network, though it is a supervised learning framework. Thus, it can be concluded that the proposed method can do well while learning representations from a collection of matrices and tensors.

Thus, in conclusion, we see that our proposed method can learn representations jointly from a collection of matrices and tensors. This is especially useful in domains such as healthcare, where a patient can be viewed from different modalities, like X-rays, clinical records etc. The proposed architecture is found to outperform benchmark matrix factorization methods and supervised multi-layer perceptrons.

Representing the text inputs using transformer models, instead of averaging Word2Vec representations, which can be trained along with the representation learning, and extending this architecture to work on sequential data, like videos, are other directions worth exploring. Replacing vanilla autoencoders with convolutional autoencoders in existing matrix factorization architectures and using variational autoencoders instead of vanilla autoencoders can also be explored.