Visually Aware Skip-Gram for Image Based Recommendations

Recommender Systems have been described as “an intuitive line of defense against consumer over-choice” (Zhang et al., 2019). With the rapid growth in the number of products available on e-commerce websites, this problem of ”over-choice” is becoming more and more significant, especially in the case of clothing related products. In fact, fashion recommendation has attracted considerable attention in recent literature. Studies tackling this problem have shown improved performance by incorporating visual information into their recommendation procedure. Hence it is safe to conclude that the visual appearance of a product plays a significant role while making purchase decisions.

Image-based recommendation systems can be broadly classified into two groups - (i) Systems which incorporate images into a product rating/rank prediction function. These systems build upon well known Latent Factor models like Singular Value Decomposition (SVD) or Bayesian Personalized Ranking (BPR). (Chen et al., 2019; He and McAuley, 2016; He et al., 2016b, a; Chen et al., 2017) (ii) Systems which learn latent representations (a.k.a embeddings) using image features and subsequently use embedding distance for making recommendations. These systems rely on extracting fine-grained embeddings which can capture user preference and/or product similarity (Guo et al., 2018; McAuley et al., 2015; Ying et al., 2018; Shankar et al., 2017; Jiang et al., 2016). Beyond images, information coming from product metadata (Vasile et al., 2016), user-product information networks (Shi et al., 2019), review-text (Guo et al., 2018; Cheng et al., 2019) etc. has also been leveraged for learning effective representations.

Making recommendations in the cold-start setting is a key area where addition of visual information has shown significant performance improvement (He and McAuley, 2016). However, we observe that there is considerable variability in the definition of cold-start setting across literature. Some studies describe cold-start products as items that have fewer instances of positive feedback in the training set (He and McAuley, 2016; He et al., 2016b; Shi et al., 2019; Chen et al., 2017; Liu et al., 2020), whereas (Vasile et al., 2016) describes its cold-start scenario when certain product-pairs have zero co-occurrences while training. Recommending completely new products (i.e. products which are unseen during the training step) is a tougher problem which is studied less frequently (Schein et al., 2002; Saveski and Mantrach, 2014; Gantner et al., 2010).

Systems falling under category (i) (as described above) rely on user and product latent factors coming from interaction records, hence, they are incapable of handling new products unless product latent factors are estimated by other methods. Systems falling under category (ii) show better promise of handling this problem. Methods like (Shankar et al., 2017; McAuley et al., 2015), which make recommendations based on product-product similarity are capable of extracting latent representations for a new product. However, here user preference is ignored while feature extraction hence making personalized recommendations is not possible. Methods which attempt to capture preferences for individual users are better suited for handling this problem.

To this end, we propose a novel framework which jointly learns (i) user and product embeddings in a common latent space and (ii) a mapping function which transforms product image features to the same latent space. In specific, we augment the Skip-Gram model (Mikolov et al., 2013b) with a decoder architecture where the decoder uses the product Skip-Gram embeddings to reconstruct their corresponding image features. Here Skip-Gram maximizes the probability of co-occurrence of users and products in a Heterogeneous Information Network (HIN) while the decoder minimizes the image feature reconstruction loss. This decoder serves two purposes- (i) it incorporates visual information into the skip gram embeddings and (ii) it is later used for learning a mapping function which transforms product image features to the Skip-Gram embedding space. The learning objective of this architecture is dependent on the genre of input (users or products) observed while training. The Skip-Gram model is optimized in case of both users and products whereas the decoder is updated only for products in a multitask fashion. We term this model as VASG (Visually Aware Skip-Gram).

The mapping from the product image feature space to VASG embedding space is subsequently learned and can be used for finding effective embeddings for cold-start products. VASG embeddings reflect user preference while simultaneously capturing a product’s purchase history and visual appearance. Since users and products are represented in the same latent space, personalized recommendations can be made directly by searching for products in the user’s vicinity. Our contributions can be summarized as -

• •

  We propose a novel multitasking Skip-Gram architecture, VASG, which is trained on two different objectives depending on the genre of the input (users or products). This architecture enables us to learn a mapping which transforms product image features to the learned embedding space.

• •

  Extensive experiments are performed on real world datasets and VASG embeddings are compared to state-of-the-art recommendation systems. The performance for unseen cold start products is also studied.

• •

  We analyze the learned embeddings in detail to shed some light on the information captured by them.

Our work relies on several advancements in image feature extraction methods, representation learning techniques in graphs and multitask learning.

With advancements in deep learning, VBPR (He and McAuley, 2016) established that the addition of visual signals can be useful for making recommendations. VBPR adds image features extracted from a pre-trained convolutional neural network to the BPR framework. Similarly, (Chen et al., 2017) extracts more fine grained image features using attention modules and incorporates them in collaborative filtering. Images have also been proven to be useful for Point-of-Interest recommendations in (Wang et al., 2017) and for tag recommendations in (Rawat and Kankanhalli, 2016).

More recently, there has been a surge of methods which combine information coming from different modalities and metadata with images (Cheng et al., 2019; He et al., 2016b; Guo et al., 2018; He et al., 2016a; Chen et al., 2019). Attention mechanisms which utilize textual reviews have been used for finding specific parts of an image where users are most interested in (Chen et al., 2019). Reviews and image features have also been used for learning user preference which is then integrated into a rating-based matrix factorization model (Cheng et al., 2019). Pre-defined product categories and hierarchy trees have been leveraged for learning hierarchical product embeddings (He et al., 2016b).

Recommendation Systems which rely on product image similarity; for example methods which address the Street-to-shop recommendation problem (Liu et al., 2012); have received a tremendous boost with improvements in metric learning methods (Gajic and Baldrich, 2018; Jiang et al., 2016; Shankar et al., 2017) . These methods use triplet loss to learn embeddings from product images which are robust to change in background, pose, lighting conditions etc. and hence can be used for making cross-domain recommendations. However, learning both user and product representations in a joint latent space is less frequently explored in existing literature. (Gong et al., 2020) embeds users and topics to the same low dimensional space to capture of their mutual dependency while (Guo et al., 2018) attempts to bring users, products and product search queries to the same latent space using both textual and visual modalities.

Heterogeneous Information Networks (HINs) can naturally model complex user and product interactions. Traditionally, meta-path based similarity and link predictions have been used for making recommendations in networks. With the development of graph representation learning methods (Perozzi et al., 2014; Grover and Leskovec, 2016; Dong et al., 2017), network embedding based recommendation systems have shown improved performance. The study (Shi et al., 2019) learns user and product representations from a HIN and incorporates them into the SVDFeature (Chen et al., 2012) framework. This idea of learning network representations has also been extended by using multitask learning for jointly optimizing the tasks of recommendation and link prediction in a HIN (Li et al., 2020). Multitask learning is a training regime where multiple objectives are optimized simultaneously. It has proven to provide performance boosts in works like (Gidaris et al., 2019; Xu et al., 2020).

A few studies which are relevant to our proposed approach should be discussed here. Learning attribute-to-feature mappings has been proposed in (Gantner et al., 2010). Here k-nearest neighbours, least squares approximation is used for approximating latent factors for new users or products. A large scale recommender system which uses images along with the HIN structure is proposed in (Ying et al., 2018). Here highly efficient Graph-Convolutional Neural Networks are used on random walks generated from the HIN and recommendations are then made using similarity between the learned embeddings. Contrastive Predictive Coding (Oord et al., 2018) (CPC) is a generic representation learning regime which uses a similar training procedure as the proposed framework. CPC uses encoders to map raw data to latent representations and subsequently trains auto-regressive models using a contrastive loss. VASG differs from CPC in terms of usage of a decoding architecture instead of an encoder; usage of Skip-Gram instead of an auto-regressive model; and training using auxiliary supervision instead of self-supervision.

To the best of our knowledge, the model architecture proposed in this paper has not been studied before. In addition, we also make a novel attempt to bring unseen cold start products to the same latent space as existing users and products using product image features.

Consider a set of users $\mathcal{U}$ and a set of products $\mathcal{P}$ where each user $u\in\mathcal{U}$ has interacted with a subset of products $\mathcal{P}\textsubscript{\it u}\subset\mathcal{P}$. Each product $p\in\mathcal{P}$ is associated with a visual feature $f\textsubscript{p}\in\mathbb{R\textsuperscript{I}}$ which is extracted from its image using a Deep Convolutional Neural Network. Only a subset of products $\mathcal{P}\textsubscript{warm}\subset\mathcal{P}$ is observed during training and is termed as the warm start set. The set of products which is never observed during training is termed as completely cold set $~{}\mathcal{P}\textsubscript{cold}\subset\mathcal{P}$ . It is to be noted that $\mathcal{P}\textsubscript{cold}\cap\mathcal{P}\textsubscript{warm}=\phi$.
Our objective is to learn (i) a D dimensional latent representation $\mathcal{X}\in\mathbb{R}\textsuperscript{\it D}$ for each $u\in\mathcal{U}$ and $p\in\mathcal{P}\textsubscript{warm}$; and (ii) a mapping function $\mathcal{M}(f\textsubscript{p}):\mathbb{R}\textsuperscript{\it I}\rightarrow\mathbb{R}\textsuperscript{\it D}$. This function is used for finding embeddings of products that have no purchase history. The learned embeddings are expected to represent characteristic features of users and products which can be exploited for making representations. The VASG framework achieves the above mentioned objectives in two steps-

• •

  The first step involves training a Skip-Gram model on sequences of users and products. Here, the Skip-Gram architecture is modified by adding an auxiliary objective of learning an inverse mapping function $\mathcal{M}\textsuperscript{-1}(\mathcal{X}\textsubscript{p}):\mathbb{R}\textsuperscript{\it D}\rightarrow\mathbb{R}\textsuperscript{\it I}$. The D dimensional embeddings corresponding to users $u\in\mathcal{U}$ and products $p\in\mathcal{P}\textsubscript{warm}$ are learned during this step.

• •

  The second step involves learning the mapping function $\mathcal{M}(f\textsubscript{p}):\mathbb{R}\textsuperscript{\it I}\rightarrow\mathbb{R}\textsuperscript{\it D}$. This is achieved by training a deep encoder-decoder architecture using the inverse mapping $\mathcal{M}\textsuperscript{-1}(.)$ learned in the previous step.

We describe the network structure used for representing user-product interactions before describing the model architecture.

We perform experiments on real-world datasets to evaluate the VASG embeddings. Our experiments attempt to answer the following research questions - (i) Can the embeddings generate personalized recommendations for each user? (ii) Can the embeddings identify product to product relationships? (iii) What are the properties displayed by the learned embedding space? (vi) How do different components and hyper-parameters of the proposed framework affect its performance?

Embedding Visualization. We visualize the learned embeddings in lower dimensions to understand the information captured. We plot 2D projections of a random set of products from each dataset in Figure 2(a). Here dimensions are reduced using Principal Component Analysis (PCA). A variance in the visual appearance of the products can be seen when we travel along the axes in these plots. For example in the Men dataset, the x-axis is populated with watches while the y-axis contains clothing related items like t-shirts. A similar trend can be observed for the Women and Shoes dataset, however, the plot for the Jewelry dataset looks noisy. Next, we visualize groups of products which have been purchased by specific users in Figure 2(b). Here we select 4 random users and plot all the products purchased by them. Dimensions are reduced using t-SNE with the perplexity parameter set to 30. Products corresponding to a specific user are marked by the colour of the frame around them. In case of co-purchased products, the frame colour is chosen randomly. It can be seen that products purchased by the same user have been clustered together. These plots show that the leaned embeddings can simultaneously capture the visual properties and the purchase history of a product.

Linear relationships. Skip-Gram word embeddings are known to follow a linear algebraic structure where embeddings display associative properties (shown by the famous example vector(”King”) - vector(”Man”) + vector(”Woman”)$\approx$vector(”Queen”) (Mikolov et al., 2013a)). We attempt to exploit a similar structure in the VASG embeddings for recommending products. Consider a pair: $\{(u1,p1),p1\in\mathcal{P}\textsubscript{u1}\}$ and a user $u2;u2\neq u1$, keeping $(u1,p1)$ and $u2$ fixed, if we can a find a product $p\in\mathcal{P}$ such that

Then we say that $p$ is a user specific recommendation generated for the user $u2$ having a query product $p1$. To test this methodology, we sample 10,000 pairs of $u1,u2$ and find the nearest neighbour of the point $\mathcal{X}\textsubscript{p1}+\mathcal{X}\textsubscript{u2}-\mathcal{X}\textsubscript{u1}$, using cosine distance, from the set $\mathcal{P}$ and treat them as recommendations generated by VASG. We obtained a precision@5 of 89.7% in the Men dataset. Figure 3. shows some of the recommendations generated by this methodology. Approximately speaking, here a product $p$ is recommended for user $u2$ when the similarity between $p,p1$ and between $p,u2$ is high and when similarity between $u1,p$ is low.
Embedding similarity distribution. This analysis is inspired by (Ying et al., 2018). Effectiveness of the learned embeddings can be judged by the distribution of distances between random pairs of embeddings. A wide distribution indicates that the embedding space has enough “resolution” to capture relevance of different product pairs. Figure 4 plots the distribution of cosine similarities between pairs of products from the Men dataset. The distribution coming from the VASG embeddings is compared against the distribution computed from product image features. VASG embeddings show a wider distribution hence proving their superiority over using raw image features. The kurtosis of this distribution for VASG embeddings is 0.87, compared to 3.67 for image features.

Influence of Embedding size. We study the variation of AUC scores with the embedding dimension size $D$. Figure 4 shows AUC scores for warm-start products when evaluating the performance of making personalized recommendations. The performance improves with increase in dimension size, however the rate of improvement seems to slow down after $D=70$.
Influence of noise addition. Autoencoders are known to suffer from over fitting. To overcome this problem, we add a small amount of Gaussian noise to the product image features while learning the mapping $\mathcal{M}(.)$. We find that noise addition helps in improving the performance of cold-start products for making personalized recommendations by an average margin of 1.8% in AUC score.
Influence of Skip-Gram window Size. The window size determines the neighbourhood size of a HIN node. For example a window size of 5 restricts the neighborhood to second degree neighbours. A smaller context window exposes the embeddings to local HIN structures which is suited for making personalized recommendations. We experiment with window size [3,5,7,9] and find that a window size of 7 gives the best results. Smaller window sizes lead to a faster convergence, but the embeddings over-fit in a small neighbourhood whereas larger window sizes make the training corpus unnecessarily large.
Influence of the auxiliary decoder. We study how VASG’s performance is affected by removing the auxiliary decoder. This reduces the proposed architecture to the general Skip-Gram model. We evaluate the performance of making personalized recommendations in the set $\mathcal{T}\textsubscript{warm}$. The performance drops by an average of 2.4% on the Women, Men and Shoes dataset while the performance remains almost consistent for the Jewelry dataset. This indicates that addition of visual features improves the convential Skip-Gram embeddings.

In this paper, we proposed a novel embedding learning architecture, VASG, which incorporates visual features into the Skip-Gram model. This model has two different loss functions depending on whether a user or a product is observed during training. VASG embeddings can be used for making personalized recommendations using nearest neighbour searches in the learned latent space. In addition, this framework enables us to find embeddings for cold start products which have never been observed during training. The embeddings are trained on real world datasets and extensive analysis brings forward several interesting properties captured by them.

There are multiple potential directions in which this work can be extended. Use of effective negative samples while training can enable us to learn a more refined latent space. Adversarial training can be used to reconstruct product images using Generative Adversarial Networks conditioned on the Skip-Gram embeddings. Additional auxiliary tasks which bring information from other modalities can be added to central Skip-Gram architecture. VASG embeddings can be also be incorporated into frameworks like SVDFeature for predicting ratings.