Attribute-driven Disentangled Representation Learning for Multimodal Recommendation

Given a set of users $\mathcal{U}\in\{u\}$ and items $\mathcal{I}\in\{i\}$, we utilize three types of information to learn user and item representations: (1) A user-item interaction matrix $\bm{R}$, where each entry $r_{ui}\in\bm{R}$ represents the implicit feedback (e.g., clicks, likes or purchases) of user $u$ to item $i$. $r_{ui}=1$ indicates that user $u$ has interacted with item $i$ and $r_{ui}=0$ indicates that there is no interaction between user $u$ and item $i$ in the observed data; (2) Multimodal features, which mainly consist of two types of features associated with items, namely reviews and images; and (3) Attribute information, consisting of $K$ attributes and their associated attribute values of items, used as supervision for disentangled representation learning. We aim to learn robust representations of users and items, thereby predicting a user’s preference for items they have not yet interacted with.

Similar to previous work (Wei et al., 2020; Liu et al., 2023a), each user $u$ and item $i$ are assigned with a unique ID and respectively represented by a vector $\bm{v}_{u}\in\mathbb{R}^{d}$ and $\bm{v}_{i}\in\mathbb{R}^{d}$, which are randomly initialized in our model. For the review and image information, we use the BERT (Devlin et al., 2019) and the ViT (Dosovitskiy et al., 2021) to extract the raw textual features $\bm{e}_{t}\in\mathbb{R}^{d_{0}}$ and visual features $\bm{e}_{v}\in\mathbb{R}^{d_{0}}$, respectively. To tailor the recommendation-oriented features, we adopt two non-linear transformations to cast $\bm{e}_{t}$ and $\bm{e}_{v}$ into the same feature space:

where $\bm{W}_{t}$, $\bm{W}_{v}\in\mathbb{R}^{d\times d_{0}}$ and $\bm{b}_{t}$, $\bm{b}_{v}\in\mathbb{R}^{d}$ denote the weight matrices and bias vectors for textual and visual modality, respectively. $\sigma(\cdot)$ is the activation function.

Following the disentangled representation learning process in previous work (Wang et al., 2020; Liu et al., 2023a), we split the feature vector into $K$ chunks, each corresponding to a specific item attribute (e.g., price, brand). For simplicity, we equally split each modality’s representation into $K$ continuous chunks. Take item ID embedding as an example:

where $\bm{v}_{i}^{k}\in\mathbb{R}^{\frac{d}{K}}$ is the item ID embedding corresponding to the $k$-th attribute. Analogously, $\bm{v}_{t}=(\bm{v}_{t}^{1},\bm{v}_{t}^{2},\cdots,\bm{v}_{t}^{K})$, $\bm{v}_{v}=(\bm{v}_{v}^{1},\bm{v}_{v}^{2},\cdots,\bm{v}_{v}^{K})$ and $\bm{v}_{u}=(\bm{v}_{u}^{1},\bm{v}_{u}^{2},\cdots,\bm{v}_{u}^{K})$ are defined as the embeddings of textual feature, visual feature and user ID embedding, respectively.

Our work not only aims to recommend accurate items to users but also advocates for attribute-driven disentanglement in multimodal recommender systems to enhance the interpretability and controllability of recommendations. Specifically, unlike the existing method that disentangles the latent factors in an unsupervised manner, we leverage the semantic labels of item attributes to learn attribute-specific sub-spaces for each attribute to obtain disentangled representations. In this paper, the vector of each modality is composed of $K$ chunks, with the assumption that each chunk is associated with an attribute (such as price or brand). The achievement of attribute-driven disentangling requires two necessary conditions. Firstly, each chunk should have a clear attribute reference, and chunks associated with different attributes should be distinguishable. For example, chunk $j$ is associated with the price, while chunk $k$ represents the vector of the brand. Secondly, each chunk of a specific attribute should accurately capture the specific attribute value. For example, chunk $j$ should reflect the price level (i.e., expensive or cheap) of the product.

To obtain robust and independent representations for each attribute factor in multimodal features, AD-DRL enables disentangling within each modality feature and ensures consistency of representations across modalities. Next, we detail the intra-modality disentanglement and inter-modality disentanglement.

Intra-Modality Disentanglement. After obtaining the feature vectors of each modality, we split these vectors into several chunks. Nevertheless, different attribute factors are still entangled within the chunks. In other words, a chunk may contain both price- and brand-related features. Therefore, in order to disentangle attribute factors in each modality feature, we employ an attribute classifier to encourage each chunk to predict the corresponding attribute, as shown in Figure 1a. Taking the $k$-th chunk $\bm{v}_{t}^{k}$ of item textual embedding as an example,

where $\bm{W}_{intra,t}\in\mathbb{R}^{K\times\frac{d}{K}}$ and $\bm{b}_{intra,t}\in\mathbb{R}^{K}$ are the weight matrix and bias vector of the classifier, respectively. $\bm{z}^{k}$ is the predicted logits and $z_{n}^{k}\in\bm{z}^{k}$. $\tilde{z}_{n}^{k}$ denotes the ground truth (attribute label) of chunk $\bm{v}_{t}^{k}$. For all the chunks in textual embedding $\bm{v}_{t}$, we have the following loss for all attribute factors in textual features:

Similarly, we can define the losses $l_{intra,u}$, $l_{intra,i}$ and $l_{intra,v}$ to encourage the features of each attribute factor to be concentrated in the corresponding chunk of user ID, item ID and visual feature, respectively. Finally, the total loss for the intra-modality disentanglement is formulated as:

Inter-Modality Disentanglement. In addition to separately disentangling each modality, a serious challenge in disentangling representation learning for multi-modal features is handling the inter-relationship between the factors disentangled from multiple modalities. Intuitively, the chunks of the same attribute factor from different modality features should be consistent. For example, for the brand attribute, an item should have the same brand information in both the visual and textual features. In other words, chunks that share the same attribute across different modalities should be highly similar, while chunks that represent different attributes across modalities should be dissimilar.

To further achieve robust representation in multimodal recommendations, we disentangle attribute factors in different modality features by ensuring consistency of the same attribute across modalities. Inspired by this, we design a cross-modal contrastive loss as shown in Figure 1b. Specifically, for any two modality representations, such as $\bm{v}_{t}$ and $\bm{v}_{v}$, we take two chunks of the same attribute factor (i.e., $(\bm{v}_{t}^{k},\bm{v}_{v}^{k})$) as positive pairs, and two chunks corresponding to different attributes (i.e., $(\bm{v}_{t}^{k},\bm{v}_{v}^{n\neq k})$, $(\bm{v}_{t}^{n\neq k},\bm{v}_{v}^{k})$) as negative pairs. After that, the cross-modal contrastive loss between textual and visual features is defined as follows:

where $\cdot$ represents the dot product and $\tau\in\mathbb{R}^{+}$ is a scalar temperature parameter. By applying this contrastive learning constraint to any two out of the three modalities, we can achieve disentanglement and alignment across features of various modalities:

Each attribute $k$ is associated with a list of possible attribute values $(\tilde{y}^{k}_{1},\tilde{y}^{k}_{2},\cdots,\tilde{y}^{k}_{A_{k}})$, where $A_{K}$ is the total number of possible values for that attribute. For example, the attribute value of popularity can be stratified into five distinct levels: Super Popular, Popular, Moderate, Emerging, and Unknown. The representation disentanglement could benefit from the attribute values by exploiting their relationships of the same factor. Therefore, to learn more robust representations, we encourage disentangled representations based on attributes to predict specific attribute values of the item.

It is obvious that any single modality may not contain sufficient information for a particular attribute. For example, images may more intuitively reflect attributes such as color and brand of an item, but may not directly reflect attributes such as price or popularity. Therefore, in order to comprehensively and accurately depict the attribute values of the item, we first integrate the features from all modalities together. To achieve this, we apply a multimodal attention mechanism to measure the emphasis of different modalities on different attributes. Specifically, for the $k$-th attribute, the attention weights assigned to different modalities are estimated by a two-layer neural network:

where $\bm{W}_{a1}\in\mathbb{R}^{3\times\frac{d}{K}}$ and $\bm{W}_{a2}\in\mathbb{R}^{3\times 3}$ are the weight matrices corresponding to the first and second layers of the neural network, respectively. $\bm{b}_{a}$ denotes the bias vector and $\tanh$ is the activation function. $Softmax$ is adopted to normalize $\hat{\bm{a}}^{k}$ to a probability distribution.

Thereafter, we obtain the final representation of the $k$-th attribute factor for the item as follows:

where $a_{i}^{k}$, $a_{t}^{k}$, $a_{v}^{k}\in\bm{a}^{k}$ are the attention weights of item ID embedding, textual feature and visual feature, respectively.

Similar to the disentangling at the intra-modality disentanglement in Section 3.2.1, we aim to achieve disentangled representation learning at the low level through attribute value prediction via a classification layer (as shown in Figure 1c):

where $\bm{W}_{k}\in\mathbb{R}^{K\times\frac{d}{K}}$ and $\bm{b}_{k}\in\mathbb{R}^{K}$ are the weight matrices and bias vector of the classifier, respectively. We supervise the training of each attribute subspace via independent attribute classification tasks defined in the form of a cross-entropy loss.

So far we have discussed how to obtain attribute-driven disentangled representations $\bm{v}_{u}=(\bm{v}_{u}^{1},\bm{v}_{u}^{2},\cdots,\bm{v}_{u}^{K})$ and $\bm{v}_{y}=(\bm{v}_{y}^{1},\bm{v}_{y}^{2},\cdots,\bm{v}_{y}^{K})$ for the user $u$ and item $i$, respectively. By assigning each disentangled representation with an attribute as described above, we are able to predict users’ preferences at the attribute level. This enhances the interpretability and controllability of our model. Specifically, to estimate a user $u$’s preference for an item $i$, it is crucial to consider her preference for each attribute of the item. To achieve this, we first compute the user’s preference score for each attribute of the item, and subsequently, integrate the scores of each attribute to estimate the user’s overall preference for the item:

where $\cdot$ symbolizes the dot product and $\sigma(\cdot)$ denotes the activation function. In this paper, we use a softplus function to ensure the resultant score $s_{u,i,k}$ is positive. $s_{u,i,k}$ denotes the user $u$’s preference score for the attribute $k$ of the item $i$.

Based on the predicted preference score above, we recommend a list of top-$n$ ranked items that match the target user’s preferences. The Bayesian Personalized Ranking (BPR) loss function (Rendle et al., 2009) is employed to optimize the model parameters $\bm{\Theta}$,

where $\lambda$ is the coefficient controlling $L_{2}$ regularization; $\mathcal{D}$ denotes the training set; $i_{+}$ and $i_{-}$ are the observed and unobserved items in the interaction records of user $u$, respectively. Overall, the total loss of AD-DRL is formulated as,

where $\alpha$, $\beta$ and $\gamma$ are the hyperparameters to control the weight of three disentanglement modules.

We use the widely adopted real-world recommendation dataset, the Amazon review dataset¹¹1http://jmcauley.ucsd.edu/data/amazon. (McAuley and Leskovec, 2013), for evaluation in our experiments. Apart from user-item interaction data, this dataset also includes multimodal information (i.e., reviews and images) and various attributes (i.e., price, brand, etc.) of the items on 24 product categories. Three product categories from this dataset are used in the evaluation: Baby, Toys Games and Sports. Following (Liu et al., 2023a), for all datasets, unpopular items and inactive users are filtered out to ensure that all the items and users have at least 5 interaction records. The basic statistics of the three datasets are shown in Table 1.

In our setting, in addition to the user-item interaction data and multimodal information of items, multiple attributes and their associated attribute values of items are required to guide the disentangled representation learning process. Specifically, we adopt four typical attributes (i.e., price, popularity ²²2Popularity is calculated by counting the number of times each item is purchased., brand and category), and the attribute values for each attribute are compiled based on the metadata provided by the Amazon dataset: for brand and category values, we directly used the values provided by Amazon; for price and popularity, following the methods used by CoHHN (Zhang et al., 2022), we discretize them into five levels according to their numerical values. Table 1 shows the specific statistical information for each attribute in our experiments. It is worth noting that our method can include a wide range of attributes that objectively reflect the character of an item, beyond the four attributes mentioned.

We compared our method with the state-of-the-art methods, including both the CF-based methods (i.e., NeuMF (He et al., 2017), NGCF (Wang et al., 2019) and DGCF (Wang et al., 2020)) and Multimodal CF-based methods (JRL (Zhang et al., 2017b), MMGCN (Wei et al., 2019), MAML (Liu et al., 2019), GRCN (Wei et al., 2020), DMRL (Liu et al., 2023a) and BM3 (Zhou et al., 2023)). Each type of method includes a disentangled representation learning model (i.e., DGCF and DMRL) for comparison. In addition to the methods mentioned above, we also created a variant of our model, denoted as $\text{AD-DRL}_{ID}$, which excludes the utilization of multimodal information, thus facilitating a fair comparison with CF-based models.

For each dataset, we randomly split the interactions from each user into training and testing sets with an $8:2$ ratio. From the training set, 10% of interactions are randomly chosen as a validation set for tuning hyperparameters. We evaluate performance on the top-$n$ recommendation task, aiming to recommend the top-$n$ items, using Recall@$n$ and NDCG@$n$ as metrics for accuracy, with $n$ defaulting to 20.

The Pytorch toolkit (Paszke et al., 2019) is utilized to implement our models. To ensure fairness, all methods are optimized using the Adam optimizer (Kingma and Ba, 2015), with a default learning rate of 0.0001 and batch size of 1024. We fixed the embedding size of each factor to 32 for AD-DRL and its variants on all datasets. The Xavier initializer (Glorot and Bengio, 2010) is used to initialize the model parameters. The $\alpha$, $\beta$, $\gamma$ and $L_{2}$ regularization coefficient are searched in $\{1e^{-3},5e^{-3},1e^{-2},\cdots,5e^{+0},1e^{+1}\}$. The number of negative examples is searched in $\{2,4,8\}$. Besides, model parameters are saved every five epochs. Early stopping strategy (Wang et al., 2019) is performed, i.e., premature stopping if Recall@20 does not increase for 50 successive epochs. Our codes and datasets are released to facilitate the replication of our experiments ³³3https://github.com/SDLZY/AD-DRL..

Figure 2 displays the disentangled vectors processed by the high-level attribute-driven disentanglement module for each modality feature, where dots of the same color denote the vectors corresponding to the same attribute. It can be observed that our model effectively distinguishes representation vectors corresponding to different attributes for both users and items across various modalities, demonstrating the effectiveness of our model in disentangling at the attribute level. Such disentangled representations can better capture user preferences and item characteristics toward different attributes.

To verify the effectiveness of low-level (attribute value-level) disentanglement, we visualize the representations of each attribute factor (i.e., $\bm{v}_{y}^{k}$ in Equation 9) in Figure 3. The dots of different colors represent different attribute values⁴⁴4For brand and category, since there are too many attribute values in the dataset, making it difficult to display all of them in Figure 3. Therefore, for these two attributes, we only selected the top 5 attribute values with the most corresponding items in the dataset for demonstration.. As observed, the representations of different attribute values learned from AD-DRL are well separated and the representations of the same attribute value are concentrated. These results demonstrate that AD-DRL can achieve finer-grained disentanglement at the attribute value level, allowing AD-DRL to capture better user preferences.