Enhancing E-Commerce Recommendation using Pre-Trained Language Model and Fine-Tuning
^††thanks: This work was done in 2021.

With the rapid development in web technology, Internet users have been provided with tremendous amount of information, which impedes the selection of relevant information to the user, resulting in protracted decision-making process and reduced decision quality. To address this problem of information overload, solid efforts have been made by researchers in various domains to develop recommendation systems that are able to generate curated information based on user preferences, ranging from online advertisement[Chaudhuri et al., 2016], music selection[Kowald et al., 2020], move rating[Nguyen and Zhu, 2013], to product recommendation[Agarwal et al., 2018]. By exploiting user-item (UI) relations, recommender systems recommender systems hold the promise of streamlining data and provide personalization, and have been a key determining factor for driving a nuanced customer experience[Ricci et al., 2015]. As a result, they have been widely adopted by online retailers and web service providers to perfect their products and acquire profit.

E-commerce has gained increasing importance and popularity among customers. Although recommendation systems are now intrinsic to many industries, there are some unique challenges that thwart the widespread adoption in the field of E-commerce. In most of the cases, potential customers do not have a crisp understanding of what they want to buy. It is hard for them to accurately express their need with limited taxonomy of attributes with the internal search engine.[Agarwal et al., 2018] As a result, they get discouraged fairly easily in the process of navigating through a plethora of options. Undoubtedly, the capability of recommender system implemented by those online retailers to capture the intrinsic item characteristics and user preferences has becoming one of the biggest differentiators against their competitors and is critical to ensuring the success of a company[Ricci et al., 2015].

There is largely a dichotomy of recommendation algorithms. Content-based algorithms build user specific and item specific profiles, usually in the form of a vector. The user profile describes the contents that he or she likes, and the item profile describes the contents that it contains. To create those profiles, a good amount of auxiliary information of users and items are required. Then a similarity metric, typically a dot product, is used to determined user preferences. Collaborative filtering approach analyzes past user behavior and finds similar users and items based on the history. The core assumption here is that the users who have agreed in the past tend to also agree in the future. People have also been using deep learning (DL) techniques to further improve the performance of those recommender models, as DL provides better modelling power, flexibility, and feature representation learning. Covington et al.[Covington et al., 2016] presented a deep neural network based recommender for video recommendation on YouTube. Cheng et al. [Cheng et al., 2016] proposed a wide & deep model for generic recommender systems with sparse inputs. Shumpei et al. [Okura, 2017] showcased a recurrent neural network (RNN) based news recommender system for Yahoo News. All of these deep learning based models have proven to have significant improvement over traditional models. Hybrid model refers to recommender system that integrates two or more types of recommendation methods.

While most of the existing recommendation algorithms are powerful in modeling user-item relationships, they have major limitations regarding the utilization of text information. Specifically, they cannot effectively incorporate textual information in e-commerce datasets, which are in abundance and might potentially be very informative. Without preparations, those recommendation algorithms also suffer acute relevance problems due to lack of negative feedback in the context of online retail[Jiang et al., 2020]. The problem regarding which recommender algorithms to use and how to effectively use those algorithms, while heatedly debated and investigated, remains challenging and unsolved.

The recent emergence and development of BERT (Bidirectional Encoder Representations from Transformers) and its variants in NLP field has drastically increased the modeling power of machine learning models to understand textual information[Devlin et al., 2019]. By leveraging the power of pretraining contextual representation on large scale text datasets and unsupervised training objectives, BERT has been able to achieve state-of-art performances in a board range of natural language processing tasks.

However, in the field of e-commerce, people could not exploit the full advantage of BERT algorithms due to the following reasons: BERT cannot always accurately understand the frequently appearing domain specific phrases because they are not included in the training objective; in the context of e-commerce, textual information such as item label or item description typically has very limited length compared with other data sources such as movie reviews or web documents, on which BERT is hard to make inference without additional domain specific pretraining and is very susceptible to noises in the dataset[Zhang et al., 2020].

It remains unclear that how the benefit of a unified language representation model would help the recommendation models to better understand and capture the underlying rules that affects people’s decision making, and how different schemes of further pretraining would influence the final recommendation results. In our paper, we address this question for a high performing pre-trained language model, RoBERTa[Liu et al., 2019], and investigate the effectiveness of various approaches to incorporate rich text features into traditional recommender system algorithms on an online retail data set. We consider two types of recommendation algorithms - content-based recommender algorithm (matrix factorization, abbreviated as MF) and machine learning based algorithm (XGBoost and decision tree). We evaluate the performance increase gained through the incorporation of sentence embeddings against baseline algorithms, and we show that the introduction of RoBERTa consistently increases model performance regardless of which recommender algorithm is used. Additionally, the effect of pretraining BERT with transformer-based denoising auto-encoder (TSDAE) and masked language model(MLM) is also investigated. For MLM, in accordance to the RoBERTa paper, we use dynamic masking rather than static masking[Liu et al., 2019]. Inspired by this process, We further propose a recommendation scheme that we consider to be the most effective for MF and XGBoost respectively. Our experiment shows that recommendation models trained under this scheme outperforms their vanilla counterparts even with highly sparse user-item interaction matrices.

In summary, our contribution includes:

• •

  a comprehensive study on incorporating rich textual information in the domain of e-commerce to improve performance of recommender algorithms;

• •

  a thorough evaluation and visualization of recommendation results made by vanilla recommendation algorithms and BERT enhanced recommendation models;

• •

  an investigation into the effectiveness of domain-adaptive pretraining.

Three different text datasets are used in this paper. We use Online Retail dataset[Chen et al., 2015] as our main target of study. It focuses on a narrow domain of e-commerce, as this British Online retailer sells a limited number of things during the period when this dataset is collected. We use Amazon dataset as a dataset that is similar but covers a broader range of things, as they are both e-commerce datasets. The openwebtext dataset[Aaron and Vanya, 2019] is considered as a close analogy to what text datasets are used for RoBERTa training, and it is only used for cross reference when computing vocabulary similarity. Only part of it is sampled due to its much larger size compared to the other two datasets.

The online retail dataset consists of transactions occurring between 01/12/2010 and 09/12/2011 for a UK-based and registered non-store online retail. The company mainly sells unique all-occasion gifts. Many customers of the company are wholesalers. Each observation has eight covariates, which are invoice ID, stock code, item description, quantity, date, unit price, customer ID, and country. The invoice ID is a 6-digit number uniquely assigned by the retail to be able to identify if different items belong to the same user. Canceled transactions is indicated by invoice number which starts with letter ’C’. The stock code is a 5-digit integral number uniquely assigned to each distinct product, while the customer ID is a 5-digit user specific integer. The item description is a short sentence describing what the item is. Date is the day and time when each transaction was generated. The last one is country, showing the name of the country where each customer resides. There are 37 countries in total, ranging from North America to EU, with most customers from UK and France. The first five lines of the dataset is shown in Figure 1.

There are some special entries and interesting characteristics in the dataset that are worth mentioning:

• •

  Transactions with negative quantity

  • –

    The magnitude of those negative values match with previous purchase records in the dataset.

  • –

    In the context of online retail, it means some items got returned.

• •

  Transactions with zero or negative price.

  • –

    Transactions with zero price does not bring useful information to analysis, moreover mainly there are not any description for such transactions.

  • –

    Transactions with negative price. From the description we could see that these transactions were probably made as corrections.

• •

  Long-tailed data distribution with extreme outliners

  • –

    Right-skewness of quantity and unit price. As shown in Figure 2, even in log scale, the data is still right-skewed. In fact, 50% of the users only purchase 1-3 items at a time, while some wholesaler does big purchases with more than 500 of the same items. In addition, most items sold in the dataset are cheap, with rare cases of high unit prices. As a result, those data are extremely right-skewed.

  • –

    Outliners. From Figure 3, most of items are common, daily products which have a normal price. However, some items in this dataset are very expensive, and that might influence the amount of purchase significantly. Also, standard deviation for quantity is 218.08, which is extremely huge. This is caused by some rare and extreme events. For example, a big purchase of 80955 items happened during the period when the dataset is recorded, but unfortunately this order is eventually cancelled and thus is not taken as part of our data.
    

There are 1336 of transactions with negative quantity value among non-cancelled transactions, and 1336 transactions with zero price and negative quantity. This corresponds to 1.71% and 0.46% of the total number of observation entries in the dataset, which is insignificant and less likely to cause loss of information if those problematic entries are directly filtered out. There are 541909 entries in this online retail dataset, and there are 406829 non-null customer IDs and 540455 item descriptions, meaning 24.9% of customer IDs and 0.27% of item descriptions are null object and thus cannot be used. After filtering out all the null entries and return cases, there are 397924 observations. Those records correspond to 4339 distinct users and 3665 unique items. This gives a user-item matrix with a sparsity of 97.5%.

To reduce the influence of outliners and uneven distribution on the performance of our models, there are normally two methods. We can standardize the features by subtracting mean and dividing the result by standard deviation, but that would result in negative values in features and potentially negative output values in matrix factorization. Here we choose the second method, evening out the distribution by creating a histogram as shown by Figure 4 and assign labels to each quantity range that approximately contains the same amount of products. Due to the discrete nature of this method, each label would not have the same amount of items, but as shown in Figure 4, the distribution is much better than the right-skewed one shown in Figure 2.

We first build baseline models without using the text information of production description. Two main recommendation models are used in this paper:

• •

  Matrix Factorization Recommender

• •

  XGBoost Factorization Recommender

And for each approach, we have done investigation on the performance of three variants of those recommendation models. The details are listed in Table I.

Two main recommender models are used in the paper, Matrix factorization and XGBoost. Various variant approaches are taken to explore the key to better recommendation performance. The detailed list is shown in Table I.

We first establish baseline performance using vanilla Matrix Factorization and XGBoost models. Based on these, we add additional tweaks to see if there is any improvement and to better understand how to apply PLMs to those recommender systems. For visualizing and evaluating PLMs, we pass the sentences in our text datasets into those trained models and generate sentence embeddings of a fixed dimension of 728. Those high dimensional sentence embeddings are then passed through dimension reducing networks, where their dimension is reduced to either 10d or 2d depending on the settings. Two choices of dimension reduction methods are available, T-SNE and PCA. PCA provides more consistent result and faster running time[Garber and Hazan, 2015], while T-SNE provides better clustering result while having slower running speed. T-SNE also does not work well on high dimensional vectors. In this paper, the embedding space is 2 dimensional, visualized through PCA and results are also generated using PCA. T-SNE are tested, and following [Van Der Maaten and Hinton, 2008], we use PCA to reduce the dimension to 10d before we use T-SNE to further reduce it to 2d. However, T-SNE does not yield better result and the analysis regarding its performance is not relevant to our study in this paper.

We use distilled RoBERTa as our baseline pretrained language model, with optional fine-tuning following two pretraining schemes introduced by Gururangan et al. [Gururangan et al., 2020], domain adaptive pretraining (DAPT) and task adaptive pretraining (TAPT). Our domain specific pretraining follows the settings prescribed for training RoBERTa. Due to the memory constraints, we did not choose to further pretrain RoBERTa model on openwebtext. Instead, we use RoBERTa as a baseline model, and investigate the effect of DAPT and TAPT using amazon description dataset and retail description dataset respectively. Two different kinds of pretraining method are used, Transformer-based Sequential Denoising Auto-Encoder (TSDAE) and masked language model (MLM). Studies have show that TSDAE is a strong pre-training method for learning sentence embeddings, significantly outperforming other approaches like MLM. It also provides better generalization and robustness for heterogeneous domains than the baseline RoBERTa model where domain knowledge is required. The pre-training is done in different orders to compare the effectiveness of different schemes.

First, we investigate the effectiveness of DAPT and TAPT individually. Amazon sentences are divided into 512 batches of size 2000, with maximum characters allowed in each sentence set to 50. Then those batches are fed into two different pretraining frameworks, TSDAE and MLM respectively. Prediction accuracy, precision, recall and F1 score are used as evaluation metrics and model performance is evaluated every 100 batch and at the end. The results are listed in Table II and Table III. The performance of XGB recommender increases with training for all evaluation metrics and the speed becomes slower with the increase of batch number. This can also be supported by the training loss graph shown in Figure 9. Training loss drops fast at first 100 batches and then becomes stable after that.

Then we explore different orders of combining TSDAE and MLM pretraining, with different combinations of DAPT and TAPT, in comparison to the baseline performance we just established above. The result is listed in Table IV.

We also study the influence of various features used in generating recommendations by XGBoost model. Random Forest algorithm is used as an optional feature selection method that filters out features with importance less than 0.04 in an attempt to reduce the chance of XGBoost model overfitting on less useful features. Different features are added in a paralleled manner except those with a * after the feature name. The details are listed in Table V.

Since the number of purchased items are labeled as 0, 1, 2, 3, 4, we also want to check the performance of the model on each label, namely, on each range of purchase numbers. The result is shown in Table VI.

As shown in Table VII, recall @5 is quite low in comparison to precision @5. This is understandable because of the characteristics of the retail dataset used in this paper. Lots of the customers of this online retailer are also retailers from other industry, and they usually purchase a lot of items of different types at once. As a fact, there are costumers who bought more than 40 items. Even if the 5 recommendations made by the model are all correct, that would only give us a 5/40 = 12.5% recall @5. Therefore, recall @5 is not a very good metric here to evaluate the performance of the matrix factorization model as the number of recommendations made here is typically much lower than the amount of items that got purchased by the customers. A better metric to look at here would be precision based, namely, f1 score or MAP(mean average precision).

We speculate that there is no strong interrelationship between different customers who have done purchases at this online retail. Customers tend to buy items that belong to their industry. For example, furniture retailers like to buy crews and other related components, and normally in a large number. That does not relate well with another user, say an individual buyer. The customer_id feature whose introduction leads to a boost in model performance supports this speculation.

Two user examples on which the model does not recommend items accurately are chosen here for visualization to better understand what happened. The examples are visualized through PCA and shown in Figure 10. We evaluate them based on two criteria, correctness on the predicted cluster label and predicted stock code of the item. For the left example, the details of which are shown in Table IX, only 1 out of 10 cluster label is predicted correctly, and none of the stock codes. From Figure 10, we observe that predicted items are mostly clustered around cluster 7 and cluster 5, while the ground truth items spread around them. This means that the recommender has picked up some patterns, and tries to focus on some dimensions of the latent space that it considers to be important. However, due to lack of modeling power, it could not fully exploit the rich information contained in the sentence embeddings. Similar pattern can also be observed from the right figure in Figure 10. As a reference, Figure 11 show the difference between the distributions of embedding vectors visualized in 2d through PCA (left) and T-SNE (right) respectively. The result obtained by PCA is more spread out while the result obtained by T-SNE is more tightly clustered. However, due to the high dimensional nature of sentence embeddings, it remains difficult to accurately interpret the result in 2D, which is subject to visualization methods.

Experiments have shown that MF-based recommenders do not work well with sentence embeddings when they are taken directly as the recommendation algorithms. Attempts are made to combine sentence embeddings generated by PLMs as item representations with item features generated by MF model. The approaches taken include:

• •

  combining top ranked recommendation results from MF recommender and PLM recommender that ranks items based on cosine similarity of their sentence embeddings.

• •

  taking each entry of the UI matrix as a weighted sum of the dot product of user vector and item vector from MF model and the similarity score from the PLM

• •

  using sentence embeddings of items purchased by other users who are considered to be similar user by MF model as auxiliary information to facilitate the MF recommender

However, none of the above-mentioned approaches lead to any observable improvement in recommendation performance. This shows that, due to the high dimensional nature of sentence embeddings and the characteristics of this Online Retail Dataset, MF is not the ideal model to be used directly as the recommendation algorithm. A model that better leverages the rich information contained within those sentence embeddings is needed. However, as suggested by LadaBERT[Mao et al., 2021], when matrix factorization is used as a model compression technique alongside with weight pruning and knowledge distillation, BERT model can be made less memory and data intensive. It also helps the model to perform better with significantly fewer training overheads. It is not covered in this paper and can be an interesting future direction.

We find that the introduction of transformer based pretrained language embedding model can boost the performance of traditional recommendation algorithms, especially on XGBoost recommender. Through carefully designed fune-tuning on RoBERTa model using TSDAE and MLM, the recommendation accuracy can be further improved. Empirical data shows that MLM followed by TSDAE achieves a better performance increase, as we speculate that MLM pretraining provides a rudimentary domain specific knowledge to the model and TSDAE further increases generalization and robustness across heterogeneous domains.

The vocabulary similarity between amazon description dataset and retail description dataset is comparably low, and this might compromise the effectiveness of domain specific pretraining. Also due to GPU memory constraints, the datasets used in this paper are small scaled in comparison to language corpus that are used to train RoBERTa. A dataset that is larger in scale and more structurally similar to the online retail dataset can be used in the future to investigate the full potential of this pretraining method.

There are more state-of-art deep learning based recommendation algorithms that could be used as the recommendation model. In this paper we choose to focus on an investigation on the effectiveness of the introduction of pre-trained language models into recommender systems using matrix factorization algorithm or XGBoost algorithm. Exploring the benefits of PLMs on deep learning based recommender systems will be an interesting future direction.

As suggested and experimented by Zhang et al.[Zhang et al., 2020], Adaptive Hybrid Masking (AHM) extends MLM by introducing a new masking strategy. Specifically, it sets two different modes, i.e., word masking mode and phrase masking mode. The former randomly masks separate words while the latter masks domain phrases. Moreover, it can adaptively switch between the two modes based on feedback losses, enabling the model to capture word-level and phrase-level knowledge progressively. Compared with MLM, it has a higher potential to increase the model performance if it is chosen to be the fine-tuning method. Additionally, like AHM, new advanced pre-training methods can be used instead of the two that are used in this paper in the future.

Recent empirical improvements due to the incorporation of PLMs and thoroughly designed fine-tuning strategies into traditional recommender systems have been shown that rich, staged, and unsupervised pre-training is a vital and indispensable part of boosting model performance in the newly thriving field of e-commerce. Our major contribution is to investigate various methods of combining PLMs and fine-tuning with MF and XGBoost recommendation algorithms, and propose a fine-tuning scheme that allows the traditional model to outperform its vanilla counterpart respectively.