An Effective Way for Cross-Market Recommendation with Hybrid Pre-Ranking and Ranking Models

In pre-ranking stage, we use several different methods or models to get user/item representations and user-to-item similarity scores which represent that how much the user is interested in the given item. Based on pre-rank scoring, hundreds of diverse features are generated for the next stage of model training, which boost our final model significantly.

To be specific, the pre-ranking models can be classified into 3 categories, which are memory-based, embedding-based and graph-based models. For memory-based models, we use some traditional collaborative filtering (CF) models like ItemCF (Linden et al., 2003), UserCF (Resnick et al., 1994), Swing (Yang et al., 2020), Loglikelihood Ratio (LLR) (Dunning, 1993), Bi-Graph (Zhou et al., 2007) to get user-to-item similarity scores. These memory-based models above score 0.59-0.62 (NDCG@10, the same below) on the leaderboard. Especially for ItemCF, which takes the least time to train and performs the best among all of the memory-based models above.

As for embedding-based models, Word2Vec, Node2Vec (Grover and Leskovec, 2016) (both in DFS and BFS ways) and NCF (He et al., 2017) are used to generate user/item embeddings and user-to-item similarity scores. NCF perform the best among embedding-based models with the score of 0.61-0.62 on the leaderboard. Scores of the rest models are ranging from 0.35 to 0.46, which do not seem good enough comparing others. These models are probably not strong solo players, but also contributing to the final ranking models in some degree based on our experiment.

Node2Vec is a simple but efficient embedding-based model. Unlike Word2Vec and DeepWalk, Node2Vec uses a biased random walk procedure to efficiently explore diverse neighborhoods in DFS or BFS ways, and thus generate richer representations.

Graph-based models are really prevalent these years, and have become new state-of-the-art for collaborative filtering. To better represent users and items, we also adopt LightGCN (He et al., 2020) to do pre-rank scoring. Specifically, LightGCN learns user and item embeddings by linearly propagating them on the user-item interaction graph. For better performance, we use 4-layer LightGCN with the embedding dimension of 2048, nodes dropout rate=0.4 and learning rate=0.001.

Moreover, considering the cross-market differentiation for user/item representations and user-to-item interest quantification, we calculate the scores based on different market combinations. For example, to generate pre-ranking scores for t1, we can use market combinations of s1-t1, s1-s2-s3-t1, s1-s2-t1, s1-s3-t1, s1-s2-s3-t1-t2, etc. as pretraining corpus, likewise for t2. However, LightGCN training is relatively time-consuming, so we only conduct some of the combinations for t2 later to get the final boosting. We use ItemCF as the example to show the results of this cross-market combination strategy in Table 4. All scores are offline NDCG@10 calculated with valid set. We can see that 10 features are generated through this process for each pre-ranking model. In most cases, our models perform better with more data. However, we can find that although the data in s1 is richer than that in s3, s3 still contributes more to t1 than s1 does with the scores 0.6805 versus 0.6781. In addition, it’s obvious that different source markets have different contributions to the two target markets. s3 contributes the most to t1, then s2 and s1 come after. As for t2, s1 dominates the main contributions, thus s3 and s2 provide limit gains. The two target markets share different characteristics. By calculating pearson correlation coefficients between the 10 pre-rank scoring features generated through different market combinations, it’s found that the pearson correlation coefficients in t2 are much higher and of less difference between each other than that in t1, as shown in Figure 3. Obviously, t2 market suffers serious multicollinearity problem when training the model with these features. Feature refining should be adopted to get a more precise and generalized model.

Hundreds of features are generated from pre-ranking stage, final ranking models are trained based on these features. However, many features are probably suffering distribution shifted issue between training and test set or redundant with severe multicollinearity problems as the analysis shown above. To make the final model more robust, we do further feature analysis and selection before the final model training. Our feature selection mainly based on 3 factors, which are covariate shift test (Shubham.jain, 2017), offline cross-validation scores and feature importance analysis. At first, we conduct covariate shift test to exclude distribution-shift features. One of the basic and common assumptions for machine learning tasks is that training and test dataset are independent and identically distributed. Covariate shift is a form of distribution shift between training and test set which is a significant obstacle in developing robust machine learning models. So we need to exclude features whose distributions are significantly shifted between training and test set. Specifically, we refer to the method mentioned in this article (Shubham.jain, 2017) to do the covariate shift test. Secondly, we do heuristic feature selection based on offline k-fold cross validation scores. Under the condition of fixed seeds, folds and model hyperparameters, we eliminate a feature or a group of similar features every time to do k-fold cross validation iteratively in heuristic way. At last, we do further feature analysis based on feature importance and null importance. Null importance (OLIVIER, 2018) is a very prevalent feature selection method in Kaggle. Firstly, Null importances distributions are created by fitting the models over several runs on a shuffled version of the target. And then, feature selection is adopted by comparing the null importances distributions with the actual importances gathered by fitting the models on the original target. The distance between null importances distributions and actual importances should be in a big gap and the variance of the null importances should be high if the given feature is important. After feature selection, we only keep 206 features for target market t1, and 147 for t2. Especially the feature selection in t2 market enables our model to get a significant boost on the leaderboard, which secure our top place on the leaderboard.

Based on the features selected after pre-ranking stage, combining with some global statistic features, similarities calculated with pretrained Word2Vec embeddings, we build two LightGBM (Ke et al., 2017) classifiers to get the final ranking scores for t1 and t2 separately.

We do not do much tuning in this stage except for some searching for model parameters num_leaves and learning_rate, which are proven to be important for the final results according to our experiments. As for model ensemble, we simply adopt bagging training with 10-fold cross validation to get a more robust model for each target market.