AntM²C: A Large Scale Dataset For Multi-Scenario Multi-Modal CTR Prediction

AntM²C collects CTR data in five scenarios on Alipay, and there are differences in the types of items in each scenario. As shown in Figure 1, the CTR prediction occurs in multiple scenarios, including services and content on search, vouchers on marketing, videos on Tab3 page, and advertisements on the membership page. In the search scenario, when a user enters search words, several relevant mini-apps of services or content are displayed for the user to click on. Marketing scenarios recommend some consumer vouchers, and users click the coupons they are willing to use. On the Tab3 page, the recommended items are primarily short videos, and users will click to watch the videos they are interested in. On the membership page, users may click on some online advertisements. In conclusion, AntM²C includes various types of items from different business scenarios. In section 3.2.2, we will show that there are differences in the data distribution of these different scenarios. The rich and diverse items provide a more comprehensive evaluation for CTR prediction.

AntM²C collects 9-day (from 20230709 to 20230717) CTR samples from the above-mentioned five scenarios and then filters out 1 billion samples of relatively high-activity users who have a total click count $\geq$ 30 across all scenarios. In the first stage of open sourcing, we randomly sampled 10 million data from these 1 billion samples, and their statistical properties are shown in Table 1. We will open all 1 billion data in the subsequent stage. For the purpose of protecting user privacy, we do not explicitly indicate the names of the scenarios in the dataset, but instead use the letters ’A-E’ as substitutes.

The AntM²C does not contain any Personal Identifiable Information (PII) and has been desensitized and encrypted. Each user in the dataset was de-linked from the production system when securely encoded into an anonymized ID. Adequate data protection measures were undertaken during the experiment to mitigate the risk of data copy leakage. It is important to note that the dataset is solely utilized for academic research purposes and does not represent any actual commercial use.

AntM²C contains a portion of overlapped users across the five scenarios. Table 2 shows the number of intersecting users among different scenarios, indicating that AntM²C can reflect the preferences of the same user for items in different scenarios to effectively conduct multi-scenario CTR evaluation. As for items, due to the significant diversity in item types among different scenarios, there is no intersection of items between different scenarios.

Figure 2 illustrates the frequency of user and item in AntM²C dataset, including all samples and samples from different scenarios (A-E). The horizontal axis represents the number of frequencies for users/items, while the vertical axis represents the number of users/items at that frequency. It can be observed that, in terms of item distribution, all scenarios exhibit a long-tail distribution, with 80% of the sample appearing less than 5 frequencies. This long-tail distribution is consistent with real-world situations. As for user distribution, there are differences between scenarios. In scenario B, the distribution of user frequency has two peaks, one at less than 5 times and the other around 50 times. After the frequency is greater than 50, the number of users decreases as the frequency increases. In other scenarios, the exposure frequency of users follows a long-tail distribution similar to that of items, where more exposure frequency leads to fewer users. Due to the overlapping users between scenarios, the long-tail distribution of users in multiple scenarios becomes a normal distribution in the global samples. Most users have an exposure frequency of around 50. Overall, the distribution of items and users in AntM²C reflects CTR prediction in practice.

The user features consist of static profile features⁶⁶6User static attributes and item title will be open-sourced in the subsequent phases. and user sequence features. The static profile features include basic user attributes such as gender, age, occupation, etc. The sequence features provide the user’s recent activities on Alipay, including clicked mini-apps, searched services, purchased items, etc. As mentioned in Section 3.1.3, these user features have been desensitized and encrypted for the purpose of user privacy protection and appear in the dataset in an encrypted ID format, making it impossible to reconstruct the original user features. In addition to the ID-based features, AntM²C also includes the raw text of user search entities to provide multi-modal evaluation.

The item features consist of item ID and item textual features. The item ID is a globally unique identifier for each item, and the encoding of item IDs varies across different scenarios. To address the inconsistency of item IDs across scenarios, AntM²C also includes the original title text of the items⁶⁶footnotemark: 6 and entities extracted based on the title text.

In addition to user and item features, AntM²C also provides additional features such as log time and scene identification. Users can utilize these extra features to flexibly split the training, validation, and testing sets based on time and evaluate the performance in different scenarios.

The label in AntM²C indicates whether the user clicked on the corresponding item. If the user clicked, the label is set to 1, otherwise it is set to 0. The ratio of positive to negative samples in AntM²C can be obtained from the click rate in Table 1. It should be noted that there are a large number of negative samples in the actual online logs (samples that were exposed but not clicked on). To address this issue, negative sampling was performed which resulted in a higher click-through rate in the AntM²C dataset compared to that in the actual online logs.

In the multi-scenario CTR evaluation, we divide the AntM²C dataset based on time, using the data before 20230717 as the training set and the data on 20230717 as the test set. The training and test sets include samples from all five scenarios, and their data distribution is shown in Table 4. It can be observed that there are differences in the number of training and test samples among different scenarios. Among them, Scenario B has the highest number of samples, which is ten times that of Scenario E. In terms of features, we use the user and item features from the ID category as shown in Table 3. The text features will be used for multi-modal evaluation (see in Section 4.3).

We mainly choose the multi-task methods as the baseline methods for multi-scenario CTR prediction. We treat the CTR estimation for each scenario as a task and share the knowledge among the scenarios at the bottom layer, with each scenario’s CTR score output at the tower layer. The baseline methods and hyperparameter settings are as follows:

• •

  DNN: The DNN is trained on a mixture of samples from all scenarios without tasks, serving as the baseline for multi-scenario CTR prediction. The DNN consists of three layers with 128, 32, and 2 units, respectively. The following multi-task model has the same number of layers and unit settings as the DNN.

• •

  Shared Bottom (Ruder, 2017): Shared bottom is the most fundamental model in multi-task learning, where the knowledge is shared among the tasks at the bottom layer. Each task has its own independent tower layer and outputs the corresponding CTR score⁷⁷7https://github.com/shenweichen/DeepCTR.

• •

  MMoE (Ma et al., 2018): Based on the shared bottom, MMOE introduces multiple expert networks, each specialized in predicting a specific task, sharing a common input layer. Additionally, MMOE adds a gating network that assigns different weights to each expert based on the input data to determine their influence on predicting the output for a specific task. In the experiment, we set the number of experts in MMOE to 6⁸⁸8https://github.com/drawbridge/keras-mmoe.

• •

  PLE (Tang et al., 2020): Based on MMOE, PLE further designs task-specific experts for each task, while retaining the shared expert. This structure allows the model to better learn the differences and correlations among tasks. We set the number of experts in PLE to be the same as MMOE, with each of the five scenarios having its own specific expert and one globally shared expert⁷⁷footnotemark: 7.

All baseline methods utilized the Adam (Kingma and Ba, 2015) optimizer with a learning rate of 1e-3 for parameter optimization. The models were trained for 5 epochs with a batch size of 512.

Table 5 shows the evaluation results of different baseline methods on multi-scenario CTR prediction, from which we can draw the following conclusions. Firstly, compared to the DNN model that trains all data together without considering scenario characteristics, all multi-task models achieve better performance. This demonstrates that in AntM²C, there are differences and commonalities between scenarios, and simply mixing training data will not achieve the best results. Secondly, the CTR performance varies across each scenario, indicating different levels of difficulty between scenarios. For example, in scenario B, where there is a large amount of data, the AUC is generally above 0.93, while in scenario D, the AUC is only around 0.68. The diverse business scenarios and items in AntM²C enable a more comprehensive and diverse evaluation of CTR. Finally, the expert-structured MMOE and PLE outperform the shared bottom model, demonstrating that refined model design can enhance the performance on AntM²C. AntM²C is capable of reflecting the differences between different models.

In cold-start CTR prediction, we split the dataset based on time, using data before 20230717 as the training set and data on 20230717 as the validation and test sets. Based on this data division, we simulated two common cold-start problems in practice: few-shot and zero-shot.

• •

  Few-shot: users and items that appear in the training set with a count greater than 0 and less than $N$⁹⁹9The selection of this threshold $N$ can vary based on experiments, and we use 100 as an example for all experiments., meaning there is only a small amount of training data for these users and items.

• •

  Zero-shot: users and items that have never appeared in the training set, indicating that either the user is visiting the scenario for the first time or the item has been launched and added to the scenario on the first day.

Table 6 shows the data distribution of the test set under cold-start CTR evaluation. By using this dataset division, we can comprehensively evaluate and compare the performance of CTR models on few-shot and zero-shot samples. For few-shot samples, we can observe the model’s performance with only a small amount of training data and evaluate the model’s generalization ability. For zero-shot samples, we can evaluate the model’s recommendation ability on samples that it has never seen before.

The key issue in cold-start modeling is how to learn user preferences and embeddings of users and items with limited data. In recent years, meta-learning-based cold-start methods have become state-of-the-art methods. We selected several representative methods with publicly available code as our baseline models.

• •

  DropoutNet (Volkovs et al., 2017): The DropoutNet is a popular cold-start method which applies dropout to control input, and exploits the average representations of interacted items/users to enhance the embeddings of users/items. We implemented the DropoutNet algorithm based on open-source code¹⁰¹⁰10https://github.com/layer6ai-labs/DropoutNet.

• •

  MAML (Finn et al., 2017): The MAML algorithm is a popular meta-learning approach that aims to enable fast adaptation to new tasks with limited data. MAML learns a good initialization of model parameters that can be effectively adapted to new tasks quickly. We treat each user and item as a task in MAML, and conduct meta-training on warm items. Then we perform meta-testing on cold-start items. The subsequent meta-learning-based algorithms will also follow this task setting.

• •

  MeLU (Lee et al., 2019): The MeLU algorithm is the first to apply the MAML to address the cold-start problem in recommender systems. Building upon MAML, MeLU ensures the stability of the learning process by not updating the embeddings in the inner loop (support set). The hyperparameter settings in MeLU were determined based on the public code¹¹¹¹11https://github.com/hoyeoplee/MeLU implementation.

• •

  MetaEmb (Pan et al., 2019): The MetaEmb algorithm also applies the MAML to address the cold-start problem in recommender systems. Unlike MeLU, MetaEmb focuses on optimizing the embeddings of items. It learns an initial representation using all training samples and then quickly adapts the embeddings of cold-start items. We implemented the MetaEmb algorithm based on open-source code¹²¹²12https://github.com/Feiyang/MetaEmbedding. Although MetaEmb only optimizes the embeddings of items, we have also applied the same approach to optimize the embeddings of users.

These base models share the common embedding and DNN structure. The dimensionality of embedding vectors of each input field is fixed to 32 for all our experiments. The Adam optimizer with a learning rate of 1e-3 is used to optimize the model parameters, and the training is performed for 3 epochs with a batch size of 512. In addition to the aforementioned cold-start algorithms, the DNN (without any cold-start optimization) is also considered as the baseline method for cold-start CTR.

Table 7 shows the CTR performance for cold-start users and items. Because there is limited data for cold start users and items, we do not calculate AUC by scenarios, and evaluate the overall performance of cold start users and items. From the table, we can observe several phenomena. Firstly, compared to the results shown in Table 5, the AUC for cold-start users and items are generally lower than the overall level, which demonstrates that AntM²C’s data can effectively reflect the differences between cold and warm items and users. Secondly, different cold-start methods show distinguishable results in AntM²C, and all of them are significantly better than the DNN model without cold-start optimization. This indicates that AntM²C can effectively compare the effects of different cold-start methods and demonstrate the distinctiveness between methods. Finally, the lower performance of zero-shot compared to few-shot indicates that zero-shot CTR prediction is more challenging than few-shot. The two cold start modes provided by AntM²C can comprehensively evaluate cold-start CTR prediction.

In multi-modal evaluation, we adapt the same data processing approach as in multi-scenario evaluation mentioned in Section 4.1.1, and additionally include the text features from Table 3: user query entities and item entities. The text features will be used as inputs to the model together with other ID features.

For the baseline model, we use the language model to process the text features, and then concatenate the text embedding with other ID features and input them into the multi-scenario model described in Section 4.1.2. For ease of evaluation, we choose MMoE as the backbone and pre-trained Bert-base¹³¹³13https://huggingface.co/docs/transformers/main/model_doc/bert (Devlin et al., 2018) as the text embedding extractor. The output dimension of Bert’s embeddings is 768. Then, a DNN with two layers, each layer having [768, 32] units, is used to reduce the dimension of Bert’s embedding to 32. This reduced embedding is concatenated with other features and input into the MMOE model. More powerful language models and the application of text features will continue to be supplemented in future works.

Table 8 shows the evaluation results of the multi-modal CTR. It can be observed that, after adding the text modality, the CTR performance is better in data-sparse scenarios C, D, and E compared to using only the ID modality in the MMoE. Since the current baseline for using the text modality is relatively simple, the improvement in performance is not significant. However, this shows the potential of the text modality provided in AntM²C to improve CTR performance.