Deconfounded Causal Collaborative Filtering

We formulate the process of user-system interaction as a causal graph shown in Figure 3. We explain the rationality of our designed causal graph from the view of data generation as follows:

• •

  Edges $\{U,C\}\rightarrow V$ denote that the user will determine which item to interact with, and the unobserved confounder may affect user’s decision.

• •

  Edge $V\rightarrow M$ denotes that the item will determine the inherent item features. Here, the inherent features of the items that are designed as is from the beginning and will never change. For example, a silver macbook has a color as silver, brand as Apple, memory as 8G, OS as Macintosh. Thus the inherent item feature is solely determined by the item $V$ and will not be directly influences by user $U$ and confounders $C$.

• •

  Edge $C\rightarrow U$ denotes that the unobserved confounders may affect user’s preference. For example, as we exemplified in Section 1, weather, as a confounder, may personally affect users and lead to different preference.

• •

  Edges $\{U,M,C\}\rightarrow Y$ denote that the user preference score is determined by user $U$, inherent item feature $M$ and unobserved confounders $C$. Specifically, the user’s preference on a certain item may be affected by inherent item features (e.g., whether the functional attributes of the item conform to user preferences) and the confounding factors (e.g., the weather, the mood, etc.).

The effect of global confounders (e.g., the air temperature as mentioned in Section 1) is represented by edge $C\rightarrow V$ and edge $C\rightarrow Y$. The effect of personal confounders (e.g. the mood as mentioned in Section 1) is represented by edge $C\rightarrow U$, edge $C\rightarrow V$ and edge $C\rightarrow Y$.

Setting variable of inherent item features as the mediator is intuitive and reasonable because inherent item features such as the color or brand of products and the genre or director of movies are inherent and fixed properties of an item, which are not influenced by users or external confounders. The inherent item features are not directly affected by users or confounders. Considering a general example, a user will click an item during browsing the website, where the inherent item features may not visible to users at this stage (for example, amazon homepage recommendations only show an image without any text information about the inherent item features). After clicking the item, based on the detailed inherent item features, combining with the effect of unobserved confounders and personal preference, the user will decide like or not, purchase or not, etc. Our model aims to estimate the user’s true preference of an item that is decided by the inherent item features and not affected by confounders.

In order to achieve our purpose during implementation, it is crucial to carefully select the inherent item features to serve as the mediator. Not all item features can be considered as inherent features since the mediator should not be directly affected by users or confounders. Specifically, we only select objective features, including features related to the appearance and functionality of the item, such as item description, color, and other similar features. Some other item features, such as sales rank and related products, are generated based on user’s subjective reviews or user interactions and cannot be selected as the mediator. In real-world scenarios, it may not always be feasible for the selected mediator to completely intercept the causal pathways from the items to the preference variable, resulting in the possibility of direct effects from items to preference. Such a situation violates the front-door criterion and may have a negative impact on the deconfounding performance. However, if the mediator is chosen appropriately, it can capture a significant portion of the causal effect, making the direct effect weak and negligible in predicting preference. Hence, the impact on performance may not be significant. We will show some experimental results to illustrate it in Section 5.

It is worth clarifying that the causal graph is used for describing how the collected data was generated. We ultimately build a recommendation model that can leverage this causal graph to produce better estimations of user preferences.

In this section, we estimate user preference from a causal view, which aims to answer a what if question: what would be the user’s preference on an item if we intervene to recommend the item to the user. In particular, we estimate a user’s preference if a certain item is recommended to the user, which can be represented as $P(y|u,do(v))$. The estimation of the preference score $P(y|u,do(v))$ is based on the designed causal graph in Figure 3. From the causal graph as shown in Figure 3, it seems that the desired estimation can be calculated by the back-door adjustment (Pearl et al., 2016, p.61) (as Eq.(1)) since a set of variables $\{C,U\}$ satisfies the back-door. However, the back-door adjustment is not applicable because variable $C$ is unobserved or unmeasurable. Therefore, considering the existence of unobserved confounders, we apply front-door adjustment (Pearl et al., 2016, p.69) (following Eq.(2)) to estimate user’s preference on items. Concretely, We estimate the preference score of a $(u,v)$ pair as follows:

where $y$ represents the value of preference score and $m$ represents a specific value of inherent item features $M$. Here $m$ integrates all inherent features of an item. For example, an item has inherent features including color, brand, etc. $m$ represents a inherent feature profile which integrates the specific value of each inherent features, such as color as silver, brand as Apple, etc.

To utilize the item features into model prediction, we represent the item features into the continuous space, then we can rewrite the preference score as follows:

We denote the described feature of item $v$ as $\mathbf{m}_{v}$, and consider the conditional distribution of item feature as $P(\mathbf{m}|v)\sim\mathcal{N}(\mathbf{m}_{v},\sigma_{m})$. Using distribution instead of deterministic value is intuitive and reasonable in practice, because the feature of individual product may slightly differ from the described feature for reasons such as quality control issue.

To calculate the integration over item’s inherent feature, we apply Monte Carlo Sampling. More specifically, we sample $d$ values of $\mathbf{m}$ (i.e., $\mathbf{m}_{1},\mathbf{m}_{2},\cdots,\mathbf{m}_{d}$) based on the distribution of $P(\mathbf{m}|v)$. Therefore, we can convert the calculation in Eq.(6) as follows:

Real-world systems may include a large scale of items, therefore, it is unrealistic to involve all items to estimate the preference of a user-item pair. To tackle this issue, we modify the original front-door adjustment to a sample-based one that is more suitable and realistic for the recommendation scenario. Specifically, we randomly sample a set of items while calculating $P(y|u,do(v))$ as a trade-off between efficiency and deconfounding performance (more discussion and results are provided in Section 5.7). The item set for a user-item pair $(u,v)$ is represented as $R(u,v)$ (including sampled items and item $v$), and the number of sampled items is $n$, which is a hyper-parameter to be selected. We can rewrite Eq.(7) into a sample-based format.

For each item $v^{\prime}\in R(u,v)$ in Eq.(8), its exposure probability (i.e. $P(v^{\prime}|u)$) and conditional preference (i.e. $P(y|u,v^{\prime},\mathbf{m}_{j})$) are two required values. We will introduce how to calculate them respectively.

In this section, we will introduce details about model learning, i.e., how to learn the estimation as Eq.(14). In this work, we use the pair-wise learning-to-rank algorithm (Rendle et al., 2012) for training. Suppose we observe user $u$ interacted with item $v_{i}$ in the dataset, The estimated preference $\hat{y}^{+}_{ui}$ is obtained from Eq.(14). We sample another item $v_{j}\in\mathcal{V}$ that user $u$ did not interact with as a negative sample. The estimated preference for negative sample $v_{j}$ is represented as $\hat{y}^{-}_{uj}$. The difference between two estimated preferences is noted as $\hat{y}_{uij}$.

Given the observed interactions, we randomly sample a negative item for each user-item pair in implementation. Specifically, we use $\mathcal{V}^{+}_{u}$ to represent the observed items set for user $u$ ($v_{i}\in\mathcal{V}^{+}_{u}$) and $\mathcal{V}^{-}_{u}$ to represent the sampled negative item set for user $u$ ($v_{j}\in\mathcal{V}^{-}_{u}$). The recommendation loss function can be written as:

where $\Theta$ represents all parameters of the model; $\lambda_{\Theta}$ is the L2-norm regularization weight; $\sigma(\cdot)$ is the sigmoid function: $\sigma(x)=\frac{1}{1+e^{-x}}$. It is worth mentioning that the representation of inherent item feature $\mathbf{m}$ and exposure probability model to calculate $P(v^{\prime}|u)$ are pre-trained and fixed during the training of our model. The overall algorithm is provided in Algorithm 1.

In this section, we discuss the identifiability of our estimated causal preference. Generally speaking, identifiability in causal inference indicates whether we can use observed data to estimate target values with consistency (Pearl et al., 2016; Mohan and Pearl, 2021). From a view of causal inference in statistics, $(y,u,v^{\prime},\mathbf{m}_{i})$ is unobserved in the data thus the target value $P(y|u,do(v))$ (Eq.(14)) is unidentifiable. Such non-identifiability problems are common problems in recommender systems. The reason is that recommender systems are often estimated from datasets containing a very high portion of missing data. Many works (Marlin et al., 2011; Wang et al., 2019; Saito et al., 2020) have shown that the missing data is often missed not at random (MNAR) in reality, and recent research (Mohan and Pearl, 2021) has shown that the target value is unidentifiable to any method if MNAR exists. Even if the target values are unidentifiable, in many practical intelligent systems such as recommender systems, we still need to estimate reasonably good values for such unidentifiable variables (Zhang et al., 2021; Yang et al., 2021; Wang et al., 2021a), so as to make it possible to provide practical services for users. One example is to estimate $P(y|u,v’)$ where $v’$ is an item that user u never interacted with before. Real-world recommender systems usually have billions of items, but each individual user only has interacted with tens or at most hundreds of them. Even though most of the $(u,v’)$ interactions cannot be observed, we still need to estimate the probability of such interactions, because only by doing so can we provide recommendations to users (Zhang et al., 2021; Yang et al., 2021; Wang et al., 2021a). If we give up any attempt to estimate such unidentifiable variables, then many intelligent services will be impossible in practice.

In our model, the key to estimate such unidentifiable variables is through “collaborative learning”. The basic idea is that, although the target value is unobserved, there could exist some other similar examples whose values are observed, so that we can borrow their values to estimate the target value. By doing so, the examples can “collaborate” with each other so as to help estimate the values of each other, which is the key idea of “collaborative filtering” – one type of collaborative learning techniques. In our model, More specifically, given $(y,u,v,\mathbf{m}_{v})$ is observed, $(y,u,v’,\mathbf{m}_{j})$ is indeed unobserved, but if $v$ and $v’$ share similar features, $\mathbf{m}_{v}$ and $\mathbf{m}_{v’}$ will also be similar representations learned by language models. Specifically, when we calculate $P(y|u,v’,\mathbf{m}_{j})$, considering $\mathbf{m}_{j}$ is similar to $\mathbf{m}_{v}$ (i.e., $\mathbf{m}_{j}$ is sampled from $\mathcal{N}(\mathbf{m}_{v},\sigma_{m})$), if $v’$ shares similar inherent item feature with $v$, then $\mathbf{m}_{v’}$ is also similar to $\mathbf{m}_{j}$, thus $P(y|u,v’,\mathbf{m}_{j})$ is similar to observed $P(y|u,v,\mathbf{m}_{v})$. On the contrary, if $v’$ is very different from $v$, then $\mathbf{m}_{v’}$ is also different from $\mathbf{m}_{j}$, which will lead to $P(y|u,v’,\mathbf{m}_{j})$ being quite different from $P(y|u,v,\mathbf{m}_{v})$. The above is a direct implication of the collaborative learning principle in representation learning techniques.

We introduce the baselines used in our experiments as follows.

• •

  MF (Rendle et al., 2012): The matrix factorization model, which uses matrix factorization as the prediction function under the Bayesian personalized ranking framework.

• •

  DMF (Xue et al., 2017): Deep Matrix Factorization is a deep learning model for recommendation, which filters the user-item interaction matrix through a neural network architecture for prediction.

• •

  DecRS (Wang et al., 2021a): A deconfounded recommendation model for alleviating bias amplification, which identifies the confounder as user historical distribution over item groups, and further applies back-door adjustment to mitigate the impact of the confounder.

• •

  IPW-MF (Schnabel et al., 2016): Inverse Propensity Weighting Matrix Factorization for recommendation, which adapts causal inference (inverse propensity weighting, specifically) on the matrix factorization model to handle the selection bias in observational data.

• •

  DCF (Wang et al., 2020): A deconfounded recommendation model, which uses an exposure model to construct a substitute confounder and then conditions on the substitute confounder for modeling.

• •

  HCR (Zhu et al., 2022): A deconfounded recommendation model, which considers click behavior as mediator and designs a multi-task learning framework that simultaneously learns the probability of click and post-click behavior to mitigate confounding effect.

• •

  DCCF$\_$NS: This is a no-item-sampling variant of our model. Specifically, for a user-item pair $(u,v)$, this variant only uses $P(v|u)f(u,v,m)$ in Eq.(13) as the estimated preference. It can be considered as a reweighting method.

• •

  DCCF$\_$ND: This is a no-deconfounding (ND) variant of our model. Specifically, for a user-item pair $(u,v)$, this variant only uses $f(u,v,m)$ in Eq.(13) as the estimated preference. It can be considered as our model (DCCF) without the front-door adjustment, which loses the deconfounding ability.

The baselines include both classical association-based recommendation models (MF and DMF) and state-of-the-art deconfounded recommendation models (DecRS, IPW-MF, DCF and HCR), as well as two ablated variants of our own model (DCCF$\_$NS and DCCF$\_$ND). Many recent deconfounded methods such as DecRS mainly focus on mitigating one specific confounder, while our model is used to mitigate the effect of unobserved confounders. Some other deconfounded methods require certain user information such as user social networks (Gao et al., 2021), which is not suitable as a baseline for our problem setting. Finally, we use DCCF to denote the complete version of our model.

Our model aims to estimate the accurate preference with the existence of unobserved confounders. To evaluate the deconfounding performance, some deconfounded models, which mainly focus on one specific confounder, group items or users by the value of the confounder and compare the performance among groups. For example, Zhang et al. (2021) treat item popularity as the confounder, thus the deconfounding performance is evaluated by grouping items by item popularity and calculating the recommendation ratio for each group. In our case, however, it is impossible to group users or items by unobserved confounders.

Since the deconfounded recommendation will obtain more accurate estimation of preference to improve the recommendation performance (Wang et al., 2020), we evaluate our model and baselines on the top-$K$ recommendation task. We adopt three widely used metrics to evaluate recommendation performance including: Normalized Discounted Cumulative Gain@$K$ (nDCG@K for short), Recall@K (Rec@K for short), and Precision@K (Pre@K for short). The calculation of the three metrics is given as follows:

where $DCG_{u}@K$ is the Discounted Cumulative Gain at position $K$ for user $u$, $IDCG_{u}@K$ is the Ideal Discounted Cumulative Gain through position $K$ for user $u$, $|TP_{u}|$ is the number of recommended items that are relevant for user $u$ at position $K$, $|TP_{u}|+|FN_{u}|$ is the total number of relevant items for user $u$, and $|TP_{u}|+|FP_{u}|$ is the total number of recommended items at position $K$ for user $u$. In our experiment, we refer to (Wang et al., 2020) and set $K=5$.

We use the same training, validation and testing sets for all models, including our models and baseline models. For evaluation, we apply real-plus-N (Said and Bellogín, 2014) to calculate the values of each metric. Concretely, for each user in the validation and testing set, we randomly sample 1000 negative items for ranking evaluation in real-world data and 100 negative items for synthetic data. All recommendation models adopt the Bayesian Personalized Ranking (BPR) (Rendle et al., 2012) framework for pair-wise learning. During the training, we sample one negative item that the user did not interact with for each interacted user-item pair. We optimize the models using mini-batch Adam with a batch size of 128.

To obtain the inherent feature representation of the items (i.e., $\mathbf{m}_{v}$ in Eq.(13)), we apply a pre-trained transformer-based sentence embedding model⁴⁴4we apply the pre-trained paraphrase-distilroberta-base-v1 sentence embedding model in a public transformer implementation: https://github.com/UKPLab/sentence-transformers to represent the textual information of an item into a dense vector embedding with dimension 768. For the Amazon Review Datasets, the textual information comes from the ’title’, ’feature’, and ’literal description’ of each item, which are included in the meta-data of the amazon review dataset. For the Yelp dataset, we include the ’name’, ’address’, ’city’, ’state’, ’attributes’, and ’categories’ of the business into the inherent feature, which can be found in the business information in the Yelp dataset. We use the above pre-trained sentence transformer to convert the textual information into separate embeddings and further take the average as the inherent feature representation of the item.

For the hyper-parameters, we perform the grid-search and search the user/item embedding dimension from {32,64,128}, the structure of the neural network is a two-layer MLP with dimension 64 for deep models. For all recommendation models, we search the learning rate from {0.0005,0.001,0.003,0.005}, the $\ell_{2}$-regularization weight is chosen from {1e-3, 1e-4, 1e-5}. The total number of epoches is set to 100. For a fair comparison, we tune the parameters to the best performance for each model on the validation data based on nDCG@5. For our model, we sample 20 items for each user-item pair for calculating the expectation (i.e., the number of sampled items $n$ in Section 4.2) and sample 2 inherent item feature values for calculating the integration (i.e., the number of sampled item features $d$ in Section 4.2). We set the variance of the distribution of inherent item feature as 0.1 (i.e., $\sigma_{m}$ in Section 4.2). To obtain the exposure probability calculated based on Eq.(11), we estimate the user independent propensity scores as (Saito et al., 2020):

Here $y_{uv}$ is an indicator: $y_{uv}=1$ if the user-item pair $(u,v)$ is observed in the dataset, $y_{uv}=0$ otherwise. We set $\eta=0.5$ in the experiment. The running time of our model is about 40 seconds/epoch for Electronics, 15 seconds/epoch for CDs and Vinyl and 35 seconds/epoch for Yelp.

In this section, we aim to answer RQ1 and RQ2. For real-world data, the recommendation performance for models on the SKEW splitting datasets are shown in Table 3. The SKEW splitting strategy simulates a test set as a result of randomized experiment, and the test distribution is different from the training distribution. The recommendation performance on the RAND splitting datasets are shown in Table 4. The RAND splitting strategy is the regular splitting strategy that randomly splits data samples into testing set, thus the training and testing distributions are constant. Comparing the RAND and SKEW splitting strategies, we can observe that the recommendation performance with the RAND splitting strategy is consistently better than the performance with the SKEW splitting strategy on all three datasets. This shows that learning recommendation models when the training and testing distributions are different (SKEW) is a more challenging task than under the same train-test distribution (RAND).

The following observations are consistent no matter what the splitting strategy is. First, the recommendation models shown in Table 3 and Table 4 can be categorized into classic association-based models and deconfounded models, where the difference between the two is whether the effects of confounders are mitigated or not. From the results, we can see that the deconfounded recommendation models (DecRS, IPW-MF, DCF, HCR and DCCF) achieve better recommendation performance than classic recommendation models (MF and DMF) in most cases on all three datasets. When averaging across all deconfounded recommendation models on all datasets using the SKEW splitting strategies, the deconfounded models achieve 54.3% relative improvement on nDCG@5 than classic recommendation models, 55.0% on Recall@5, and 51.7% on Precision@5. When averaging across all deconfounded recommendation models on all datasets using the RAND splitting strategies, the deconfounded models achieve 4.9% relative improvement on nDCG@5, 2.9% on Recall@5, and 8.3% on Precision@5. According to this comparison, we can observe that the existence of confounders will lead to inaccurate recommendations and deconfounded recommendation models will improve the recommendation performance by mitigating the effect of confounders. Meanwhile, comparing the improvement on both splitting strategies, the improvement brought by the deconfounded models is more significant on the SKEW datasets than the RAND datasets.

Among deconfounded recommendation models, we can see that our DCCF model achieves the best recommendation performance in most cases. Compared with the strongest deconfounded model in baseline, when averaging across all datasets using the SKEW splitting strategies, our model yields 18.2% improvement on nDCG@5, 16.9% improvement on Recall@5, and 17.2% improvement on Precision@5. When averaging across all datasets using the RAND splitting strategies, our model yields 4.9% improvement on nDCG@5, 5.7% improvement on Recall@5, and 4.4% improvement on Precision@5. These observations imply that by applying the front-door adjustment into the estimation of user’s preference, our model is capable of reducing the effect of unobserved confounders and further improving the recommendation performance. Moreover, based on the results, applying the front-door adjustment is more effective than other deconfounded models.

Although the front-door adjustment is effective on reducing the effect of unobserved confounders, it requires inherent item features. We need to confirm that the improved performance is indeed brought by the deconfounding effect instead of the use of inherent item features. Therefore, we consider a variant of our model DCCF$\_$ND, which also involves inherent item feature representation into estimation but without the front-door adjustment as deconfounding component. More specifically, unlike DCCF that applies the front-door adjustment with weighted sum over sampled items to obtain the estimation of a user-item pair, DCCF$\_$ND only considers the user-item pair itself when calculating the estimation. From the performance in Table 3 and Table 4, we can see that DCCF$\_$ND is even worse than classic models in many cases. Therefore, the improvement of our model is brought by deconfounding with the front-door adjustment instead of involving inherent item feature representation into calculation. We also include a variant of our model DCCF$\_$NS, which involves exposure probability but without sampled items. It can be considered as a reweighting method, instead of applying the front-door adjustment. From the performance in Table 3 and Table 4, we can see that DCCF$\_$NS is able to improve recommendation performance compared with classic models, and even achieve better performance than deconfounded models in some cases. However, it cannot obtain better performance than our model. Therefore, applying front-door adjustment is more effective than using exposure probability for reweighting.

For synthetic datasets, the recommendation performance for models are shown in Table 5. Dataset SD1 simulates a scenario where the selected inherent item features completely intercept the causal effect from items to preference, while SD2 represents a situation where the mediator intercepts a significant portion of the causal effect from items to preference. We can observe that deconfounded models still achieve better performance than classical models. Among different deconfounded models, DecRS and IPW-MF mainly focus on mitigating the effect of item popularity, therefore, they can only achieve lightly improved performance since the confounders in synthetic data are not necessary related to item popularity. The remaining deconfounded models (i.e., DCF, HCR and DCCF) are not restricted to a specific confounder, thus are capable of achieving better deconfounding performance. Our DCCF model achieves the best performance on both synthetic datasets, which illustrate the effectiveness of our model on mitigating the effect of unobserved confounders, even when the mediator cannot fully intercept the causal effect from items to preference.

In this section, we will discuss the influence of the number of sampled items to answer RQ3. As we mentioned in Section 4.2, instead of strictly following the front-door adjustment as Eq.(7), we randomly sample a set of items and estimate the preference by the sample-based front-door adjustment (i.e., as Eq.(8)).

To discuss how the number of sampled items influence the recommendation performance, in Figure 5, we plot nDCG@5 for DCCF (i.e., DCCF$\_$Unbias in Figure 5) with different numbers of sampled items. We can observe that the recommendation performance increases with the increasing of the number of sampled items, but the magnitude of the increment decreases as the number of sampled items increases. Considering the model applies a two layer MLP with dimension $D_{uv}$, the time complexity of calculating Eq.(13) would be $(D_{uv}+D_{m})D_{uv}+2D_{uv}^{2}$. Suppose $D_{m}$ is linear to $D_{uv}$, the time complexity is $O(D_{uv}^{2})$. For a fixed number of sampled inherent item feature $d$ ($d\ll D_{uv}$), if the number of sampled item $n$ is small enough (i.e., $n\ll D_{uv}$), the time complexity of calculating $P(y|u,do(v))$ is $O(D_{uv}^{2})$. However, if $n$ is large (i.e., linear to $D_{uv}$), the time complexity would be $O(D_{uv}^{3})$. Additionally, larger $n$ will occupy more memory during the training. Therefore, it is important to choose an appropriate value of $n$ to balance the recommendation performance and efficiency when applying the DCCF model.

In this section, we will discuss the influence of the exposure models to answer RQ4. As we mentioned before, the exposure probability, which is required by the front-door adjustment but not available in the feedback data, is an essential component of our model. To empirically show the influence of it, we design three versions of our model with different exposure probabilities. The only difference among them is how to obtain the exposure probability.

• •

  DCCF$\_$Random: In this model, the exposure probabilities are not learned from the data, instead, we randomly generate a matrix to represent the exposure probability.

• •

  DCCF$\_$Uniform: In this model, we assume that each item has an equal probability to be exposed for each user.

• •

  DCCF$\_$Bias: In this model, we estimate the exposure probability as Eq.(10), and use the pair-wise learning method to train the model based on implicit feedback.

• •

  DCCF$\_$Unbias: This model is the same as DCCF in Table 3 and Table 4. Concretely, we consider the bias in the implicit feedback data, and apply Eq.(11) to obtain unbiased estimation of exposure probabilities.

Following the setting in main experiment, we evaluate three datasets using two splitting strategies on ranking task with metrics nDCG@5, Recall@5 and Precision@5. The recommendation performance on the SKEW splitting strategy is shown in Table 6 and the performance on the RAND splitting strategy is shown in Table 7. For each of the version, we use the same representation of inherent item features and sample 10 items for each pair to calculate the estimated preference.

From the results, we can see that DCCF$\_$Unbias achieves the best performance; DCCF$\_$Random and DCCF$\_$Uniform have the worst performance (DCCF$\_$Random and DCCF$\_$Uniform get similar performance) in most cases. This observation is consistent on both RAND and SKEW splitting strategies. Comparing the four versions with different exposure models shows that accurate exposure probability will lead to accurate estimation under the front-door adjustment. Specifically, the exposure probability is not observed in the feedback data, therefore, we need to learn a model to estimate it from observational feedback data. DCCF$\_$Unbias uses the unbiased estimation and gets the most accurate exposure probability, while DCCF$\_$Random randomly generates exposure probabilities and DCCF$\_$Uniform applyies uniform probabilities, which are irrelevant with the feedback data, and thus will get the least accurate probability. We can see that the ranking of the exposure probability accuracy matches the ranking of recommendation performance.

The exposure model affects our model in several ways, i.e., not only the best recommendation performance, but also the trend of the recommendation performance as the number of sampled item increases. As we mentioned in Section 4 and Eq.(14), we apply the front-door adjustment over a set of sampled items to make it suitable for the recommendation scenario. Ideally, if there exist the ground truth values of exposure probability and we apply them into the calculation, then increasing the number of sampled items may reduce the efficiency but will at least not hurt the performance significantly.

In order to investigate how our model changes with increasing the number of sampled items under different exposure models, we plot nDCG@5 for four versions of the model with different numbers of sampled items, as shown in Figure 5. As we can see from the figure, the performance of DCCF$\_$Unbias increases with the increasing of sample number, the performance of DCCF$\_$Bias increases first and then drops after 5 samples, and the performance of DCCF$\_$Random and DCCF$\_$Uniform is monotonically decreasing. Meanwhile, for a given number of sampled items, the performance among the four models follows the conclusion as Table 6 and Table 7 in most cases, i.e., DCCF$\_$Unbias $>$ DCCF$\_$Bias $>$ DCCF$\_$Random/DCCF$\_$Uniform.

For DCCF$\_$Unbias, since it has the most accurate exposure probability, it will achieve better performance with more sampled items. For DCCF$\_$Bias, the probability is also learned from data as DCCF$\_$Unbias but not accurate enough due to ignoring the bias. Therefore, when the number of sampled items is small, the model can get slightly better performance with the help of sampled items. However, when more sampled items are involved into the estimation, inaccurate probabilities may mislead front-door adjustment and hurt the performance, thus resulting in performance drops. For DCCF$\_$Random/DCCF$\_$Uniform, the sampled items do not provide useful information but instead introduce noises that completely mislead the front-door adjustment, and thus hurt the performance. In summary, accurate exposure probability will improve the performance while inaccurate probability hurts the performance. Therefore, it is important to adopt an accurate exposure model to obtain better performance.