Information Theoretic Counterfactual Learning from Missing-Not-At-Random Feedback

Embedding is a conceptually classical approach for modeling the user and item in rating prediction. For example, in collaborative filtering [22], it represents an event $x=(u,i)$ by concatenation as $z=(e_{u},e_{i})$, then generates outcome prediction by $\hat{y}=e_{u}^{\top}e_{i}$. In deep learning models [13], there are multiple hidden layers $h_{1},\dots,h_{L}$ sequentially processing the embedding $z$, such as $h_{1}=W^{(1)}z$ and $h_{l}=W^{(l)}h_{l-1}$ for $l=2,\dots,L$. In this work, we view the feedforward neural network layers as a Markov chain of successive representations, indicated with

where $y$ is the true feedback returned by users (may be unknown to our learning algorithm). According to Tishby et al. [36], a standard information bottleneck has the following form

where $I(\cdot;\cdot)$ denotes mutual information of two random variables, and $\beta$ is a Lagrangian multiplier. Minimizing $I(z;x)$ encourages the representation $z$ to be compressed, i.e., being minimal to $x$. And, maximizing the second term $I(z;y)$ encourages $z$ to be sufficient to task $y$, such that $z$ can be predictive of $y$. Essentially, $\mathcal{L}_{IB}$ in Eq. (4) can be regarded as a supervised loss term plus an additional information regularization term on the representation [1]. Moreover, there is another random variable $O$ that affects the appearance of events $x$, but is not informative to the true outcomes $y$, i.e., $O{\perp\!\!\!\!\perp}y$. In this scenario, we would like $z$ to be independent of $O$, thus being free of policy bias.

The main challenge in adopting IB for optimization is that the mutual information in $\mathcal{L}_{IB}$ is cumbersome for calculation. Although previous works try to derive computable proxy for specific tasks [3], they are not suitable for MNAR data. Since we only have partial feedback about $y$, i.e., the majority of events are counterfactuals, we have no access to their true outcomes. Next, we will turn to derive a new objective function addressing this challenge.

For conciseness, we focus on a simple model with only the embedding layer $z$, namely $x\to z\to\hat{y}$, and extend it to multi-layer scenario in Section 2.3. Specifically, the second term in Eq. (4) is mutual information between embedding $z$ and target task $y$. We separate the embeddings into two parts: $z^{+}$ and $z^{-}$, which represent factual and counterfactual embeddings, respectively, i.e., $z^{+}\sim p(z|x^{+})$ and $z^{-}\sim p(z|x^{-})$.

As shown in Fig. 2, we factorize the original mutual information term by $I(z;y)=I(z^{+},z^{-};y)$. We postulate that $z^{+}$ and $z^{-}$ are independent, therefore according to the chain rule of mutual information:

However, as the outcomes $y$ of the counterfactuals are unknown, we have to identify another refined solution. Specifically, we cast Eq. (5) to

from which we derive a contrastive term between $z^{+}$ and $z^{-}$. This characterization is helpful for us to introduce a hyperparameter $\alpha$ to control the degree of this contrastive penalty. We then rewrite the original IB loss to

as our new objective function, where we propose an information theoretic regularization on $z^{+}$ and $z^{-}$. Intuitively, minimizing this term corresponds to encouraging $z^{+}$ and $z^{-}$ to be equally informative to the targeted task variable $y$, thus resulting in a more balanced model. More importantly, it does not need access to the counterfactual outcomes, which will be specified in Section 3.2.

Aside from the task-aware mutual information $I(z;y)$ in Eq. (7), $I(z;x)$ corresponds to the minimality of the learned embedding. Recall that we assume that the event $x$ follows the generative process $p(x,O)=p(O)p(x|O)$, where $O$ influences the appearance of $x$. Because $O$ is independent to task $y$, we hope the predicted outcome $\hat{y}$ is not influenced by $O$, or the learned embedding $z$ should contain low information about $O$. In this viewpoint, following the practice by Achille and Soatto [1], we identify that the minimality term is actually beneficial for embedding’s insensitivity against the nuisance $O$.

Please refer to Appendix A for a proof. Above proposition implies that an embedding $z$ is insensitive to the policy bias $O$, by simply reducing $I(z;x)$. Meanwhile, by maximizing $I(z;y)$ in IB Lagrangian, embedding $z$ will be forced to retain minimum information from $x$ that is pertinent to the task $y$. In deep models, according to the Data Processing Inequality (DPI) [9], minimizing $I(z;x)$ also works for controlling the policy bias of the successive layers.

The minimality term $I(z;x)$ in Eq. (7) can be measured with a Kullback-Leibler (KL) divergence as

To avoid operating on the marginal $p(z)=\int p(z|x)p(x)dx$, we use a variational approximation of $q(z)$ as the marginal $p(z)$, which renders

Suppose the posterior $p(z|x)=\mathcal{N}(e(x);{\rm diag}(\sigma))$ is a Gaussian distribution, where $e(x)$ is the encoded embedding of input event $x$ and ${\rm diag}(\sigma)$ indicates a diagonal matrix with elements $\sigma=\{\sigma_{d}\}_{d=1}^{D}$. In other words, we assume the embedding is generated by

If we fix $\sigma_{d}=0\ ,\forall d$, then $z$ would default to a deterministic embedding. Moreover, by considering a standard Gaussian variational marginal $q(z)=\mathcal{N}(0,I)$, the KL term reduces to

which means for a deterministic embedding, minimizing term $I(z;x)$ is equivalent to directly applying $\ell_{2}$-norm regularization on the embedding vector.

The mutual information $I(z;y)=H_{p}(y)-H_{p}(y|z)$, where $H_{p}(\cdot)$ demotes the entropy of $p(\cdot)$. The first entropy term is constant, which means maximizing $I(z;y)$ is equivalent to minimizing the second term $H_{p}(y|z)$. We then further derive the lower bound of $-H_{p}(y|z)$ as²²2Here we slightly abuse notation by denoting $H_{p,q}(y|z):=\mathbb{E}_{x\sim p(x)}\mathbb{E}_{z\sim p(z|x)}\int-p(y|x)\log q(y|z)dy$.

The term $q(y|z)$ is an estimate of $p(y|z)$ with a classifier parameterized by $\theta$, e.g., weight matrices in deep networks or embedding parameters. We can use the cross entropy as a proxy for the mutual information in the IB objective [1, 15], because $\max I(z;y)\Leftrightarrow\min H_{p,q}(y|z)$ as shown above. Therefore, replacing the contrastive term $I(z^{+};y)-I(z^{-};y)$ in Eq. (7) with $H_{p,q}(y|z^{-})-H_{p,q}(y|z^{+})$ obtains

Since we assume $z^{+}$ and $z^{-}$ are independent, the generative process can be written as $p(y,z^{+},z^{-})=p(y|z^{+},z^{-})p(z^{+})p(z^{-})$. In order to make the term tractable, we here approximate $p(y|z^{+},z^{-})$ by $q(y|z^{+})$,³³3We may also pick $p(y|z^{+})$ as the approximation, which renders $H_{p,q}(y|z^{+},y|z^{-})-H_{p,q}(y|z^{+})$. However, it has less operational meaning and its last term cancels out with another task-aware term $I(z^{+};y)$. then cast Eq.(14) to

This term further goes to our final results as

where the first term of the right hand side is the cross entropy between $q(y|z^{+})$ and $q(y|z^{-})$. We identify that the second term is in line with the maximum entropy principle [17, 4] and the confidence penalty proposed by Pereyra et al. [27] that is imposed on the output distribution of deep neural networks. While out of our derivation, the confidence penalty is only imposed on the factual output $q(y|z^{+})$. We hence propose to restrict the distance between factual and counterfactual posterior, which provably strengthens model’s generalization over the underlying users’ true preference distribution.

We next omit the superscript of $z^{+}$ in $I(z^{+};y)$ for simplicity in the following. Similar to the operation of task-aware mutual information in Eq. (13), we use an approximation $q(y|z)$ to substitute $p(y|z)$

which indicates that this term is a proxy of the cross entropy loss between the true outcome $p(y|x)$ and the prediction $q(y|z)$.

Taking all the above derivations together, we conclude to the final objective function $\hat{\mathcal{L}}_{CVIB}$, which encompasses four terms:

where term $①$ denotes the cross entropy between $p(y|z^{+})$ and $q(y|z^{+})$; term $②$ is cross entropy between $q(y|z^{+})$ and $q(y|z^{-})$; and term $③$ is entropy of $q(y|z^{+})$. For computing this objective, we need to draw $x^{+},y^{+}$ from the factual set $\Omega^{+}$ and $x^{-}$ from the counterfactual set $\Omega^{-}$ in one iteration. Specifically, term $①$ is supervised loss on the factual outcomes; terms $②$, $③$ are contrastive regularization for balancing between factual and counterfactual domains; and term $④$ is minimality loss to improve the model’s robustness against policy bias. The whole optimization process over the $\hat{\mathcal{L}}_{CVIB}$ is hence summarized in Algorithm 1, where we should specify the values of $\alpha$, $\beta$ and $\gamma$ to balance these terms in optimization.

In order to evaluate the learned model’s generalization ability on the underlying groundtruth distribution $p(x,y)$, rather than the only on the logged feedback, i.e., the factual domain, we usually need an additional MAR test set, where the users are served with uniformly displayed items, namely RCTs. As far as we know, there are only two open datasets that satisfy this requirement:

Yahoo! R3 Dataset [25]. This is a user-song rating dataset, where there are over 300K ratings self-selected by 15,400 users in its training set, hence it is MNAR data. Besides, they collect an additional MAR test set by asking 5,400 users to rate 10 randomly displayed songs.

Coat Shopping Dataset [30]. This dataset consists of 290 users and 300 items. Each user rates 24 items by themselves, and is asked to rate 16 uniformly displayed items as the MAR test set.

We simplify the underlying rating prediction problem to binary classification in our experiments, by making rating which is $3$ or higher be positive feedback and those lower than $3$ as negative.

Our CVIB is model-agnostic, thus applicable for most models in recommendation that take embeddings to encode the events, including the shallow and deep models. In our experiments, we pick matrix factorization (MF) [21] as shallow backbone and neural collaborative filtering (NCF) [13] as the deep backbone. Both of these methods project users and items into a shared space and represent them with unique vectors, namely embeddings.

The most popular technique to debias MNAR data by involving RCTs is the inverse propensity score (IPS) method [30]. Its variants, the self-normalized IPS (SNIPS) [33], doubly robust (DR) [18] and joint learning doubly robust (DRJL) [37] are also widely used. In our experiments, we take $5\%$ of test data to learn the propensity scores via a naive Bayes estimator [30]

for IPS, SNIPS, DR, and DRJL. Applying these methods to both shallow and deep backbones, we involve muliple baselines in comparison with our MF-CVIB and NCF-CVIB. Note that only CVIB is RCT-free among all methods.

We implement all the methods on PyTorch [26]. For both the MF and NCF, we fix the embedding size of both users and items to be $4$ because in our experiments, we find when embedding size gets larger, the performance of all methods on the MAR test set decays, which may be caused by overfitting. We randomly draw $30\%$ data from the training set for validation, on which we apply a grid search for hyperparameters to pick the best configuration. Adam [20] is utilized as the optimizer for fast convergence during training, with its learning rate in $\{0.1,0.05,0.01,0.005,0.001\}$, weight decay in $\{10^{-3},10^{-4},10^{-5}\}$, and batch size in $\{128,256,512,1024,2048\}$. For NCF, we set an additional hidden layer with width $8$. Specifically for CVIB, we set the hyperparameters $\alpha\in\{2,1,0.5,0.1\}$, and $\gamma\in\{1,0.1,10^{-2},10^{-3}\}$. Since we already set weight decay for Adam, we do not apply the $\ell_{2}$-norm term on the embeddings. After finding out the best configuration on the validation set, we evaluate the trained models on the MAR test set.

The evaluation results are reported in Table 2. There are three main observations:

(1) Even if they utilize additional RCTs, the IPS, SNIPS, DR and DRJL methods sometimes work worse than the naive model. By contrast, we identify that our RCT-free CVIB method is capable of enhancing both the shallow and deep models significantly in all experiments. It indicates that the contrastive regularization on the task-aware information, contained in embeddings of factual and counterfactual events, results in improvement of model generalization ability. To further evaluate our method in terms of ranking quality, we report the results of nDCG in Table 3. CVIB shows more significant gain over the baselines than on AUC.

(2) We perform repeated experiments to quantify our CVIB’s sensitivity to the balancing term weight $\alpha$ and confident penalty term weight $\gamma$ in Eq. (18). From Fig. 3 we identify that $\alpha$ influences results significantly on COAT, while dose not make much difference on YAHOO. In general, increasing $\alpha$ enhances the test AUC, which is aligned with our argument that contrastive information term benefits in balancing model between factual and counterfactual domains and then leads to better generalization ability. The confidence penalty $\gamma$ plays less role in accuracy, but we should not set it too high to avoid underfitting.

(3) One would notice that the performance of NCF-CVIB in terms of MSE is relatively weak, while we argue that good recommendation does not necessarily rely on low MSE prediction. For instance, for the true outcomes $y=\{1,0,0,0\}$, the predictions $\hat{y}_{1}=\{0,0.1,0.1,0.1\}$ have much lower MSE than $\hat{y}_{2}=\{0.6,0.4,0.4,0.4\}$, although the former is obviously a worse prediction than the latter in terms of ranking. It turns out that NCF-CVIB overestimates the outcomes on the test set of YAHOO, nevertheless it still reaches the best ranking quality measured by AUC. In practice of recommender systems, instead of low MSE, we would rather appreciate high ranking quality, namely AUC, which demonstrates the model’s capability of finding out the positive examples.