Device-cloud Collaborative Recommendation via Meta Controller

Recommendation on the basis of cloud computing (Duc et al., 2019) has achieved great success in the industrial companies like Microsoft, Amazon, Netflix, Youtube and Alibaba etc. A range of algorithms from the low-level feature interaction to the high-level semantic understanding have been developed in the last decades (Yao et al., 2017a, b). Some popular methods like Wide$\&$Deep (Cheng et al., 2016) and DeepFM (Guo et al., 2017) explore the combination among features to construct the high-order information, which reduces the burden of the subsequent classifier. Several approaches like DeepFM (Xue et al., 2017), DIN (Zhou et al., 2018) and Bert4Rec (Sun et al., 2019) resort to deep learning to hierarchically extract the semantics about user interests for recommendation (Pan et al., 2021; Zhang et al., 2022). More recent works like MIMN (Pi et al., 2019), SIM (Pi et al., 2020) and Linformer (Wang et al., 2020) solve the scalability and efficiency issues in comprehensively understanding user intention from lifelong sequences. Although the impressive performance has been achieved in various industrial recommendation applications, almost all of them depends on the rich computing power of the cloud server to support their computational cost (Sun et al., 2020a).

The cloud-based paradigm greatly depends on the network conditions and the transmission delay to respond to users’ interaction in real time (Yao et al., 2022). With the increasing capacity of smart phones, it becomes possible to alleviate this problem by deploying some lightweight AI models on the remote side to save the communication cost and access some real-time features (Zhou et al., 2019). Previous works in the area of computer vision and natural language processing have explored the possible network compression techniques, e.g., MobileNet (Sandler et al., 2018) and MobileBert (Sun et al., 2020b), and some neural architecture search approaches (Zoph and Le, 2016; Cai et al., 2018). The recent CpRec for on-device recommendation studied the corresponding compression by leveraging the adaptive decomposition and the parameter sharing scheme (Sun et al., 2020c). In comparison, EdgeRec adopted a split-deployment strategy to reduce the model size without compression, which distributed the huge id embeddings on the cloud server and sent them to the device side along with items (Gong et al., 2020). Some other works, DCCL (Yao et al., 2021) and FedRec (Yang et al., 2020), respectively leveraged the on-device learning for the model personalization and the data privacy protection. Nevertheless, the design of the on-device recommendation like EdgeRec and DCCL is sequentially appended after the cloud-based recommendation results, which might be stuck by the limited item cache.

Causal inference has been an important research topic complementary to the current machine learning area, which improves the generalization ability of AI models based on the limited data or from one problem to the next (Schölkopf et al., 2021). The broadly review of the causality can be referred to this literature (Pearl and Mackenzie, 2018) and in this paper, we specifically focus on the counterfactual inference to estimate the heterogeneous treatment effect (Künzel et al., 2019). Early studies mainly start from the statistical perspective like S-Learner and X-Learner (Künzel et al., 2019), which alleviates the treatment imbalance issue and leverages the structural knowledge. Recent works explore the combination with deep learning to leverage the advantages of DNNs in representation learning (Johansson et al., 2016; Shalit et al., 2017; Schölkopf et al., 2021). In the industrial recommendation, the idea of causal inference is widely applied for the online A/B testing, debiasing (Zhang et al., 2021) and generalization (Kuang et al., 2020). Except above works, the most related methods about our model are uplift modeling, which is extensively used to estimate the causal effect of each treatment (Olaya et al., 2020). Specifically, the constrained uplift modeling is practically important to meet the requirements of the restricted constraint for each treatment (Zhao and Harinen, 2019). In this paper, we are first to explore its application in solving the dataset absence issue of training the meta controller for device-cloud collaborative recommendation.

Given the aforementioned definitions, our goal in this paper is to construct a meta controller that manages the collaboration of the above mechanisms to more accurately recommend the items according to the user interests. Let $f_{\text{meta}}(\cdot)$ represents the model of meta controller, which inputs the same features as $f_{\text{device}}(\cdot)$ and is deployed in the device. Assuming $t=0,~{}1,~{}2$ represents invoking three recommendation mechanisms in Figure 1 respectively, we expect $f_{\text{meta}}(\cdot)$ makes a decision to minimize the following problem.

where $P_{\text{meta}}(\mathbf{X},\mathbf{Y})$ is the underlying distribution about the user interaction $\mathbf{Y}$ (click or not) on the item $\mathbf{X}$, $\ell$ is the cross-entropy loss function about the label $\mathbf{Y}$ and the model prediction, and $f_{t}(\cdot)$ corresponds to one of previous recommendation mechanisms based on the choice $t$. Although the above Eq. (1) resembles learning a Mixture-of-Expert problem (Shazeer et al., 2017), it is different as we only learn $f_{\text{meta}}(\cdot)$ excluding all pre-trained $f_{t}(\cdot)$. In addition, this problem is specially challenging, since we do not have the samples from the unknown distribution $P_{\text{meta}}(\mathbf{X},\mathbf{Y})$ to train the meta controller.

One important branch of causal inference is counterfactual inference under interventions (Pearl and Mackenzie, 2018), which evaluates the causal effect of one treatment based on the previous observational studies under different treatments. This coincidentally corresponds to the case in device-cloud collaboration to train the meta controller, where we can collect the training samples $\mathcal{D}_{t}$ of three different scenarios, the cloud-based session recommendation, the on-device recommendation, the cloud-based re-fresh, denoting as follows,

Note that, $\mathbf{H}_{(.)}$ is the user interaction in three scenarios (same to previous definitions), and $\mathbf{O}(\cdot)$ denotes the binary outcome of $\mathbf{H}_{(.)}$, which means under the treatment $t$ whether the user will click in the following $L$ exposed items $\{(x_{l},y_{l})\}_{l\in(1,2,...,L)}$ in the same session. With these three heterogeneous sample sets, we can leverage the idea of counterfactual inference to compare the causal effect of recommendation mechanisms and construct the training samples of meta controller. To be simple, consider the following estimation of Conditional Average Treatment Effect (CATE)

where we can directly treat the regular cloud-based session recommendation as the control group, on-device recommendation and re-fresh as the treatment groups. Then, by separately training a supervised model for each group, we can approximately compute the CATE of two treatments via T-learner or X-learner (Künzel et al., 2019). Specially, the recent progress in the uplift modeling combined with deep learning (Johansson et al., 2016; Shalit et al., 2017) shows the promise about the gain in the homogeneous representation learning. Empirically, we can share a backbone of representation learning and the base output network among different supervised models (Wenwei Ke, 2021).

In Figure 2, we illustrate the proposed dual-uplift modeling method that constructs the counterfactual samples to train the meta controller. As shown in the figure, we build two uplift networks along with the base network to respectively estimate the CATE of two treatments, and the base network is to build the baseline for the data $\mathcal{D}_{0}$ in the control group. Formally, the base network is defined in the following to align our illustration in Figure 2,

where $\phi_{\text{seq}}(\cdot)$ is the encoder for the historical click interactions $\mathbf{H}_{\text{cloud}}$ or $\mathbf{H}_{\text{device}}$ and $\circ$ is the function composition operator⁴⁴4https://en.wikipedia.org/wiki/Function_composition, and $g_{\text{base}}$ is named as the outcome network of the baseline session recommendation in the taxonomy of causal inference (Shalit et al., 2017). Similarly, for the treatment effect of the on-device recommendation, we follow the design of $f_{\text{device}}$ to construct an uplift network $g_{\text{uplift1}}(\cdot)$ to fit the data $\mathcal{D}_{1}$. As shown in Figure 2, its formulation is as follows,

where $g_{\text{uplift1}}(\cdot)$ is the outcome network of the on-device recommendation. Note that, $g_{\text{cloud}}(\cdot)$ and $g_{\text{device}}(\cdot)$ share the same feature encoder $\phi_{\text{seq}}(\cdot)$ to make the representation more homogeneous that satisfies the assumption of the causal inference. Finally, for the re-fresh mechanism, we use another uplift network to fit its data $\mathcal{D}_{2}$, which is denoted by the following equation,

Note that, although the features of $g_{\text{cloud}}(\cdot)$ and $g_{\text{re-fresh}}(\cdot)$ have the same forms, they are actually non-independent-identically-distributed (non-IID) due to different invoking frequencies. Therefore, we use $g_{\text{uplift2}}(\cdot)$ instead of $g_{\text{cloud}}(\cdot)$ to more flexibly model the outcome of the cloud-based re-fresh mechanism. With the above definitions, we can solve the following optimization problems to learn all surrogate models $g_{\text{cloud}}(\cdot)$, $g_{\text{device}}(\cdot)$ and $g_{\text{re-fresh}}(\cdot)$.

where $\ell_{b}(\cdot)$, $\ell_{t}^{1}(\cdot)$ and $\ell_{t}^{2}(\cdot)$ are the cross-entropy loss. After learning $g_{\text{cloud}}(\cdot)$, $g_{\text{device}}(\cdot)$ and $g_{\text{re-fresh}}(\cdot)$, we can acquire a new counterfactual training dataset by comparing the CATE of each sample under three mechanisms defined as follows.

where $\text{IND}(\cdot,\cdot,\cdot)$ means the position indicator ($[1,0,0]^{\top}$, $[0,1,0]^{\top}$, or $[0,0,1]^{\top}$) of the maximum in three CATEs, $\tau_{\text{device}}(h)=g_{\text{device}}(h)-g_{\text{cloud}}(h)$ and $\tau_{\text{re-fresh}}(h)=g_{\text{re-fresh}}(h)-g_{\text{cloud}}(h)$ if T-Learner is used. Here, the counterfactual inference is used to compute the CATE of each sample collected under different treatments. Note that, if X-Learner is used, CATE can be computed in a more comprehensive but complex manner and we kindly refer to this work (Künzel et al., 2019).

Remark about the difference of $f(\cdot)$ and $g(\cdot)$. Previous paragraphs mentioned different parameterized models e.g., $f_{\text{cloud}}(\cdot)$ and $g_{\text{cloud}}(\cdot)$, which could be classified into two categories by $f(\cdot)$ and $g(\cdot)$. Their biggest difference is whether these models will be used online. For the models described in Preliminary, they are all real-world models serving online and in most cases are developed and maintained by different teams in the industrial scenarios. For the $g(\cdot)$-style models described in this paragraph, they are all surrogate models, which are only used for the sample construction and not deployed online. Roughly, it will be better if they can be consistent with the well-optimized $f(\cdot)$-style models, but in some extreme cases they actually can be very different only if the approximate CATE is accurate enough to make the constructed $\mathcal{D}_{\text{meta}}$ contain the sufficient statistics about the distribution $P_{\text{meta}}$ in Eq. (1).

Generally, it is straightforward to train the meta controller by a single-label multi-class empirical risk minimization with the aid of the counterfactual dataset $\mathcal{D}_{\text{meta}}$ in Eq. (3) as follows,

Unfortunately, not all recommendation mechanisms in Figure 1 are cost-effective. Specially, the cloud-based recommendation re-fresh usually suffers from the restricted computing resources in the infinite feeds recommendation, yielding a hard invoking constraint in practice. This then introduces a constrained optimization problem in device-cloud collaboration and breaks the regular training of meta controller like Eq. (4). Actually, this is a classical constrained budget allocation problem, which has been well studied in last decades (Zhao et al., 2019; Meister et al., 2020). In this paper, we introduces a Label Smoothing (LS) solution to generally train the meta controller under the specific constraint on the cloud-based recommendation re-fresh. In detail, given the maximal invoking ratio $\epsilon$ about the cloud-based re-fresh, we make the relaxation for Eq. (4) via the label smoothing.

where we use $\hat{c}$ to substitute $c$ to supervise the model training and $\hat{c}$ is smoothed when the ratio of the batch of predictions is beyond $\epsilon$, $c^{-}$ is the one-hot vector corresponding to the second largest prediction in $f_{\text{meta}}(h)$. The relaxation in Eq. (5) means when the invoking ratio is over-reached, the supervision should be biased to the second possible choice $c^{-}$ for the prediction of $c=[0,0,1]^{\top}$. Note that, when the hard ratio $\epsilon$ is not reached, $\hat{c}$ still keeps the original label $c$ in the default case, namely degenerates to Eq. (4). Note that, Eq. (5) can be easily implemented in the stochastic training manner and after training until the invoking ratio is almost under $\epsilon$, we acquire the final meta controller that can be deployed online in the device-cloud collaborative recommendation. Up to now, we describe all the contents of our meta controller method and for clarity, we summarize the whole framework in Algorithm 1.