Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation

With the great success of pre-trained language models (PLMs) in multiple NLP tasks, many researchers have proposed to incorporate the PLM in news recommendation and have achieved substantial gain Zhang et al. (2021b); Jia et al. (2021); Wu et al. (2021b). For example, Zhang et al. (2021b) proposed UNBERT, which utilizes the PLM to capture multi-grained user-news matching signals at both word-level and news-level. Wu et al. (2021b) proposed a state-of-the-art PLM-based news recommendation method named PLM-NR, which instantiates the news encoder with a PLM to capture the deep semantic information in news texts and generate high-quality news representations. However, these methods simply finetune the PLM with the news recommendation task, the supervision from which may be insufficient to fill the domain gap between the generic pre-training corpus and downstream news texts Gururangan et al. (2020); Madan et al. (2021). Besides, PLMs usually contain a large number of parameters and have high computational overhead. Different from these methods, our approach can better mitigate the domain shift problem with an additional domain-specific post-training task and further reduce the computational cost with a two-stage knowledge distillation method.

Finetuning a PLM has become a standard procedure for many NLP tasks Devlin et al. (2019); Raffel et al. (2020). These models are first pre-trained on large generic corpora (e.g., BookCorpus and Wikipedia) and then finetuned on the downstream task data. Even though this paradigm has achieved great success, it suffers from the known domain shift problem between the pre-training and downstream corpus Howard and Ruder (2018); Lee et al. (2019); Beltagy et al. (2019). A technique commonly used to mitigate this problem is continuing to pre-train the general PLM on additional corpora related to the downstream task Logeswaran et al. (2019); Chakrabarty et al. (2019); Han and Eisenstein (2019). For example, Gururangan et al. (2020) proposed domain-adaptive pre-training (DAPT) and task-adaptive pre-training (TAPT), which further pre-trains the PLM on a large corpus of unlabeled domain-specific text and the training text set for a given task before the task-specific finetuning, respectively. Instead of continued pre-training, we utilize the natural matching relation between different parts of a news article and design a domain-specific post-training method with a contrastive matching task between news titles and news bodies. It can make the PLM better capture the high-level semantic information in news texts and generate more discriminative news representations, which are beneficial for news recommendation.

Knowledge distillation (KD) is a technique that aims to compress a heavy teacher model into a lightweight student model while maintaining its performance Hinton et al. (2015). In recent years, many works explore compressing large-scale PLMs via KD Sun et al. (2019); Wang et al. (2020); Sun et al. (2020); Xu et al. (2020). For example, Jiao et al. (2020) proposed TinyBERT, which lets the student model imitate the intermediate and final outputs of the teacher model in both the pre-training and finetuning stages. There are also a few works that aim to distill the PLM for specific downstream tasks Lu et al. (2020); Wu et al. (2021c). For example, Wu et al. (2021c) proposed NewsBERT for intelligent news applications. A teacher-student joint distillation framework is proposed to collaboratively learn both teacher and student models. Considering that the guidance provided by a single teacher may be limited or even biased, some works propose to conduct KD with multiple teacher models Liu et al. (2020); Wu et al. (2021a). However, all these works neglect the potential domain gap between the pre-training corpus and the downstream task domain. To our best knowledge, we are the first to conduct KD during the domain adaptation of PLMs. Both domain-specific and task-specific knowledge are transferred to the student model in our two-stage knowledge distillation method. Besides, multiple teacher models are used to provide more comprehensive guidance to the student in both stages.

We first introduce the structure of the PLM-based news recommendation model used in our Tiny-NewsRec. As shown in Fig. 1(b), it consists of three major components, i.e., a news encoder, a user encoder, and a click prediction module. The news encoder aims to learn the news representation from news texts. Following the state-of-the-art PLM-based news recommendation method Wu et al. (2021b), we use a PLM to capture the deep context in news texts and an attention network to aggregate the output of the PLM. The user encoder aims to learn the user representation from the representations of the user’s last $L$ clicked news, i.e., $[\boldsymbol{n}_{1},\boldsymbol{n}_{2},...,\boldsymbol{n}_{\scriptscriptstyle L}]$. Following Wu et al. (2019a), we implement it with an attention network to select important news from the user’s historical interactions. In the click prediction module, we take the dot product of the candidate news representation $\boldsymbol{n}_{c}$ and the target user representation $\boldsymbol{u}$ as the predicted score $\hat{y}_{\scriptscriptstyle\rm{FT}}$. It is noted that our Tiny-NewsRec is decoupled from the structure of the news recommendation model. Other PLM-based news recommendation models Jia et al. (2021); Zhang et al. (2021a, b) can also be adopted.

Since directly finetuning the PLM with the downstream news recommendation task may be insufficient to fill the domain gap between the general corpus and news texts Gururangan et al. (2020); Madan et al. (2021), we propose to conduct domain-specific post-training to the PLM before the task-specific finetuning. Considering the natural matching relation between different parts of a news article, we design a self-supervised contrastive matching task between news titles and news bodies. The model framework for this task is shown in Fig. 1(a).

Given a news article, we regard its news body $nb$ as the anchor and take its news title $nt^{+}$ as the positive sample. We randomly select $N$ other news titles $[nt_{1}^{-},nt_{2}^{-},\cdots,nt_{\scriptscriptstyle N}^{-}]$ from the news pool as negative samples. We use the PLM-based news encoder to get the news body representation $\boldsymbol{h}_{nb}$ and these news title representations $[\boldsymbol{h}_{nt^{\scalebox{0.5}{+}}},\boldsymbol{h}_{nt_{1}^{\scalebox{1.0}{-}}},\boldsymbol{h}_{nt_{2}^{\scalebox{1.0}{-}}},\cdots,\boldsymbol{h}_{nt_{\scalebox{0.5}{{N}}}^{\scalebox{1.0}{-}}}]$. We adopt the InfoNCE loss Oord et al. (2018) as the contrastive loss function. It is formulated as follows:

$$\mathcal{L}_{\scriptscriptstyle\rm{DP}}=-\operatorname{log}\frac{\exp({\hat{y}_{nt^{\scalebox{0.5}{+}}}})}{\exp({\hat{y}_{nt^{\scalebox{0.5}{+}}}})+\sum_{i=1}^{\scriptscriptstyle N}\exp({\hat{y}_{nt^{\scalebox{1.0}{-}}_{\scalebox{0.5}{{i}}}}})},$$

where $\hat{y}_{nt^{\scalebox{0.5}{+}}}=\boldsymbol{h}_{nb}^{\scriptscriptstyle\rm{T}}\boldsymbol{h}_{nt^{\scalebox{0.5}{+}}}$ and $\hat{y}_{nt^{\scalebox{1.0}{-}}_{\scalebox{0.5}{{i}}}}=\boldsymbol{h}_{nb}^{\scriptscriptstyle\rm{T}}\boldsymbol{h}_{nt^{\scalebox{1.0}{-}}_{\scalebox{0.5}{{i}}}}$. As proved by Oord et al. (2018), minimizing $\mathcal{L}_{\scriptscriptstyle\rm{DP}}$ can maximize the lower bound of the mutual information between $\boldsymbol{h}_{nb}$ and $\boldsymbol{h}_{nt^{\scalebox{0.5}{+}}}$. Therefore, the post-trained PLM-based news encoder can better capture and match the high-level semantic information in news texts. It will generate more similar representations for related texts (i.e., the news body and its corresponding news title) and distinguish them from the others, which can also ease the anisotropy problem of the sentence representation generated by the PLM Gao et al. (2019); Ethayarajh (2019); Li et al. (2020a). Thus, our proposed domain-specific post-training method is beneficial to both news understanding and user interest matching in the following news recommendation task.

To achieve our goal of efficiency, we further propose a two-stage knowledge distillation method, whose framework is shown in Fig. 1. In our framework, the lightweight student model is trained to imitate the large teacher model in both its post-training stage and finetuning stage. Besides, multiple teacher models originated from different time steps of our post-training procedure are used to transfer more comprehensive knowledge to the student model in both stages.

In Stage I, we first conduct domain-specific post-training towards the teacher PLM-based news encoder (Step 1). During the post-training procedure, a copy of the current teacher news encoder is saved every $K$ steps after convergency and we save $M$ teacher models in total. Then we use these teacher models to transfer comprehensive domain-specific knowledge to the student model during its post-training (Step 2). Since these teacher models at different time steps may have different performance on an input sample, we assign an adaptive weight to each teacher for each training sample, which is measured by the cross-entropy loss between its predicted scores $\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{DP}}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}=[\hat{y}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{nt^{\scalebox{0.5}{+}}},\hat{y}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{nt_{\scalebox{0.4}{1}}^{\scalebox{1.0}{-}}},\hat{y}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{nt_{\scalebox{0.4}{2}}^{\scalebox{1.0}{-}}},\cdots,\hat{y}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{nt_{\scalebox{0.4}{{N}}}^{\scalebox{1.0}{-}}}]$ and the ground-truth label $y_{\scriptscriptstyle\rm{DP}}$. Denote the weight of the $i$-th teacher model on a given sample as $\alpha^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}$, it is formulated as follows:

$$\alpha^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}=\frac{\exp(-\operatorname{CE}(\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{DP}}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})},y_{\scriptscriptstyle\rm{DP}}))}{\sum_{j=1}^{\scriptscriptstyle M}{\exp(-\operatorname{CE}(\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{DP}}^{\scriptscriptstyle(t_{j})},y_{\scriptscriptstyle\rm{DP}}))}}.$$

To encourage the student model to make similar predictions to the best teacher model, we use a distillation loss to regularize its output soft labels, which is formulated as follows:

$$\mathcal{L}_{\scriptscriptstyle\rm{DP}}^{\scriptscriptstyle\rm{distill}}=T_{\scriptscriptstyle\rm{DP}}^{\scriptscriptstyle 2}\cdot\operatorname{CE}(\sum_{i=1}^{M}{\alpha^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}\hat{\boldsymbol{y}}_{\scriptscriptstyle{\rm{DP}}}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}}/T_{\scriptscriptstyle\rm{DP}},\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{DP}}^{\scriptscriptstyle(s)}/T_{\scriptscriptstyle\rm{DP}}).$$

$T_{\scriptscriptstyle\rm{DP}}$ is a temperature hyper-parameter that controls the smoothness of the predicted probability distribution of the teacher models. Besides, since we expect the representations generated by the student model and these teacher models to be similar in a unified space, we propose to apply an additional embedding loss to align these representations. The embedding loss between the $i$-th teacher model and the student model is formulated as follows:

	$\displaystyle\mathcal{L}^{\scriptscriptstyle{\rm{emb}_{\scalebox{0.5}{{i}}}}}_{\scriptscriptstyle\rm{DP}}=$	$\displaystyle\operatorname{MSE}(\boldsymbol{W}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}\boldsymbol{h}_{nt}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}+\boldsymbol{b}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})},\boldsymbol{h}_{nt}^{\scriptscriptstyle(s)})+$
		$\displaystyle\operatorname{MSE}(\boldsymbol{W}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}\boldsymbol{h}_{nb}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}+\boldsymbol{b}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})},\boldsymbol{h}_{nb}^{\scriptscriptstyle(s)}),$

where $\boldsymbol{W}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}$ and $\boldsymbol{b}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}$ are the learnable parameters in the additional linear projection layer of the $i$-th teacher model. The overall embedding loss is the weighted summation of all these embedding losses, i.e., $\mathcal{L}^{\scriptscriptstyle\rm{emb}}_{\scriptscriptstyle\rm{DP}}=\sum_{i=1}^{\scriptscriptstyle M}\alpha^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}\mathcal{L}^{\scriptscriptstyle{\rm{emb}_{\scalebox{0.5}{{i}}}}}_{\scriptscriptstyle\rm{DP}}$. The loss function for the student model in Stage I is the summation of the distillation loss, the overall embedding loss, and its InfoNCE loss in our domain-specific post-training task, which is formulated as follows:

$$\mathcal{L}_{\scriptscriptstyle 1}=\mathcal{L}^{\scriptscriptstyle\rm{distill}}_{\scriptscriptstyle\rm{DP}}+\mathcal{L}^{\scriptscriptstyle\rm{emb}}_{\scriptscriptstyle\rm{DP}}+\mathcal{L}_{\scriptscriptstyle\rm{DP}}^{\scriptscriptstyle(s)}.$$

Next, in Stage II, we first finetune these $M$ post-trained teacher news encoders with the news recommendation task (Step 3). Then they are used to transfer rich task-specific knowledge to the student during its finetuning (Step 4). Similar to Stage I, we assign a weight $\beta^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}$ to each finetuned teacher model based on its cross-entropy loss given an input sample of the news recommendation task and apply the following distillation loss to adjust the output of the student model during its finetuning:

	$$\displaystyle\beta^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}=\frac{\exp(-\operatorname{CE}(\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{FT}}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})},y_{\scriptscriptstyle\rm{FT}}))}{\sum_{j=1}^{\scriptscriptstyle M}{\exp(-\operatorname{CE}(\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{FT}}^{\scriptscriptstyle(t_{j})},y_{\scriptscriptstyle\rm{FT}}))}},$$
	$$\displaystyle\mathcal{L}^{\scriptscriptstyle\rm{distill}}_{\scriptscriptstyle\rm{FT}}=T_{\scriptscriptstyle\rm{FT}}^{\scriptscriptstyle 2}\cdot\operatorname{CE}(\sum_{i=1}^{M}{\beta^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{FT}}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}}/T_{\scriptscriptstyle\rm{FT}},\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{FT}}^{\scriptscriptstyle(s)}/T_{\scriptscriptstyle\rm{FT}}),$$

where $\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{FT}}$ denotes the predicted score of the model on the news recommendation task and $T_{\scriptscriptstyle\rm{FT}}$ is another temperature hyper-parameter. We also use an additional embedding loss to align both the news representation and the user representation of the student model and the teacher models, which is formulated as follows:

$\displaystyle\begin{aligned} \mathcal{L}^{\scriptscriptstyle\rm{emb}}_{\scriptscriptstyle\rm{FT}}=\sum_{i=1}^{M}\beta^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}[&\operatorname{MSE}(\boldsymbol{W}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{n}\boldsymbol{n}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}+\boldsymbol{b}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{n},\boldsymbol{n}^{\scriptscriptstyle(s)})+\\[-8.53581pt] &\operatorname{MSE}(\boldsymbol{W}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{u}\boldsymbol{u}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}+\boldsymbol{b}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{u},\boldsymbol{u}^{\scriptscriptstyle(s)})],\end{aligned}$

where $\boldsymbol{W}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{n}$, $\boldsymbol{b}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{n}$ and $\boldsymbol{W}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{u}$, $\boldsymbol{b}^{\scriptscriptstyle(t_{\scalebox{0.5}{{i}}})}_{u}$ are the learnable parameters used to project the news representations and the user presentations learned by the $i$-th teacher model into a unified space, respectively. The student model is also tuned to minimize the cross-entropy loss between its predicted score $\hat{\boldsymbol{y}}_{\scriptscriptstyle{\rm{FT}}}^{\scriptscriptstyle(s)}$ and the ground-truth label $y_{\scriptscriptstyle\rm{FT}}$ of the news recommendation task, i.e., $\mathcal{L}_{\scriptscriptstyle\rm{FT}}^{\scriptscriptstyle(s)}=\operatorname{CE}(\hat{\boldsymbol{y}}_{\scriptscriptstyle\rm{FT}}^{\scriptscriptstyle(s)},y_{\scriptscriptstyle\rm{FT}})$. The overall loss function for the student model in Stage II is the summation of the distillation loss, the embedding loss, and its finetuning loss, which is formulated as follows:

$$\mathcal{L}_{\scriptscriptstyle 2}=\mathcal{L}^{\scriptscriptstyle\rm{distill}}_{\scriptscriptstyle\rm{FT}}+\mathcal{L}^{\scriptscriptstyle\rm{emb}}_{\scriptscriptstyle\rm{FT}}+\mathcal{L}_{\scriptscriptstyle\rm{FT}}^{\scriptscriptstyle(s)}.$$

We conduct experiments with three real-world datasets, i.e., MIND, Feeds, and News. MIND is a public dataset for news recommendation Wu et al. (2020), which contains the news click logs of 1,000,000 users on the Microsoft News website in six weeks. We use its public training set, validation set, and test set for experiments³³3We randomly choose 1/2 samples from the original training set as our training data due to the limit of training speed.. Feeds is also a news recommendation dataset collected on the Microsoft News App from 2020-08-01 to 2020-09-01. We use the impressions in the last week for testing and randomly sampled 20% impressions from the training set for validation. News contains news articles collected on the Microsoft News website from 2020-09-01 to 2020-10-01, which is used for our domain-specific post-training task. Detailed statistics of these datasets are summarized in Table 2.

In our experiments, following PLM-NR Wu et al. (2021b), we apply the pre-trained UniLMv2 Bao et al. (2020) to initialize the PLM in the news encoder due to its superior text modeling capability. The dimensions of the news representation and the user representation are both 256. The temperature hyper-parameters $T_{\scriptscriptstyle\rm{DP}}$ and $T_{\scriptscriptstyle\rm{FT}}$ are both set to 1. A copy of the teacher model is saved every $K=500$ steps during post-training and the number of teacher models $M$ is set to 4. We use the Adam optimizer Kingma and Ba (2015) for training. The detailed experimental settings are listed in the Appendix. All the hyper-parameters are tuned on the validation set. Following Wu et al. (2020), we use AUC, MRR, and nDCG@10 to measure the performance of news recommendation models. We independently repeat each experiment 5 times and report the average results with standard deviations.

In this section, we compare the performance of the 12-layer teacher model PLM-NR₁₂ (DP) which is domain-specifically post-trained before finetuning, and the student models trained with our Tiny-NewsRec with the following baseline methods:

• •

  PLM-NR (FT) Wu et al. (2021b), the state-of-the-art PLM-based news recommendation method which applies the PLM to the news encoder and directly fine-tunes it with the news recommendation task. We compare the performance of its 12-layer version and its variant using the first 1, 2, or 4 layers of the PLM.

• •

  PLM-NR (DAPT), a variant of PLM-NR which first adapts the PLM to the news domain via domain-adaptive pre-training Gururangan et al. (2020). It continues to pre-train the PLM on a corpus of unlabeled news domain texts and then finetunes it with the news recommendation task.

• •

  PLM-NR (TAPT), another variant of PLM-NR which first adapts the PLM to the downstream task with task-adaptive pre-training Gururangan et al. (2020). It continues to pre-train the PLM on the unlabeled news set provided along with the downstream training data and then finetunes it with the news recommendation task.

• •

  TinyBERT Jiao et al. (2020), a state-of-the-art two-stage knowledge distillation method for compressing the PLM which conducts knowledge distillation in both the pre-training stage and the finetuning stage. For a fair comparison, we use the PLM-NR₁₂ (DP) as the teacher model.

• •

  NewsBERT Wu et al. (2021c), a PLM knowledge distillation method specialized for intelligent news applications which jointly trains the student model and the teacher model during finetuning. For a fair comparison, we use the 12-layer domain-specifically post-trained news encoder to initialize the teacher model.

Table 3 shows the performance of all these methods on the MIND and Feeds datasets. From the results, we have the following observations. First, both PLM-NR₁₂ (DAPT) and PLM-NR₁₂ (TAPT) outperform PLM-NR₁₂ (FT). It validates that continued pre-training on the corpus related to the downstream task can mitigate the domain shift problem to some extent. Second, our PLM-NR₁₂ (DP) achieves the best performance among all 12-layer models. This is because our proposed self-supervised domain-specific post-training task can help the PLM better capture the semantic information in news texts and generate more discriminative news representations, which is beneficial to the news understanding and user interest matching in the news recommendation task. Third, compared with state-of-the-art knowledge distillation methods (i.e., NewsBERT and TinyBERT), our Tiny-NewsRec achieves the best performance in all 1-layer, 2-layer, and 4-layer student models, and our further t-test results show the improvements are significant at $p<0.01$ (by comparing the models with the same number of layers). This is because the student model can better adapt to the news domain with supervision from the domain-specifically post-trained teacher models in Stage I, and task-specific knowledge is also transferred to it during the knowledge distillation in Stage II. Finally, our Tiny-NewsRec even achieves comparable performance with the teacher model PLM-NR₁₂ (DP) while having much fewer parameters and lower computational overhead. This is because these multiple teacher models originated from different time steps of the post-training procedure may complement each other and provide more comprehensive knowledge to the student model in both stages.

To better understand where the performance improvement of our approach comes from, we further compare our Tiny-NewsRec with the following methods which use multiple teacher models:

• •

  Ensemble-Teacher, which is the ensemble of the multiple 12-layer teacher models used by Tiny-NewsRec. The average predicted score of these teacher models is used for evaluation.

• •

  TinyBERT-MT and NewsBERT-MT, the modified version of TinyBERT Jiao et al. (2020) and NewsBERT Wu et al. (2021c), which utilize the multiple teacher models used by Tiny-NewsRec. Each teacher model is adaptively weighted according to its performance on the input training sample, which is the same as the one used in our two-stage knowledge distillation method.

• •

  MT-BERT Wu et al. (2021a), which jointly finetunes the student model and multiple teacher models with different PLMs on the downstream news recommendation task.

Table 4 shows the performance of the ensemble teacher models and the 4-layer student models on the Feeds dataset. Comparing with the results of PLM-NR₁₂ (DP) in Table 3, we first find that the simple ensemble of multiple teacher models cannot bring much performance gain. The reason is that these teachers are treated equally during testing. However, in our Tiny-NewsRec, for each training sample, we assign an adaptive weight to each teacher model based on their performance. The student can always learn more from the best teacher model on each sample and receive more comprehensive knowledge. Second, even with the same teacher models, Tiny-NewsRec still outperforms TinyBERT-MT and NewsBERT-MT. This is because we are the first to use multiple teacher models to transfer domain-specific knowledge to the student before the task-specific finetuning, which can help the student model better adapt to the news domain. Besides, we find that MT-BERT achieves the worst performance among all the compared methods. It verifies that the multiple teacher models originating from different time steps of our post-training procedure can provide more comprehensive knowledge than these jointly finetuned teacher models with different PLMs used in MT-BERT.

In this subsection, we conduct experiments to explore the impact of the number of teacher models in our Tiny-NewsRec. We vary the number of teacher models $M$ from 1 to 6 and compare the performance of the 4-layer student model on the Feeds dataset⁴⁴4The results on the MIND dataset show similar trends and are placed in the Appendix.. The results are shown in Fig. 2. From the results, we find that the performance of the student model first greatly improves with the number of teacher models. This is because these teacher models at different time steps of the post-training procedure usually can complement each other. With more teacher models, the student model can receive more comprehensive knowledge and obtain better generalization ability. However, increasing the number of teacher models can only bring marginal improvement when $M$ is larger than 4, which may reach the upper bound of the performance gain brought by the ensemble of multiple teachers. Thus we set $M$ to 4 in our Tiny-NewsRec as a balance between the model performance and the additional training cost of these teacher models.

In this subsection, we further conduct several experiments to verify the effectiveness of each stage in our two-stage knowledge distillation method. We compare the performance of the 4-layer student model distilled with our Tiny-NewsRec and its variant with one stage removed on the Feeds dataset⁴. The results are shown in Fig. 3. From the results, we first find that the knowledge distillation in Stage II plays a critical role in our approach as the performance of the student model declines significantly when it is removed. This is because the guidance from the teacher models in the second stage such as learned news and user representations can provide much more useful information than the one-hot ground-truth label, which encourages the student model to behave similarly to the teacher models in the news recommendation task. The complement between multiple teacher models also enables the student model to achieve better generalization ability. Second, the performance of the student model also declines after we remove Stage I. This is because our self-supervised domain-specific post-training task can make the PLM better adapt to the news domain and generate more discriminative news representations. The multiple teacher models can also transfer useful domain-specific knowledge to the student model during its domain adaptation.

In this subsection, we further explore the impact of each part of the overall loss function in our two-stage knowledge distillation method, i.e., the distillation loss ($\mathcal{L}^{\scriptscriptstyle\rm{distill}}_{\scriptscriptstyle\rm{DP}}$ and $\mathcal{L}^{\scriptscriptstyle\rm{distill}}_{\scriptscriptstyle\rm{FT}}$), the embedding loss ($\mathcal{L}^{\scriptscriptstyle\rm{emb}}_{\scriptscriptstyle\rm{DP}}$ and $\mathcal{L}^{\scriptscriptstyle\rm{emb}}_{\scriptscriptstyle\rm{FT}}$), and the target loss ($\mathcal{L}_{\scriptscriptstyle\rm{DP}}^{\scriptscriptstyle(s)}$ and $\mathcal{L}_{\scriptscriptstyle\rm{FT}}^{\scriptscriptstyle(s)}$). We compare the performance of the student models distilled with our Tiny-NewsRec approach and its variant with one part of the overall loss function removed. The results on the Feeds dataset are shown in Fig. 4. From the results, we have several findings. First, the distillation loss is the most essential part of the overall loss function as the performance of the student model drops significantly after it is removed. This is because the distillation loss can force the student model to make similar predictions as the teacher model, which directly decides the performance of the student model on the news recommendation task. In addition, the embedding loss is also important in our approach. It may be because the embedding loss aligns the news representations and the user representations learned by the student model and the teacher models, which can help the student model better imitate the teacher models. Besides, the target loss is also useful for the training of the student model. This may be because these finetuned teacher models will still make some mistakes in certain training samples. The supervision from the ground-truth label is still necessary for the student model.

In this subsection, we conduct experiments to evaluate the efficiency of the student models distilled with our Tiny-NewsRec. As in news recommendation, encoding news with the PLM-based news encoder is the main computational overhead, we measure the inference speed of the model in terms of the number of news that can be encoded per second with a single GPU. We also measure the model size by the number of parameters. The evaluation results of the 1-layer, 2-layer, and 4-layer student models and the 12-layer teacher model are shown in Fig. 5. The results show that our Tiny-NewsRec can reduce the model size by 50%-70% and increase the inference speed by 2-8 times while achieving better performance. These results verify that our approach can improve the effectiveness and efficiency of the PLM-based news recommendation model at the same time. It is noted that the student model distilled with other knowledge distillation methods (e.g., TinyBERT and NewsBERT) can achieve the same inference speed as Tiny-NewsRec since the structure of the final student model is the same. However, our Tiny-NewsRec can get much better performance as shown in Table 3 and 4.