Continual Learning for Generative Retrieval
over Dynamic Corpora

To incrementally index new documents, we need to encode new documents into docids with low computational cost, while learning associations between new documents and their docids.

During continual indexing of new documents, it is important for the GR models to prevent forgetting the retrieval ability. We are inspired by the fact that humans benefit from previous similar experiences when taking actions (Clark and Haviland, 1974; Haviland and Clark, 1974; Clark et al., 1977) and propose a memory-augmented learning mechanism to build meaningful connections between new and old documents. Specifically, we first construct a memory bank with similar documents for each new session and replay the process of indexing them alongside the indexing of new documents. Then, we leverage a query generator model to sample pseudo-queries for documents and the resulting query-docid pairs are employed to maintain the retrieval ability. The overall learning process is visualized in Figure 2 (b).

Finally, we group the similar documents of each document in $\mathcal{D}_{t}$ to construct a specific memory bank $\mathcal{B}_{t}$ at the session $t$. Note that the memory bank is dynamically updated at each new session.

For each new session $t$, we generate pseudo-queries for each document in $\mathcal{D}_{t}$ and $\mathcal{B}_{t}$ via $\Theta_{qg}$ and denote the obtained pairs of pseudo-queries and documents as $\mathcal{P}_{t}^{\mathcal{D}_{t}}$ and $\mathcal{P}_{t}^{\mathcal{B}_{t}}$, respectively. Given the meta model parameters $\Theta_{t-1}$ of the GR model, we also apply MLE to maximize the likelihood of a relevant docid conditioned on each pseudo query in $\mathcal{P}_{t}^{\mathcal{D}_{t}}$ and $\mathcal{P}_{t}^{\mathcal{B}_{t}}$, denoted as,

where $(q_{j},\smash{d_{t}^{\psi(q_{j})}})\in\{\smash{\mathcal{P}_{t}^{\mathcal{D}_{t}}},\smash{\mathcal{P}_{t}^{\mathcal{B}_{t}}}\}$ , $\psi(q_{j})$ is the index of the relevant document to $q_{j}$ and $j\in\{1,\dots,|\{\smash{\mathcal{P}_{t}^{\mathcal{D}_{t}}},\smash{\mathcal{P}_{t}^{\mathcal{B}_{t}}}\}|\}$. $\smash{u_{d_{t}^{\psi(q_{j})}}}\in\{\smash{\mathcal{U}_{t}^{\mathcal{D}_{t}}},\smash{\mathcal{U}_{t}^{\mathcal{B}_{t}}}\}$ is the relevant docid.

In the training phase, we sequentially train the GR model on each session $t$ by combining the objective for indexing and retrieval, i.e.,

where $\lambda$ is a hyper-parameter. The elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) loss $\mathcal{L}_{EWC}^{t}$ is used to regularize the model parameters, via the weighted distance between $\Theta_{t-1}$ and $\Theta_{t}$,

where $F$ is the Fisher information matrix (Kirkpatrick et al., 2017), and $F_{l}$ denotes each model parameter.

To facilitate the study of continual document learning for GR, we build two benchmark datasets, i.e., CDI-MS and CDI-NQ, from MS MARCO Document Ranking (Nguyen et al., 2016) and Natural Questions (NQ) (Kwiatkowski et al., 2019), respectively. MS MARCO contains $367,013$ query-document training pairs, $3,213,835$ documents, and $5,192$ queries in the dev set. NQ contains 307k query-document training pairs, 231k documents, and 7.8k queries in the dev set. We report the performance results on the dev sets as both MS MARCO Document Ranking and NQ leaderboard limit the frequency of submissions (Tay et al., 2022; Mehta et al., 2022).

To mimic the new arrival of documents in MS MARCO and NQ, we first randomly sample $60\%$ documents from the whole document set as the base documents $\mathcal{D}_{0}$, and leverage their corresponding relevance labels to construct the query-document pairs $\mathcal{P}_{0}^{\mathcal{D}_{0}}$. Then, we randomly sample $10\%$ documents from the remaining document set as the new document set, and this operation is repeated for 4 times to obtain $\mathcal{D}_{1}$–$\mathcal{D}_{4}$. The test query set is defined as follows: (i) for a single query set, all dev queries are denoted as $\mathcal{Q}^{\mathit{test}}$, and (ii) for sequential query set, we sample $60\%,10\%,10\%,10\%$ and $10\%$ queries from the whole dev query set as $\mathcal{Q}^{\mathit{test}}_{0}$, …, $\mathcal{Q}^{\mathit{test}}_{4}$, respectively.

The evaluation metric $g(\cdot)$ for IR in Section 2 is usually taken to be mean reciprocal rank (MRR@$N$), recall (R@$N$), hit ratio (Hits@$N$) and top-$N$ retrieval accuracy (ACC$@N$). Following (Tay et al., 2022; Mehta et al., 2022; Wang et al., 2022; Zhou et al., 2022), we show the continual results in terms of MRR$@10$ and HIT$@10$ for CDI-MS and CDI-NQ, respectively. By conducting further analyses, we find that the relative order of different models on other IR metrics is quite consistent with that on the MRR$@10$ and Hits$@10$.

For IPQ, the length $M$ of PQ codes is 24, the number of clusters $K$ is 256, and the dimension of vectors $D$ is 768. For the contrastive loss in the document encoder, $\ell_{\mathit{min}}$ and $\ell_{\mathit{max}}$ for sampling phrase-level spans are 4 and 16, respectively. For sentence-level spans, $\ell_{\mathit{min}}$ and $\ell_{\mathit{max}}$ are 16 and 64, respectively. For paragraph-level spans, $\ell_{\mathit{min}}$ and $\ell_{\mathit{max}}$ are 64 and 128, respectively. The $\alpha$ and $\beta$ in the beta distribution are 4 and 2, respectively, which skews sampling towards longer spans. The number of spans sampled per granularity $G$ is 5. For the memory-augmented learning mechanism, the repeat time $c$ is 10, the probability $p$ is 0.2, the scale $r$ is 0.2, and $\lambda$ is 0.5.

To train the document encoder in IPQ, we initialize the document encoder from the official BERT’s checkpoint. We use a learning rate of $5e^{-5}$ and Adam optimizer (Kingma and Ba, 2015) with a linear warmup over the first 10% steps. Long input documents are truncated into several chunks with a maximum length of 512. The hyper-parameter of $\tau$ is 0.1. We train for 6 epochs on four NVIDIA Tesla A100 40GB GPUs.

The GR baselines and all variants of CLEVER, are based on the transformer-based encoder-decoder architecture, where the hidden size is 768, the feed-forward layer size is 3072, the number of transformer layers is 12, and the number of self-attention heads is 12, for both encoder and decoder. We implement the generative model in PyTorch based on Huggingface’s Transformers library. We initialize the parameters of the encoder-decoder architecture from the official checkpoint of T5_base  (Raffel et al., 2020). We use a learning rate of $3e^{-5}$ and Adam optimizer with the warmup technique, where the learning rate increases over the first 10% of batches, and then decays linearly to zero. The max length of the input is 512, the label smoothing is 0.1, the weight decay is 0.01, and the gradient norm clipping is 0.1. We train in batches of 8192 tokens on four NVIDIA Tesla A100 40GB GPUs. At inference time, we adopt constrained beam search (De Cao et al., 2020) to decode the docids with 24 timesteps and 15 beams. To train the query generator, we also initialize the parameters from the official checkpoint of T5_base  (Raffel et al., 2020), with a learning rate of $5e^{-4}$. For each new document, we adopt beam search to decode the pseudo queries with utmost 32 timesteps and 10 beams.

Incremental performance on a single query set. The performance comparisons between CLEVER and baselines on the single query set are shown in Table 1. BM25 exhibits better performance than DPR and the underlying reason may be that BM25 is a data-independent probabilistic model, which renders it adaptable in the face of dynamic corpora. And the BASE method suffers a significant drop as new documents are added. By assigning new documents with atomic docids and using sampled old documents, DSI++ shows slight improvements over BASE. These results show that continual document learning for GR is a non-trivial challenge.

When we look at variants of CLEVER in terms of IPQ, we find that: (i) CLEVER_PQ performs worse than CLEVER_atomic which updates the embeddings for each individual docid (also used in DSI++), showing that it is difficult for the GR models to accommodate to new documents without updating docids. However, as shown in Section 5.4, for CLEVER_atomic, its memory continues to grow with the increase of documents and the time consumption of each update step increases with the number of steps. (ii) CLEVER_PQ+Re and CLEVER_PQ+Dis+Re give the worst performance. The reason might be that re-clustering old and new documents changes the previously learned centroids and thus the docids of old documents, which makes the learned document-docid mapping invalid. (iii) The improvements of CLEVER_PQ+Dis over CLEVER_PQ demonstrate the need to learn discriminative document representations and quantization centroids. (iv) CLEVER_PQ+Dis+ad and CLEVER_PQ+Dis+md outperform CLEVER_PQ+Dis, showing that incorporating updates to old centroids and introducing new centroids could facilitate the assimilation of new documents.

When we look at variants of CLEVER with different learning mechanisms, we observe that: (i) CLEVER^-MLE(q) performs worse than CLEVER^-MLE(d-) and CLEVER^-EWC, showing that constructing pairs of pseudo queries and docids and supplementing them during continual indexing contributes to preventing forgetting for the retrieval ability. (ii) CLEVER^Random shows the worst performance. Randomly selected old documents do not provide insights into the new documents and may introduce noise for continual indexing. (iii) CLEVER^-EWC performs worse than CLEVER, showing the effectiveness of limiting the scope of model updates.

Finally, CLEVER achieves the best performance. The results imply that applying an adaptive update strategy for PQ codes can assign effective docids to new documents without changing the old docids. And rehearsing old similar documents and generating pseudo queries, can actively absorb knowledge from new documents while preserving previously learned retrieval ability.

Incremental performance on a sequential query set. The performance comparisons between CLEVER and the baselines on the sequential query set are shown in Table 2 (recall that the metrics were introduced in Section 2). The relative order of different models under this setting in terms of VERT is quite consistent with that on the previous setting of a single query set. For the evaluation metrics that measure the performance across different sessions in terms of AP, BWT and FWT, the full version of CLEVER achieves the best performance, again demonstrating the effectiveness of the proposed IPQ and learning mechanisms.

Non-incremental performance. To assess the performance of CLEVER before getting into the incremental aspect, we evaluate the performance of $\mathcal{Q}^{\mathit{test}}$ on $\mathcal{D}_{0}$ under the single query set setting. As shown in Table 3, we see that: (i) Compared with traditional IR models, CLEVER and existing generative retrieval methods achieve better performance, indicating the effectiveness of integrating different components into a single consolidated model. (ii) CLEVER achieves better results than existing generative retrieval models, demonstrating that the two-step iterative process to learn discriminative PQ codes as docids contributes to the retrieval effectiveness.

To assess catastrophic forgetting of proposed methods, we show how the performance of the base session $\mathcal{D}_{0}$ and first incremental learning session $\mathcal{D}_{1}$ varies over the training process on the remaining sessions. For the indexing task, we evaluate the overall indexing accuracy of $\mathcal{D}_{0}$ and $\mathcal{D}_{1}$, i.e., we take a document as input of the GR model, if the generated sequence exactly matches with the correct docid, we treat the document as a positive sample. Otherwise, the document is a negative sample. For the retrieval task, we evaluate the performance of $\mathcal{Q}_{0}^{\mathit{test}}$ and $\mathcal{Q}_{1}^{\mathit{test}}$, i.e., MRR@10. See Figure 3.

We observe that: (i) CLEVER_PQ, CLEVER^-MLE(d-) as well as CLEVER^-MLE(q) suffer from catastrophic forgetting. Applying IPQ and the memory-augmented learning mechanism separately does not provide sufficient assurance for the model to perform well during continual document learning. (ii) DSI++ underperforms CLEVER by a large margin. A possible reason is that the atomic integers used in DSI++ as docids are difficult to quickly adapt to new documents and have a large impact on the docids of old documents, which may result in the loss of previously learned knowledge. (iii) Compared to CLEVER, the phenomenon of catastrophic forgetting is not as well mitigated in CLEVER^-MLE(d-) and CLEVER^-MLE(q), which underlines the importance of rehearsing old documents and the generated pseudo-queries. And (iv) CLEVER almost avoids catastrophic forgetting on both indexing and retrieval tasks, showing its effectiveness in a dynamic setting.

Positive forward knowledge transfer is an essential ability during continual document learning including indexing and retrieval. Therefore, in this section, we explore the forward transfer ability of the CLEVER model, i.e., transferring knowledge from old documents to new documents. For CLEVER without initialization from the previous sessions we write CLEVER_init and for individually fine-tuning the GR model on each session we write INDIVIDUAL. Table 4 displays the results. We see that: (i) CLEVER consistently and significantly outperforms INDIVIDUAL in the last four sessions. The underlying reason may be that CLEVER transfers old knowledge to new settings when continuously indexing new documents. INDIVIDUAL learns the indexing and retrieval tasks from each new session independently, which has a small size of data. (ii) The performance improvements of CLEVER over CLEVER_init further demonstrate the need for prior initialization in CLEVER, i.e., initializing new model parameters from the last parameters.

We evaluate the effectiveness and efficiency of different models on $\mathcal{Q}_{0}^{\mathit{test}}$ in terms of VERT(%) after training all sessions. Regarding training time, we compare the overall training time by the end of the sequential training. For memory footprint, we compute the disk space occupied by the model at the training end. Figure 4 shows the relative memory and training time, i.e., the memory ratio and the time ratio of these methods with respect to BASE, respectively.

Figure 4 (a) shows the effectiveness-memory trade-off. We find that: (i) Traditional IR models (BM25 and DPR) consume much larger memory footprints than generative IR models, without discernible advantages in retrieval performance. This result indicates the significant memory consumption of re-computing representations and re-indexing for new documents. (ii) Although CLEVER_atomic and DSI++ are much more effective than the BASE model and CLEVER_PQ+Re, they suffer from severe memory inefficiency since they need the large softmax output space that comes with atomic docids and the embedding for each individual docid must be added to the model as new documents arrive. (iii) CLEVER performs best in effectiveness and is almost as efficient as the BASE model. CLEVER only occupies a small amount of additional memory compared to BASE, which does not grow over sessions.

Figure 4 (b) shows the effectiveness-training time trade-off. We observe that: (i) BM25 exhibits a swift training process. However, its performance may be deemed suboptimal due to its vulnerability to the vocabulary mismatch issue, as well as its inability to adequately encapsulate semantic information. (ii) The BASE model achieves training acceleration at the cost of compromised performance, which suggests that maintaining effectiveness is a non-trivial challenge for GR in dynamic corpora setting. (iii) CLEVER_atomic and DSI++ sacrifice training time for effectiveness since the randomly initial embeddings require training from scratch. (iv) CLEVER_PQ+Re re-trains all the centroids every session, leading to some computational overhead. However, the performance is still not improved too much. (v) CLEVER performs best in effectiveness and requires similar training times as the BASE model. These results demonstrate that CLEVER can be well deployed in practical environments due to its high effectiveness and efficiency.