Visual Prompt Tuning in Null Space for Continual Learning

As can be seen in Figure 1, those outputs of LayerNorm and qkv-transformations corresponding to the image tokens remain unaffected by the updates to the prompts. Hence, the essence of attaining the consistency objective Eq. (10) can be turned into analyzing how to keep the output of self-attention in Eq. (3) unchanged as the prompts are updated, i.e., satisfying:

However, the nonlinear operation (i.e., $\mathrm{softmax}$) and the potential higher-order term ${\bf W}_{k}^{\top}{\bf Z}^{\top}{\bf Z}{\bf W}_{v}$ arising from ${\bf K}_{\bf Z}^{\top}{\bf V}_{\bf Z}$ in Eq. (3) complicate the direct resolution of this objective. Specifically, the non-injection property of the $\mathrm{softmax}$ function causes non-unique solutions. The multiplication between ${\bf K}_{{\bf Z}_{t+1}^{\top}}{\bf V}_{{\bf Z}_{t+1}}$ derives a quadratic term $\mathrm{LN}({\bf P}_{t}+\Delta{\bf P})^{\top}\mathrm{LN}({\bf P}_{t}+\Delta{\bf P})$, which result in difficult optimization for $\Delta{\bf P}$.

To address this issue, we propose two sufficient conditions consisting solely of linear operations. Specifically, we split the process of self-attention into two primary stages, i.e., the Affinity described by Eq. (4) and the Aggregation outlined in Eq. (6). We can achieve Eq. (11) by ensuring the consistency of each stage:

We first analyze the consistency objective of Affinity, i.e., Eq. (12), for ${\bf Z}_{t}$ and ${\bf Z}_{t+1}$:

where $\sqrt{D}$ is omitted for simplicity. Upon fulfilling Eq. (12), we can obtain ${\bf S}_{{\bf Z}_{t}}={\bf S}_{{\bf Z}_{t+1}}$, corresponding to the output of Eq. (5). Subsequently, we analyze the consistency objective of Aggregation in Eq. (13), yielding results for ${\bf Z}_{t}$ and ${\bf Z}_{t+1}$ as:

Based on Eq. (12$-$17), we are able to derive the following two equations, respectively:

Note that we expect to further deduce Eq. (18) and Eq. (19) to obtain equations among $\mathrm{LN}({\bf P}_{t})$, $\mathrm{LN}({\bf P}_{t}+\Delta{\bf P})$ and $\Delta{\bf P}$. However, due to the square root and quadratic terms in the expressions of the standard deviations $\boldsymbol{\sigma}_{{\bf P}_{t}}$ and $\boldsymbol{\sigma}_{{\bf P}_{t}+\Delta{\bf P}}$, it is difficult to express $\boldsymbol{\sigma}_{{\bf P}_{t}+\Delta{\bf P}}$ in terms of $\boldsymbol{\sigma}_{{\bf P}_{t}}$ and $\boldsymbol{\sigma}_{\Delta{\bf P}}$. Consequently, it is challenging to derive a straightforward equation that relates $\mathrm{LN}({\bf P}_{t})$ and $\mathrm{LN}({\bf P}_{t}+\Delta{\bf P})$ through $\Delta{\bf P}$.

To simplify the problem, we introduce an additional constraint on the distribution of prompts. Concretely, we require that the updated prompts ${\bf P}_{t}+\Delta{\bf P}$ retain the same distribution as ${\bf P}_{t}$, i.e., meeting the following assumption:

In this way, we can establish a straightforward mathematical relationship connecting $\mathrm{LN}({\bf P}_{t}+\Delta{\bf P})$, $\mathrm{LN}({\bf P}_{t})$ and $\Delta{\bf P}$:

Consequently, we can apply Eq. (21) to simplify Eq. (18) and (19) as:

It should be noted that in Eq. 22 and Eq. 23 , ${\bf W}_{k}$, ${\bf W}_{v}$ and $\boldsymbol{\alpha}$ are pre-trained parameters kept frozen throughout the continual learning process. ${\bf Q}_{{\bf X}_{t}}$ and ${\bf S}_{{\bf P}_{t}}$ are two matrices derived from the input ${\bf X}_{t}$. As our objective is to ensure that the above two equations remain valid for the variables ${\bf Q}_{{\bf X}_{t}}$ and ${\bf S}_{{\bf P}_{t}}$, it is sufficient to meet the following conditions, in which ${\bf W}_{v}$ can be ignored whereas ${\bf W}_{k}$ remains crucial:

Now we have obtained the simplified formulas expressed by $\Delta{\bf P}$ in Eq. (24) and (25).

To sum up, we convert the overall consistency equation Eq. (11) into two sufficient conditions Eq. (12) and (13) for Affinity and Aggregation, respectively. Consequently, we derive two corresponding consistency conditions Eq. (24) and (25) expressed by the prompt update $\Delta{\bf P}$, under the constraint of invariant prompt distribution formulated in Eq. (20). The deduced conditions can satisfy the consistency objective in Eq. (10), thereby achieving the goal of eliminating the interference of the new task on the old task for visual prompt tuning.

As ${\bf Q}_{{\bf X}_{t}}=\mathrm{LN}({\bf X}_{t}){\bf W}_{q}+\boldsymbol{b}_{q}$, Eq. (24) implies that if the (transposed) prompt update can be orthogonal to the normalized previous input image tokens ${\bf X}_{t}$ projected with a second-order transformation matrices ${\bf W}_{q}{\bf W}_{k}^{\top}$ of the pre-trained ViT, the consistency for Affinity can be guaranteed. When we ignore the normalization and the bias term in ${\bf Q}_{{\bf X}_{t}}$, Eq. (24) can be simplified as ${\bf X}_{t}{\bf W}_{q}{\bf W}_{k}^{\top}\Delta{\bf P}^{\top}=\mathbf{0}$. The simplified condition is still essentially different from the consistency condition of linear layers (i.e., Eq. (9)) and that deduced in [26] (i.e., ${\bf X}_{t}\Delta{\bf P}^{\top}=\mathbf{0}$). It indicates the interaction between the image tokens and prompts within ViT layers is fundamentally distinct, leading to a unique consistency condition related to the second-order transformation matrices ${\bf W}_{q}{\bf W}_{k}^{\top}$ of the pre-trained model. Moreover, Eq. (25) is also an essential condition served as one of the sufficient conditions for the consistency of the whole ViT layer. It implies that if the prompt update can be orthogonal to the activated attention map generated by the image queries (${\bf Q}_{\bf X}$) and prompt keys (${\bf K}_{\bf P}$), the consistency of Aggregation can be achieved.

To jointly optimize Eq. (24) and (25), we need to solve $\Delta{\bf P}$ that can meet both equations concurrently. Here, we employ a separate optimization approach to get an approximate solution efficiently. Initially, it ensures $\Delta{\bf P}^{\top}$ is orthogonal to the subspace spanned by ${\bf Q}_{{\bf X}_{t}}{\bf W}_{k}^{\top}$ to satisfy Eq. (24). Subsequently, it makes $\Delta{\bf P}$ orthogonal to the subspace spanned by ${\bf S}_{{\bf P}_{t}}$ to satisfy Eq. (25).

Specifically, we use ${\bf P}_{\mathcal{G}}$ to denote the candidate parameter update generated by the optimizer for the prompts. We aim to obtain a projection matrix $\mathcal{B}$ such that $\Delta{\bf P}=\mathcal{B}{\bf P}_{\mathcal{G}}$. Following the previously mentioned separate optimization strategy, we first ensure $\Delta{\bf P}^{\top}$ is orthogonal to ${\bf Q}_{{\bf X}_{t}}{\bf W}_{k}^{\top}$ by the projection matrix $\mathcal{B}_{1}$: $\Delta{\bf P}^{\top}=\mathcal{B}_{1}{\bf P}_{\mathcal{G}}^{\top}$. Then $\Delta{\bf P}$ is made orthogonal to ${\bf S}_{{\bf P}_{t}}$ by another projection matrix $\mathcal{B}_{2}$: $\Delta{\bf P}=\mathcal{B}_{2}{\bf P}_{\mathcal{G}}$. Therefore, the objective of the optimization turns into obtaining the two projection matrices $\mathcal{B}_{1}$ and $\mathcal{B}_{2}$ to satisfy Eq. (24) and (25). Inspired by the null-space projection method [36], the bases of $\mathcal{B}_{1}$ and $\mathcal{B}_{2}$ correspond to the null-space bases of ${\bf Q}_{{\bf X}_{t}}{\bf W}_{k}^{\top}$ and ${\bf S}_{{\bf P}_{t}}$, respectively. We use ${\bf U}_{1,0}\in\mathbb{R}^{D\times R_{1}}$ and ${\bf U}_{2,0}\times\mathbb{R}^{M\times R_{2}}$ to denote the bases of the null spaces for ${\bf Q}_{{\bf X}_{t}}{\bf W}_{k}^{\top}$ and ${\bf S}_{{\bf P}_{t}}$, where $R_{1}$ and $R_{2}$ indicate their nullities. ${\bf U}_{1,0}$ and ${\bf U}_{2,0}$ can be obtained from the right singular vectors associated with the zero singular values, through the process of singular value decomposition (SVD) applied by $\mathrm{SVD}(({\bf Q}_{{\bf X}_{t}}{\bf W}_{k}^{\top})^{\top}{\bf Q}_{{\bf X}_{t}}{\bf W}_{k}^{\top})$ and $\mathrm{SVD}({\bf S}_{{\bf P}_{t}}^{\top}{\bf S}_{{\bf P}_{t}})$, respectively. In this way, we get the projection matrices $\mathcal{B}_{1}={\bf U}_{1,0}{\bf U}_{1,0}^{\top}\in\mathbb{R}^{D\times D}$ and $\mathcal{B}_{2}={\bf U}_{2,0}{\bf U}_{2,0}^{\top}\in\mathbb{R}^{M\times M}$, which are the solutions enabling $\Delta{\bf P}$ to jointly satisfy Eq. (24) and (25):

For the constraint Eq. (20), we incorporate an additional loss function aimed at penalizing the drift of prompt distribution, hence realizing a relaxed version of this constraint:

In Eq. (27), $\boldsymbol{\mu}_{{\bf P}_{t}}$ and $\boldsymbol{\sigma}_{{\bf P}_{t}}$ represent the target prompt distribution obtained in task $\mathcal{T}_{t}$, while $\boldsymbol{\mu}_{{\bf P}_{t+1}}$ and $\boldsymbol{\sigma}_{{\bf P}_{t+1}}$ denote the distribution to be optimized in task $\mathcal{T}_{t+1}$.

To sum up, we use Eq. (26) to realize Eq. (24) and (25), and use Eq. (27) to meet Eq. (20), thereby achieving the consistency objective Eq. (10) for anti-forgetting. We provide a full algorithm of our approach in the appendix A .

We further extend the consistency conditions Eq. (24) and (25) to multi-head self-attention, a common feature in current transformer-based models. Suppose there are $H$ heads and $d=D/H$ represents the dimension of each token in a head. We use ${\bf Q}_{{\bf X}_{t}.h}\in\mathbb{R}^{N\times d}$, ${\bf W}_{k.h}\in\mathbb{R}^{D\times d}$ and ${\bf S}_{{\bf P}_{t}.h}\in\mathbb{R}^{N\times M}$ to denote the corresponding matrices in Eq. (24) and (25) for the $h$-th head, respectively. The objective is to ensure these conditions are met across all heads, i.e., ${\bf Q}_{{\bf X}_{t}.h}{\bf W}_{k.h}^{\top}\Delta{\bf P}^{\top}=\mathbf{0}$ and ${\bf S}_{{\bf P}_{t}.h}\Delta{\bf P}=\mathbf{0}$, $\forall h\in\{1,2,\cdots,H\}$. Let $\mathbf{\Omega}_{1,t}=[{\bf Q}_{{\bf X}_{t}.1}{\bf W}_{k.1}^{\top};\cdots;{\bf Q}_{{\bf X}_{t}.H}{\bf W}_{k.H}^{\top}]\in\mathbb{R}^{HN\times D}$ and $\mathbf{\Omega}_{2,t}=[{\bf S}_{{\bf P}_{t}.1};\cdots;{\bf S}_{{\bf P}_{t}.H}]\in\mathbb{R}^{HN\times M}$ represent the concatenated matrices from all the heads, respectively. Based on block matrix properties, those two sets of conditions can be formulated as $\mathbf{\Omega}_{1,t}\Delta{\bf P}^{\top}=\mathbf{0}$ and $\mathbf{\Omega}_{2,t}\Delta{\bf P}=\mathbf{0}$. To sum up, The main difference between single-head and multi-heads is that the parameter update should be orthogonal to the subspace spanned by the concatenation matrices from all heads for multi-heads self-attention. Therefore, for the multi-heads variant, only an additional step of concatenation of the matrices from all heads is required in our algorithm.

In our experiments, we mainly utilize the VPT [13] with a ViT-B/16 backbone [9] pre-trained on ImageNet-21k. Additionally, we validate the effectiveness on the CLIP [27] model, wherein the visual prompts are inserted into the image encoder. Our experiments are conducted across 4 class-incremental benchmarks: 10- and 20-split CIFAR-100, 10-split ImageNet-R and 10-split DomainNet. We report the mean values of the final average accuracy and final average forgetting over 3 runs with different random seeds. Given that the null spaces of ${\bf Q}_{{\bf X}_{t}}{\bf W}_{k}^{\top}$ and ${\bf S}_{{\bf P}_{t}}$ may not always exist in practice, we compute the approximate null spaces and determine the nullities $R_{1}$ and $R_{2}$ in an adaptive manner, rather than the way suggested in [36]. For more detailed information regarding the experimental setups, please refer to Appendix B.

Validation of Effectiveness: The comparison between our approach and the sequential fine-tuning VPT and CLIP baselines is shown in Table 1 . For the VPT model, the proposed NSP² achieves 4.47%$\sim$10.26% improvements in accuracy on the 4 benchmarks. Meanwhile, it reduces the forgetting by 9.05%$\sim$17.11%. As to the CLIP model, the NSP² improves the accuracy by 6.48%$\sim$9.31%, and reduces the forgetting by 2.68%$\sim$17.27%. We calculate the accuracy across all previously encountered tasks after completing training on each task. The accuracy curves of VPT-Seq and VPT-NSP² on 10-split CIFAR-100 and 10-split ImageNet-R are displayed in Figure 2. They demonstrate our approach consistently outperforms the baseline throughout the sequential learning of tasks.

We conduct additional experiments with the VPT model, utilizing the weights pre-trained on different datasets as well as different paradigms, as shown in Figure 3 . The pre-training paradigms and datasets include: naive classification on ImageNet-1k [30], DINO [3] on ImageNet-1k, MIIL [29] on ImageNet21k-P and CLIP on LAION-2B [6] (we only use its image encoder). As can be seen from the figure, our approach not only significantly enhances accuracy but also markedly mitigates forgetting. These results further demonstrate the generalizability of the proposed approach.

Comparison with Existing Methods: We compare our method with existing methods in Table 2, where the competitors include many recent works. The proposed VPT-NSP² achieves state-of-the-art performance on the four benchmarks, with surpassing the second best approach by an average of 1.49% in accuracy. The forgetting of our approach is not the lowest, which is reasonable since our approach sacrifices some stability for a better trade-off between stability and plasticity. The outperforming accuracy can demonstrate the superiority of our method.

Ablation Study: The two consistency conditions Eq. (24) and (25), along with the constraint Eq. (20), constitute the main components of our approach. They correspond to $\mathcal{B}_{1}$, $\mathcal{B}_{2}$ in Eq. (26), and $\mathcal{L}_{\mathrm{LN}}$ in Eq. (27). We study their effects on the four benchmarks using VPT-NSP², with results presented in Table 3. We can see that the projection for Affinity ($\mathcal{B}_{1}$) plays a crucial role, which brings 3.31%$\sim$9.03% improvement in accuracy and 5.42%$\sim$14.76% decline in forgetting. Furthermore, the projection for Aggregation ($\mathcal{B}_{2}$) and the loss $\mathcal{L}_{\mathrm{LN}}$ for invariant prompt distribution are indispensable as well for minimizing forgetting. Optimal accuracy is achieved when all three conditions are applied.

Model Analysis: We analyze the evolution of training losses on the 10-split CIFAR-100 and 10-split ImageNet-R benchmarks, as shown in Figure 4 . Each point on the curve represents the training loss of the data in $\mathcal{T}_{1}$/$\mathcal{T}_{2}$ after the model has been trained on subsequent tasks. As can be seen, the losses of VPT-NSP² on previous tasks can be almost retained, confirming that our approach can effectively mitigate the interference of new tasks on old tasks.

Trade-off between Stability and Plasticity: We first adaptively determine the nullities $R_{1}$ and $R_{2}$ for $\mathcal{B}_{1}$ and $\mathcal{B}_{2}$ to achieve near-minimum forgetting. Based on this, we assign two weights $\eta_{1}$ and $\eta_{2}$ to the projection matrices to control the trade-off between stability and plasticity: $\Delta{\bf P}=\left[\eta_{2}\mathcal{B}_{2}+(1-\eta_{2}){\bf I}\right]{\bf P}_{\mathcal{G}}\left[\eta_{1}\mathcal{B}_{1}+(1-\eta_{1}){\bf I}\right]$, where ${\bf I}$ denotes the identity matrix. The effects of $\eta_{1}$ and $\eta_{2}$ which are set to a same value $\bar{\eta}$ is shown in Figure 5 . As the weight decreases, the accuracy increases first owing to better plasticity, and then decreases due to worse stability caused by the forgetting. It implies that a trade-off can be achieved by the two weights of projections.

Long-sequence Continual Learning We experiment on 5 benchmarks under the protocols of 50 tasks and 100 tasks to validate that our approach remains effective even within the context of long-sequence continual learning. The results are presented in Table 4. Despite lacking plasticity enhancement, VPT-NSP² can outperform existing state-of-the-art approaches and especially surpasses L2P by a large margin. This demonstrates that forgetting is still the predominant factor affecting performance in long sequence of tasks. With the plasticity enhancement, VPT-NSP² achieves significant increase in accuracy (by 1.1%$\sim$2.9%). This demonstrates that our plasticity enhancement is effective in learning new knowledge in long-sequence continual learning.