DTL: Disentangled Transfer Learning for Visual Recognition

We first show the simplest version of our solution. In Fig. 2 we illustrate the pipeline of the proposed architecture for the ViT (Dosovitskiy et al. 2021) backbone, which is mainly built up with a Compact Side Network (CSN). CSN is plugged into the backbone for information aggregation and feature adaptation. Note that the proposed method is compatible with other types of backbones, which will be discussed soon.

Given a ViT backbone containing $N$ blocks, the forward calculation can be formulated as $z=b_{N}(b_{N-1}(...b_{1}(x)))$, where $b_{i}$ is the $i$-th block, $x\in\mathbb{R}^{(n+1)\times d}$ is the input tokens (patch tokens plus one cls token) and $z\in\mathbb{R}^{(n+1)\times d}$ is the output tokens, respectively. Denote $z_{i+1}$ as the output of $b_{i}$, hence $z_{i+1}=b_{i}(z_{i})$ and $z_{1}=x$. Our CSN composes of $N$ low-rank linear transformation matrices (Hu et al. 2022), with each being plugged into one block to extract task-specific information. Denote $w_{i}=a_{i}c_{i}\in\mathbb{R}^{d\times d}$ as the weight matrix accounting for the $i$-th block, with $a_{i}\in\mathbb{R}^{d\times d^{\prime}}$, $c_{i}\in\mathbb{R}^{d^{\prime}\times d}$ and $d^{\prime}\ll d$, CSN progressively gathers information from each block as

	$\displaystyle h_{i+1}$	$\displaystyle=h_{i}+z_{i}w_{i}\,,$		(5)
	$\displaystyle z_{i+1}$	$\displaystyle=b_{i}(z_{i})\,,$		(6)

where $h_{i+1}$ is the output of the $i$-th layer of CSN ($h_{1}=0$). After that, starting from the $M$-th block, the aggregated task-specific information $h_{i+1}$ is used to adapt $z_{i+1}$ to downstream tasks by adding it back to $z_{i+1}$. Hence, when $i\geq M$,

$$z_{i+1}^{\prime}=z_{i+1}+\theta(h_{i+1})\,,$$

(7)

where $z_{i+1}^{\prime}$ is the adapted output of $b_{i}$ and $\theta$ is the Swish activation (Ramachandran, Zoph, and Le 2017), where $\theta(x)=\frac{x}{1+e^{-\beta x}}$. To prevent $z_{i+1}^{\prime}$ from drastically shifting away from $z_{i+1}$ at the beginning of fine-tuning, $a_{i}$ is initialized following a uniform distribution and $c_{i}$ is zero-initialized. To sum up, the output from the $i$-th block is

$$z_{i+1}^{\prime}=\left\{\begin{aligned} &z_{i+1}+\theta(h_{i+1})&&\text{if}\ \ i\geq M\\ &z_{i+1}&&\text{otherwise}\,.\end{aligned}\right.$$

(8)

We find that a small $d^{\prime}$ (2 or 4) performs fairly well, which suggests high redundancy in the backbone features. Therefore, in addition to keep $d^{\prime}$ small, we use a large $\beta$ (100) in Swish (i.e., $\theta$) to further reduce the redundancy. Consequently, about half of $\theta(h_{i+1})$ is close to zero.

To further boost the effectiveness of the proposed method, we append an additional global depthwise separable convolution (DWConv) layer (Chollet 2017) $g$ to each side layer after $\theta$ is applied. The formulation of DTL+ is

$$z_{i+1}^{\prime}=\left\{\begin{aligned} &z_{i+1}+g(\theta(h_{i+1}))&&\text{if}\ \ i\geq M\\ &z_{i+1}&&\text{otherwise}\,.\end{aligned}\right.$$

(9)

The stride of $g$ is set to 1 and zero-padding is used to ensure that $g$ does not change feature size. Note that $g$ is shared across different CSN layers, so that the number of trainable parameters in $g$ is small compared to the initial CSN, and the whole CSN module is still lightweight. The introduction of $g$ makes spatial information properly processed by our CSN module. With this operation, it’s easier for the model to recognize new categories.

The proposed approach has some significant advantages, which we discuss explicitly.

Disentangled. As shown in Fig. 2, the proposed CSN is a plugin mostly detached from the backbone, which interacts with the backbone in a plug-and-play manner. This characteristic makes our method easy to implement, and is compatible with almost all backbone networks. Modern deep neural networks are mostly divided into several intermediate stages and the feature dimensionality within one stage is the same. By re-initializing the hidden state of CSN $h_{i}$ to 0 at the beginning of each stage, our method can be easily transferred to different backbone architectures.

From the perspective of GPU memory usage, in previous methods weight update is directly entangled with the backbone. As we have analyzed before, although the number of trainable parameters is small, they still require a lot of GPU memory to cache many {$\sigma_{i}^{\prime}$} for gradient propagation. Our method alleviates this issue by 1) separating the forward pass of the backbone from CSN; and 2) only entangling them at late stages ($i\geq M$). Within our framework, no gradients are back propagated to the first $M$ blocks in the backbone (the gray region in Fig. 2). Hence, the number of cached {$\sigma_{i}^{\prime}$} is drastically reduced in CSN, resulting in a highly efficient way to realize GPU memory reduction.

Finally, we discuss another advantage drawn from our disentangled architecture: the possibility of feature reuse. Consider a scenario where we need to perform different tasks on one input image (e.g., simultaneously predict age and gender for a human). We have several fine-tuned models, and in previous methods the intermediate features $z_{i+1}$ generated by these fine-tuned models are different to each other. In other words, there is no way to share computation in the backbone across different tasks. Therefore a standard process is learning a group of task-specific parameters for each task (cf. Table 1) and conducting each task individually.

Conversely, as shown in the gray region of Fig. 2, in our DTL the intermediate features before block $M$ remain the same after fine-tuning, such that we can share part of the backbone computation between different tasks.

We take the 19 datasets in VTAB-1K (Zhai et al. 2019) to illustrate the above-described situation. We fine-tune to obtain 19 models, and assume that we need to get all 19 classification results for the one input image using all these 19 models. The goal is to check how much speedup can be achieved during inference. Firstly, we feed the same input image into 19 different models after fine-tuning with LoRA (Hu et al. 2022), which acts as the baseline. Then we implement our method to simultaneously conduct 19 tasks but with the backbone feature shared in the first 6 blocks (by setting $M=7$). Experimental results show that approximately 45% inference latency is saved during inference.

Simple. Since the CSN is disentangled from the backbone network, our method naturally shows higher simplicity compared to previous methods. Since all PETL methods add various types of structural units as extra trainable parameters, to verify the simplicity of our DTL in more detail, we compare the number of such minimal structural units of our method with previous methods in Table 1.

In this context, the phrase minimal structural unit means the atomic modules inserted into the backbone network. For example, in LoRA (Hu et al. 2022), one minimal structural unit comprises of a pair of $A$ and $B$ matrices to constitute $\Delta W$ (cf. Eq. 2). Since it inserts $\Delta W$ into both $W_{q}$ and $W_{v}$ for MHSA in every Transformer block, the total number of these units is 24. It is similarly defined for other methods as well, which include: 1) pairs of $\gamma$ and $\beta$ in SSF; 2) the modules to be searched in supernet and maintained in subnet in NOAH; 3) decomposed tensors in FacT; 4) pairs of matrices $a_{i}$ and $c_{i}$ in our DTL; 5) the additional global DWConv layer in DTL+. As shown in Table 1, the proposed method requires much fewer minimal structural units compared to existing methods.

We notice that a previous work LST (Sung, Cho, and Bansal 2022) also has a side network design. However, their architecture is very complicated and requires sophisticated techniques (Li et al. 2017) to initialize, resulting in the existence of large number of trainable parameters as shown in Table 2 (about 50$\times$ compared to ours). As analyzed before, we reduce the fine-tuning redundancy by setting $d^{\prime}$ to a very small value (2 or 4), which is far less than previous counterparts (e.g., 8 in LoRA and Adapter). This choice makes our DTL not only simple in structure, but also contains much fewer trainable parameters than other methods.

Effective. We conducted extensive experiments to verify the effectiveness of the proposed method. The results demonstrate that our method shows superior recognition accuracy across multiple architectures, achieving new state-of-the-art on several standard benchmarks.

Following previous work (Lian et al. 2022; Jie and Deng 2023), we take AdamW (Loshchilov and Hutter 2019) with cosine learning rate schedule as the optimizer. $\beta$ in Swish is fixed to 100. All pre-trained models are fine-tuned by 100 epochs with batch size 32. The rank $d^{\prime}$ of low-rank linear mappings in CSN is 2 for ViT and 4 for Swin-B. We set $M$ (cf. Eq. 8-9) of DTL and DTL+ as 7 for the ViT backbone, which means half of the later blocks’ output is calibrated by adding back the output from CSN. It is similarly defined for Swin-B with half of the layers adapted as well. Note that unlike previous methods (Zhang, Zhou, and Liu 2022; Lian et al. 2022), except for the standard data augmentation, we do not use any additional tricks such as mixup (Zhang et al. 2018), cutmix (Yun et al. 2019) or label smoothing (Szegedy et al. 2016). More details are available at https://www.lamda.nju.edu.cn/fumh/files/DTL/DTL˙appendix.pdf.

Datasets. VTAB-1K was introduced by Zhai et al. (2019) to evaluate the generalization ability of representation learning approaches. It contains diverse images from 19 different datasets, grouped as 1) Natural images captured by standard cameras; 2) Specialized images captured by specialist equipment; and 3) Structured images generated in simulated environments. They vary in task-specific objectives (e.g., classic visual recognition, object counting or depth prediction) and there are only 1,000 images in each dataset for training. It is a challenging benchmark to evaluate PETL methods. We report the top-1 recognition accuracy on the test set.

Baseline methods. First, two traditional fine-tuning techniques are included in all experiments. One is ‘Full’, which fine-tunes the entire pre-trained model. The other is ‘Linear’, which only fine-tunes task-specific classification head. Second, we choose BitFit (Ben Zaken, Goldberg, and Ravfogel 2022), VPT (Jia et al. 2022), LST (Sung, Cho, and Bansal 2022), AdaptFormer (Chen et al. 2022), LoRA (Hu et al. 2022), NOHA (Zhang, Zhou, and Liu 2022), FacT (Jie and Deng 2023) and SSF (Lian et al. 2022) as PETL baselines. We follow the setting in (Zhang, Zhou, and Liu 2022; Lian et al. 2022) to report the results for a fair comparison.

In addition to DTL and DTL+ ($M=7$), we further extend DTL+ where all of the blocks are adapted (i.e., $M=1$, denoted as ‘DTL+*’).

Main results. Results on ViT-B/16 are shown in Table 2. Our DTL shows a 1.0% gain on average accuracy compared to the previous state-of-the-art method SSF. By integrating a global DWConv, DTL+ further increases the average improvement to 2.0%. Specifically, DTL+ reaches the best top-1 accuracy on 11 out of 19 datasets, where the improvements compared to SSF range from 0.3% to 19.9%. Even if the dataset ‘dSpr-Loc’ with the most significant gain of 19.9% is removed, DTL+ is still far ahead on average accuracy of the remaining 18 datasets and outperforms SSF by 1.3%.

DTL only introduces 0.04M trainable parameters, which is 43% less compared to FacT. DTL+ specifies a few more trainable parameters (+0.01M) because of the introduction of a shared DWConv, but is still significantly less than previous PETL methods. Moreover, DTL and DTL+ consume 1.6GB and 1.7GB GPU memory during fine-tuning, respectively, which is far less than other PETL baselines. Compared to full fine-tuning, the GPU memory saving rate is about 65% on average (or saving roughly two thirds).

Another interesting observation is DTL+ and DTL+* show different distributions on accuracy improvements between three groups. For the ‘Natural’ group with smaller domain discrepancy between pre-trained models and downstream tasks, DTL+* outperforms DTL+ by 1%. For the ‘Structured’ group with a large domain gap, DTL+ surpasses DTL+* by 0.5% instead. This observation also appears similarly in Table 3 on Swin-B. We thus conjecture that since DTL+* has larger capacity than DTL+, it is prone to over-fitting when facing large domain gaps.

We observe that DTL+* shows more improvements in top-1 accuracy compared to DTL+, which indicates that the feature adaptation in early blocks within backbone is, in general, useful for transfer learning. However, the GPU memory usage in this case is increased to 3.1 GB, yet with only 0.2% accuracy gain compared to DTL+, implying the limited cost-efficiency of adapting early features.

As presented in Table 3, the performance of DTL and DTL+ on Swin-B shows similar trends with ViT. DTL+ achieves new state-of-the-art, outperforming FacT with a significant margin of 1% on average accuracy. DTL keeps the least trainable parameters and GPU memory footprint. Compared to full fine-tuning, DTL drastically saves GPU memory usage by 75%.

Datasets. Now we further evaluate on five fine-grained few-shot learning benchmark: Aircraft (Maji et al. 2013), Pets (Parkhi et al. 2012), Food-101 (Bossard, Guillaumin, and Van Gool 2014), Cars (Krause et al. 2013) and Flowers102 (Nilsback and Zisserman 2008). Following Jie and Deng (2023), we fine-tune the pre-trained model with training set containing {1, 2, 4, 8, 16}-shot per class and report the average accuracy on test set over 3 seeds.

Main results. As illustrated in Fig. 3, the proposed DTL and DTL+ outperform all baseline PETL methods in all cases. Furthermore, we observe that the average improvements of DTL+ compared to previous state-of-the-art FacT across different shots are gradually decreased from 4.7% in 1-shot to 0.8% in 16-shot, which reveals that our method is consistently effective, especially in low-data regimes.

Datasets. We follow Zhang, Zhou, and Liu (2022) to conduct experiments on domain generalization to evaluate the robustness of our method when domain shift (Zhou et al. 2023) is inevitable. In this scenario, the training set to fine-tune the pre-trained ViT-B/16 model is sampled from the original training set of ImageNet-1K, with each class containing 16 shot of images. Apart from the validation set of ImageNet-1K, the model is evaluated on 4 datasets, which are 1) ImageNet-Sketch (Wang et al. 2019) composed of sketch images sharing the same label space with ImageNet-1K, 2) ImageNet-V2 (Recht et al. 2019) collected from different sources compared with ImageNet-1K, 3) ImageNet-A (Hendrycks et al. 2021b) consisting of adversarial examples, and 4) ImageNet-R (Hendrycks et al. 2021a) containing various artistic renditions of ImageNet-1K. The reported accuracy is average by 3 different random seeds.

Main results. The results of domain generalization experiments are shown in Table 4. We observe that compared to previous state-of-the-art method NOAH, DTL and DTL+ achieve impressive gains in evaluation accuracy, especially on ImageNet, ImageNet-Sketch and ImageNet-R, where the average improvement is up to about 8%. These comparisons show excellent robustness of DTL and DTL+ for alleviating the domain shift problem and well demonstrate the effectiveness of the proposed method together with previous experiments.

Sensitivity to $M$. In DTL and DTL+, the beginning index ($M$) of blocks in the backbone to add back the output of CSN for feature adaptation is critical for final performance. In Fig. 4 we plot the curve of accuracy and GPU memory footprint by varying $M$. There is a clear trend that the number of GPU memory usage decreases almost linearly as $M$ becomes larger. By decreasing $M$ from 12 to 11, the recognition accuracy is significantly boosted to 76.7% from 75.9%, and the improvements gradually saturate when $M<6$, implying the feasibility and effectiveness of feature sharing as we described in the methods section. Finally, the range of accuracy across different $M$ is 2.1%, indicating that the recognition accuracy is not sensitive to the exact value of $M$, so by default we set $M=7$ to pursue the best trade-off between effectiveness and efficiency.

Modular ablation. In Table 5, we provide ablation results on the VTAB-1K benchmark from ViT-B/16. The first line, where the layer-wise output of CSN with rank ($d^{\prime}=2$) is directly added back to backbone (i.e, $z_{i+1}^{\prime}=z_{i+1}+h_{i+1}$ when $i\geq M$), is the baseline. It achieves a 76.0% average accuracy. By progressively integrating the Swish activation function $\theta$ in DTL and a global DWConv $g$ in DTL+, the accuracy is consistently improved. For DTL+, a higher ($d^{\prime}=4$) or lower ($d^{\prime}=1$) rank both make the accuracy worse than the default ($d^{\prime}=2$). Interestingly even when the trainable parameters of CSN are extremely limited with $d^{\prime}=1$, our DTL+ is still more effective than previous PETL methods.

Inference efficiency. We further study the efficiency of our method during inference by comparing the throughput with some baselines. As illustrated in Table 6, thanks to its simplicity, the empirical throughput of DTL+ is consistently higher than previous PETL counterparts. The simplest version of our method, DTL, boosts the inference efficiency a step further and significantly shows more speedup compared to traditional PETL methods.