Supplementary Materials of
Hint-Aug: Drawing Hints from Vision Foundation Models
towards Boosted Few-shot Parameter-Efficient ViT Tuning

In this supplement, we provide more experiments and analysis as a complement to the main content, which are outlined below:

• •

  We evaluate Hint-Aug’s ability in boosting the FViTs’ few-shot parameter-efficient tuning accuracy across different pretraining strategies in Supple. 2.

• •

  We evaluate Hint-Aug’s ability in boosting the FViTs’ accuracy under the less data-limited transfer learning setting in Supple. 3.

• •

  We conduct an ablation study on Hint-Aug’s sensitivity to $\lambda$ in Eq. 3 from the main paper in Supple. 4.

The pretraining strategy is a key factor in fully exploiting the FViTs’ potential [he2021masked, bao2021beit, chen2021empirical]. As mentioned in [he2021masked, chen2021empirical], it plays an even more important role than the FViT models’ architectures, given that we are now benchmarking under transformer-based models of similar architectures (e.g., applying the pretraining strategy as in [he2021masked] on ViT-Base can compensate for its accuracy gap with ViT-Huge, which has nine times more number of parameters). With this in mind, we aim to validate Hint-Aug’s capability in boosting the FViTs’ accuracy across models trained with different pretraining strategies. To do this, we validate the proposed Hint-Aug by applying it on top of the ViT-Base [dosovitskiy2020image] models pretrained with a widely used SOTA pretraining strategy, MAE [he2021masked], while keeping all other experiment settings the same as those used in the experiments described in the main paper.

As shown in Table 1, Hint-Aug can consistently boost the achievable accuracy over that of the baseline tuning methods by $0.31\%\sim 3.60\%$ and $0.45\%\sim 3.27\%$ on an 8-shot and 16-shot setting, respectively. This validates that Hint-Aug is a general method that can consistently improve the effectiveness of few-shot parameter-efficient tuning of FViTs.

We further validate Hint-Aug’s achieved accuracy when we use more tuning data as in the conventional transfer learning scenario because it can evaluate Hint-Aug’s potential as a general technique to improve parameter-efficient tuning accuracy under various scenarios. In this set of experiments, we tune the ImageNet pretrained ViT-Base [dosovitskiy2020image] on five datasets including Caltech-101 [fei2004learning], CIFAR-100 [krizhevsky2009learning], DTD [cimpoi2014describing], Flowers [nilsback2006visual], and Pets [parkhi2012cats]. We follow the same tuning setting in [zhang2022neural] and show our results in Table 2. We can see that Hint-Aug achieves as high as 2.47%, 0.90%, and 3.77% accuracy improvement over the baseline method when tuning Adapter, VPT, and LoRA, respectively. This further validates that (1) Hint-Aug can generally boost the achievable accuracy under various settings in parameter-efficient tuning, and (2) the features generated in Hint-Aug are of high quality and can further improve the feature diversity even when more data (e.g., 50,000 tuning data in CIFAR-100 dataset) are provided.

To validate Hint-Aug’s ability when being applied to new tasks, we perform an ablation study regarding the impact of the only hyperparameter, $\lambda$ in Eq. 3, in Hint-Aug, under 8-shot Adapter tuning of ViT-Base [dosovitskiy2020image] on the Food [bossard2014food] dataset. As shown in Table 3, different values of $\lambda$ lead to marginal differences in the achievable accuracy (i.e., lower than 0.2%). This validates that Hint-Aug is not sensitive to the hyperparameter being used, alleviating the potential burden of having to tune hyperparameters when applying Hint-Aug to a new task.