Feature Transformation Ensemble Model with Batch Spectral Regularization for Cross-Domain Few-Shot Classification

We build the ensemble model by increasing the diversity of the feature representation space while maintaining the usage of the entire training data for each prediction branch network. As shown in Fig. 1, we use a Convolutional Neural Network (CNN) $F^{B}$ to extract advanced visual features $f^{B}\in{\mathbb{R}^{m}}$ from the input data, and then transform the features into multiple diverse feature representation spaces using different random orthogonal projection matrices $\{E_{(1)},E_{(2)},\cdots,E_{(M)}\}$ on different branches. Each projection matrix $E_{(i)}$ is generated in the follow way. We randomly generate a symmetric matrix $Z_{(i)}\in[0,1]^{m\times m}$, and then form the orthogonal matrix $E_{(i)}$ using the eigenvectors of $Z_{(i)}$ such that $E_{(i)}=[e_{1},e_{2},...,e_{m-1}]^{\top}$, where $e_{i}$ represents the eigenvector corresponding to the top $i$-th eigenvalue of $Z_{(i)}$. With each projection matrix $E_{(i)}$, we can transform the extracted features into a new feature representation space such that $f^{E}_{(i)}=E_{(i)}f^{B}_{(i)}$, and build a soft-max predictor $C_{(i)}$ in this feature space. By using $M$ randomly generated orthogonal projection matrices, we can then build $M$ classifiers. In the pre-training stage in the source domain, all the labeled source data is used to train each branch network, which includes the composite feature extractor $F^{E}_{(i)}(x)=E_{(i)}F^{B}_{(i)}(x)$, and the classifier $C_{(i)}$, by minimizing the cross-entropy loss. In a training batch $(X_{B},Y_{B})=\{(X_{1},Y_{1}),\cdots,(X_{b},Y_{b})\}$, the loss function can be written as

$$\ell_{ce}(X_{B},Y_{B};C_{(i)}\circ F^{E}_{(i)})=\frac{1}{b}\sum\limits_{j=1}^{b}L_{ce}\left(C_{(i)}\circ F^{E}_{(i)}(X_{j}),Y_{j}\right)$$

(1)

where ${L_{ce}}$ denotes the cross-entropy loss function. After pre-training in the source domain, the fine-tuning on the labeled support set of the target domain can be conducted in the similar way, while the testing on the query instances can be naturally produced in an ensemble manner by taking the average of the $M$ classifiers’ prediction results.

Previous work [1] shows that penalizing smaller singular values of a feature matrix can help mitigate negative transfer in fine-tuning. We extend this penalizer into the full spectrum and propose a batch spectral regularization (BSR) mechanism to suppress all the singular values of the batch feature matrices in pre-training, aiming to avoid overfitting to the source domain and increase generalization ability to the target domain. This regularization is applied for each branch network of the ensemble model separately in the same way. For simplicity, we omitted the branch network index in the following presentation.

Specifically, for a stochastic gradient descent based training algorithm, we work with training batches. Given a batch of training instances $(X_{B},Y_{B})$, its feature matrix can be obtained as $A=\left[f^{E}_{1},\cdots,f^{E}_{b}\right]$, where $b$ is the batch size and ${f^{E}_{i}}=F^{E}(X_{i})$ is the feature vector for the $i$-th instance in the batch. The BSR term can then be written as

$${\ell_{bsr}}\left(A\right)=\sum\limits_{i=1}^{b}{\sigma_{i}^{2}}$$

(2)

where ${\sigma_{1}},{\sigma_{2}},\cdots,{\sigma_{b}}$ are singular values of the batch feature matrix $A$. The spectral regularized training loss for each batch will be:

$$\mathcal{L}=\ell_{ce}\left(X_{B},Y_{B};C\circ F^{E}\right)+\lambda\ell_{bsr}(F^{E}(X_{B}))$$

(3)

Due to the lack of labeled data in the target domain, the model fine-tuned with the support set can can easily make wrong predictions on the query instances. Following the effective label refinement procedure in [3], we propose to apply a label propagation (LP) step to exploit the semantic information of unlabeled test data in the extracted feature space and refine the original classification results.

Given the prediction score matrix $\hat{Y}^{0}$ on the query instances $X_{Q}$ with the fine-tuned classifier $C_{t}$, we keep the top-$\delta$ fraction of scores in each class (the columns of $\hat{Y}^{0}$) and set other values to zeros in order to propagate only the most confident predictions. We then build a k-NN graph over the query instances based on the extracted features $F^{E}(X_{Q})$. We use the squared Euclidean distance between each pair of images such as $d(i,j)=\left\|F^{E}(x_{i})-F^{E}(x_{j})\right\|^{2}$ to determine the k-NN graph. The RBF kernel based affinity matrix $W$ can be computed as follows:

$${W_{ij}}=\left\{{\begin{array}[]{*{20}{c}}{\exp\left(\frac{-d(i,j)}{2{\gamma^{2}}}\right),}&{i\in{\rm{KNN}}\left(j\right){\rm{or\ }}j\in{\rm{KNN}}\left(i\right)}\\ {0,}&{{\rm{otherwise}}}\end{array}}\right.$$

(4)

where ${\gamma^{2}}$ is the radius of the RBF kernel and ${\rm{KNN}}\left(i\right)$ denotes the k-nearest neighbors of the $i$-th image. The normalized Laplacian matrix $L$ can then be calculated as $L={Q^{-1/2}}W{Q^{-1/2}}$, where $Q$ is a diagonal matrix with ${Q_{ii}}=\sum\nolimits_{j}{{W_{ij}}}$. The label propagation is then performed to provide the following refined prediction score matrix:

$${Y^{*}}={\left({I-\alpha L}\right)^{-1}}\times\hat{Y}^{0}$$

(5)

where $I$ is an identity matrix and $\alpha\in\left[{0,1}\right]$ is a trade-off parameter. After LP, ${\hat{y}_{i}}=\arg{\max_{j}}Y_{ij}^{*}$ is used as the predicted class for the $i$-th image.

We extend the semi-supervised learning mechanism into the fine-tuning phase in the target domain by minimizing the prediction entropy on the unlabeled query set:

$${\ell_{ent}}(X_{B}^{q};C\circ{F^{E}})=-\frac{1}{b}\sum\limits_{i=1}^{b}{C\circ{F^{E}}(X_{i}^{q})\log(C\circ{F^{E}}(X_{i}^{q}))}$$

(6)

where $X_{B}^{q}$ denotes a query batch. We can add this term to the original cross-entropy loss on the support set batch $(X_{B}^{s},Y_{B}^{s})$ and form a transductive fine-tuning loss function:

$${{\cal L}_{ft}}={\ell_{ce}}(X_{B}^{s},Y_{B}^{s};C\circ{F^{E}})+\beta{\ell_{ent}}(X_{B}^{q};C\circ{F^{E}})$$

(7)

where $\beta$ is a trade-off parameter.

We also exploit data augmentation (DA) strategy with several random operations to supplement the support set and make the models learn with more variations. In particular, we use combinations of some operations such as image scaling, random crop, random flip, random rotation and color jitter to generate a few variants for each image. The fine-tuning can be conducted on the augmented support set. The same augmentation can be conducted for the query set as well, where several variants of each image can be generated to share the same label. Thus the prediction result on each image can be determined by averaging the prediction results on all the augmented variants of the same image.

In the experiments, we use the evaluation protocol in the CD-FSL challenge [2], which takes 15 images from each class as the query set and uses 600 randomly sampled few-shot episodes in each target domain, The average accuracy and 95% confidence interval are reported.

As for the model architecture, we use ResNet-10 as the CNN feature extractor ${F^{B}}$ and a fully-connected layer with soft-max activation as the classifier ${C}$. We set the trade-off parameters $\lambda=0.001$, $\beta=0.1$ and the number of branches $M=10$. For the label propagation step, we use $k=10$ for the k-NN graph construction, and set ${\gamma^{2}}$ as the average of the squared distances of the edges in the k-NN graph. The parameters $\delta$ and $\alpha$ are set to 0.2 and 0.5 respectively. We adopt mini-batch SGD with momentum of 0.9 for both pre-training and fine-tuning. During the pre-training stage, models are trained for 400 epochs. The learning rate and the weight decay are set to 0.001 and 0.0005 respectively. During fine-tuning, we set the learning rate to 0.01 and fine-tune for 100 epochs.

For data augmentation (DA), we choose 5 types of augmentation operations as shown in Table 1. We use 5 compound modes of these operations to generate data in different target domains. The specific operations used in each target domain are shown in Table 2.

We investigate a number of variants of the proposed model by comparing with the strong fine-tuning baseline result reported in [2]. We first investigate a single prediction network with batch spectral regularization (BSR) without ensemble and its other variants that further incorporate label propagation (LP) or/and data augmentation (DA). Then we extend these variants into the ensemble model framework with $M=10$. The results are reported in Table 3, and the average accuracies (and 95% confidence internals) across all datasets and shot levels are shown in Table 4.

We can see that even with only BSR, the proposed method can already significantly outperform the fine-tuning baseline (average 70.76% vs 67.23%). The ensemble BSR further improves the results (72.35%). The LP and DA components can also help improve the CD-FSL performance. We observe that on target domains more similar to the source domain, LP performs better than DA and vice versa. This shows that LP and DA focus on different aspects of the data. As a result combining LP and DA can further improve the performances. Moreover, DA is not very effective for the ISIC domain, where it even degrades the performance in some cases. Also the experiment’s running time is typically longer with DA. By replacing DA with ENT, we can obtain similar overall performance. With a single model, the best average result achieved by BSR+LP+DA is 72.85%, while BSR+LP+ENT achieves 72.68%. With the ensemble model, BSR+LP+DA (Ensemble) produces the best average result 73.94%, while BSR+LP+ENT (Ensemble) yields 73.62%.