Rethinking binary hyperparameters for deep transfer learning for image classification

We performed evaluations on four different small target datasets, each being less than 10,000 training images. We chose one that is very similar to Imagenet 1K that transfer learning and sub methods have traditionally performed best on. This is used as a baseline for comparison to show that the default methods that perform badly on the other datasets chosen do perform well on this one. For each of the other target datasets it has been shown that traditional transfer learning strategies:

• •

  do not perform well on them and/or

• •

  they are very different to the source dataset used for pre-training.

We used the standard available train, validation and test splits for the three datasets for which they were available. For Caltech256-30 we used the first 30 items from each class for the train split, the next 20 for the validation split and the remainder for the test split.

Inception v4 [36] was selected for evaluation as it has been shown to have state of the art performance for transferring from Imagenet 1K to a broad range of target tasks [15]. Our code is adapted from [38] and we used the publicly available pre-trained Inception v4 model. Our code is available at https://anonymous.4open.science/r/Non-binary-deep-transfer-learning-for-image-classification-872D. We did not experiment with pre-training settings, for example removing regularization settings as suggested by [15], as it was beyond the scope of this work and the capacity of our compute resources.

We used top-1 accuracy for all results for easiest comparison with existing results on the chosen datasets. Graphs showing ablation studies with different hyperparameters show results on one run. Final reported results are averages and standard deviations over four runs. For all experiments we used single crop and for our final comparison to state of the art for each dataset we used an ensemble of all four runs and 10-crop.

The experiments were run on compute nodes containing four Nvidia V100 GPUs and two 24-core Intel Xeon Scalable ’Cascade Lake’ processors.

Each individual experiment in the ablation studies ran on one GPU and 12 cores from one processor. Running time varied between approximately six hours for DTD to one day for Caltech.

Each final experiment was run on a full compute node with four GPUs and two 24-core processors. Runnng time varied between approximately one day for DTD and Aircraft to two days for Caltech.

We first examined performance using the pre-trained Inception v4 model as a fixed feature extractor for comparison and insight into the suitability of the pre-trained features. We trained a simple fully connected neural network classification layer for each dataset and compared it to training the full model from random initialization. The comparisons for all datasets are shown in Table 1.

The class normalized Fisher score on target datasets using the pre-trained but not fine-tuned weights, was used as a measure of how well the pre-trained weights separate the classes in the target dataset [7]. A larger normalized Fisher score shows more separation between classes in the feature space.

$$F\left(\mathbf{W}\right)=Tr\left\{s_{w}^{-1}s_{B}\right\}/N_{c}$$

where $s_{w}^{-1}$ is the inverse of the within class covariance, $s_{B}$ is the between class covariance and $N_{c}$ is the number of classes for the target dataset being measured. See [7] for further details.

We also calculated domain similarity between the fixed features of the source and target domain using the earth mover distance (EMD) [32, 29] and applying the procedure defined in [6].

The domain similarity calculated using the EMD has been shown to correlate well with the improvement in performance on the target task from pre-training with a particular source dataset [6].

For the first experiment we transferred all but the final classification layer and trained all layers at the same learning rate as per standard transfer learning practice. We performed a search using the validation set to find the optimal single fine-tuning learning rate for each dataset. The results in Figure 1 show the accuracy for each learning rate for each dataset.

The optimal single learning rate for Caltech, the most similar dataset to ImageNet, is an order of magnitude lower than the optimal learning rate for the fine-grained classification tasks Stanford Cars and Aircraft. This shows that the optimal weights for Caltech are much closer to the pre-trained values. The surprising result was that the optimal learning rate for DTD was very similar to Caltech and also an order of magnitude lower than Stanford Cars and Aircraft. Final results on the test set for each dataset are shown in Table 2.

To the best of our knowledge the combination of learning rate and number of layers reinitialized when performing transfer learning has not been examined in modern residual models. To examine the relationship between learning rate and number of layers reinitialized, we searched for the optimal learning rate for each of 1-3 blocks of layers reinitialized across each of the four datasets. One layer is the default with the final classification layer only being reinitialized. Two and three involve reinitializing an additional one or two Inception C blocks of layers respectively as well as the final layer. For consistency we refer to both the final classification layer and the Inception C blocks as layers. Figure 2 shows the performance of fine-tuning with various combinations of learning rate and layers reinitialized. The optimal learning rate when reinitializing more than one layer is always lower than when reinitializing only the final classification layer. Also the optimal number of layers to reinitialize is more than one for all datasets except Cars. Table 3 shows the final results for the optimal learning rate and accuracy for each number of layers reinitialized.

Conventionally learning rates for different layers are set so that either all layers are fine-tuned with the same learning rate or some are frozen. The thinking is that as lower layers are more general and higher layers more task specific [39] the lower layers do not need to be fine-tuned. Recent work has shown that fine-tuning tends to work better in most cases [30]. However, setting different learning rates for different layers is not generally considered. We examined the effects of applying lower learning rates to lower layers that are likely to generalise better to the target task. Figure 4 shows that for the Stanford Cars and Aircraft where the optimal initial learning rate is higher, setting lower learning rates for lower layers significantly improves performance whereas for Caltech and DTD it does not.

The L2SP regularizer decays pre-trained weights towards their pre-trained values rather than towards zero during fine-tuning. In the standard transfer learning paradigm with only the final classification reinitialized, when the L2SP regularizer is applied the final classification layer only is decayed towards 0.

The values $\alpha$ and $\beta$ are tuneable hyperparameters to control the amount of regularization applied to the pre-trained and randomly initialized layers respectively.

The original experiments showing the effectiveness of the L2SP regularizer [21] were done on target datasets that are extremely similar to the source datasets used. They showed that a high level of regularization decaying towards the pre-trained weights is beneficial on these datasets. It has since been shown that the L2SP regularizer can result in minimal improvement or even worse performance when the source and target datasets are less related [18, 37].

Our results shown in Figure 4 align with the original paper [21] for the dataset Caltech showing that standard L2SP regularization does result in an improvement in performance over L2 regularization. Our results also align with [18, 37] in showing that for the datasets we have chosen to be different from Imagenet, and known to be more difficult to transfer to, L2SP regularization performs worse than L2 regularization. In general the lower the setting for alpha (the L2SP regularization hyperparameter) the better the performance.

We relaxed the binary assumption that L2SP regularization must be applied to all pre-trained layers. We used L2SP regularization for lower layers that we expected to be more similar to the source dataset and L2 regularization for upper layers to allow them to specialise to the target dataset. We searched for the optimal combinations of L2SP and L2 weight regularization along with the $\alpha$ and $\beta$ hyperparameters for each dataset. We show the best settings for number of layers and amount of L2SP regularization in Table 5 and graph the optimal settings compared to the best default settings in Figure 5.

We make the following observations:

1. 1.

   A combination of L2SP and L2 regularization is optimal for most settings of the L2SP regularization hyperparameter ($\alpha$) for all datasets except Caltech.

2. 2.

   When more layers are trained with L2 rather than L2SP regularization the optimal L2 regularization hyperparameter ($\beta$) is lower as the squared sum of the weights in these layers will be larger for the same model.

Further experiments were performed to search for the combination of optimal L2SP vs L2 regularization settings with optimal number of layers transferred and different learning rates for different layers. These results are also shown in Table 5.

The final results and optimal settings are shown in Table 6 and Figure 6. Table 7 shows a clear distinction between target datasets that are more similar to the source dataset and for which the pre-trained weights are better able to separate the classes in feature space and those that are less similar with pre-trained weights that fit the target task poorly. The best measure for determining whether the pre-trained weights are well suited is a comparison of the performance achieved with fixed pre-trained features and that achieved through training from random initialization. As this method is computationally intensive a reasonable alternative may be the normalized Fisher Ratio, but as the differences are not as pronounced in all cases it should be further investigated on more datasets to see how reliable it is as a heuristic. The EMD measure of domain similarity is a poor predictor of the suitability of the pre-trained weights.

Targets datasets for which the pre-trained weights are well suited need:

• •

  a much lower learning rate for fine-tuning,

• •

  more than one layer to be reinitialized from random weights, and

• •

  some or all layers trained with L2SP regularization.

Target datasets where the pre-trained weights are not well suited need:

• •

  a much higher learning rate for fine-tuning with a lower learning rate for lower layers,

• •

  only the final classification layer reinitialized, and

• •

  L2 rather than L2SP regularization.

Using the above best practice, non-binary transfer learning procedures we achieved state of the art or close to, on three out of the four datasets. We used publicly available pre-trained weights and no additional methods for either pre-training or fine-tuning.