Deep transfer learning for image classification: a survey

We are interested in finding a function $f$ that minimises the expected risk:

with

$R_{TRUE}\left(f\right)$ is the true risk if we have access to an infinite set of all possible data and labels, with $\hat{f}$ being the function that minimizes the true risk. In practical applications however the joint probability distribution $P(x,y)=P(y|x)P(x)$ is unknown and the only available information is contained in the training set. For this reason the true risk is replaced by the empirical risk, which is the average of sample losses over the training set $D$

leading to empirical risk minimisation vapnik1992principles .

Before we begin training our model we must choose a family of candidate functions $\mathcal{F}$. In the case of CNNs this involves choosing the relevant hyperparameters that determine our model architecture, including number of layers, the number and shape of filters in each convolutional layer, whether and where to include features like residual connections and normalization layers, and many more. This constrains our final function to the family of candidate functions defined by the free parameters that make up the given architecture. We are then attempting to find a function in $\mathcal{F}$ which minimises the empirical risk:

Since the optimal function $f^{*}$ is unlikely to be in $\mathcal{F}$ we also define:

to be the function in $\mathcal{F}$ that minimises the true risk. We can then decompose the excess error that comes from choosing the function in $\mathcal{F}$ that minimizes $R_{n}\left(f\right)$:

The approximation error $\varepsilon_{app}$ measures how closely functions in $\mathcal{F}$ can approximate the optimal solution $f^{*}$ . The estimation error $\varepsilon_{est}$ measures the effect of minimizing the empirical risk $R(f_{n})$ instead of the expected risk $R(f^{*})$ bottou2008tradeoffs . So finding a function that is as close as possible to $f^{*}$ can be broken down into:

1. 1.

   choosing a class of models that is more likely to contain the optimal function

2. 2.

   having a large and broad range of training examples in $D$ to better approximate an infinite set of all possible data and labels.

In general, $\varepsilon_{est}$ can be reduced by having a larger number of examples wang2020generalizing . Thus, when there are sufficient and varied labelled training examples in $D$, the empirical risk $R(f_{n})$ can provide a good approximation to $R(f_{\mathcal{F}}^{*})$ the optimal $f$ in $\mathcal{F}$. When $n$ the number of training examples in $D$ is small the empirical risk $R(f_{n})$ may not be a good approximation of the expected risk $R(f_{\mathcal{F}}^{*})$. In this case the empirical risk minimizer overfits.

To alleviate the problem of having an unreliable empirical risk minimizer when $D_{train}$ is not sufficient, prior knowledge can be used. Prior knowledge can be used to augment the data in $D_{train}$, constrain the candidate functions $\mathcal{F}$, or constrain the parameters of $f$ via initialization or regularization wang2020generalizing . Task specific deep neural network architectures such as CNNs and Recurrent Neural Networks (RNNs) are examples of constraining the candidate functions $\mathcal{F}$ through prior knowledge of what the optimal function form may be.

In this review we focus on transfer learning as a form of constraining the parameters of $f$ to address the unreliable empirical risk minimizer problem. Section 6 discusses how deep transfer learning relates to other techniques that use prior knowledge to solve the small dataset problem.

Most regularization based techniques aim to solve the problem of the unreliable empirical risk minimizer 3.2.2 by restricting the model weights or the features produced by them so they can’t fit small idiosyncrasies in the data. They achieve this by adding a regularization term $\lambda\cdot\Omega\left(.\right)$ to the loss function to make it:

with the first term $\frac{1}{n}\sum_{i=1}^{n}L\left(z\left(x_{i},w\right),y_{i}\right)$ being the empirical loss and the second term being the regularization term. The tuning parameter $\lambda>0$ balances the trade-off between the two.

Weight regularization directly restricts how much the model weights can move.

Knowledge distillation or feature based regularization uses the distance between the feature maps output from one or more layers of the source and target networks to regularize the model:

where $F_{j}\left(w_{t},x_{i}\right)$ is the feature map output by the jth filter in the target network defined by weights $w_{t}$ for input value $x_{i}$, and $d\left(.\right)$ is a measure of dissimilarity between two feature maps.

The success of regularization based techniques for deep transfer learning rely heavily on the assumption that the source and target datasets are closely related. This is required to ensure that the optimal weights or features for the target dataset are not far from those trained on the source dataset.

There have been many new regularization based techniques introduced in the last three years. We review major new techniques in chronological order.

1. 1.

   L2-SP li2018explicit ; li2020baseline is a form of weight regularization. The aim of transfer learning is to create models that are regularized by keeping features that are reasonably close to those trained on a source dataset for which overfitting is not as much of a problem. The authors argue that because of this, during the target dataset training phase the fine tuned weights should be decayed towards the pretrained weights, not zero. Several regularizers that decay weights towards their starting point, denoted SP regularizers, were tested in the original papers. The L2-SP regularizer $\Omega(w)=\frac{\alpha}{2}\left\|w-w^{0}\right\|_{2}^{2}$ which is the L2 loss between the source weights and the current weights is shown to significantly outperform the standard L2 loss on the four target datasets shown in the paper with a Resent-101 model. The original paper showed results for transferring to four small target datasets that were very similar to the two source datasets used for pretraining. It has since been shown that the L2-SP regularizer can result in minimal improvement or even negative transfer when the source and target datasets are less related li2020rethinking ; wan2019towards ; plested2021non ; chen2019catastrophic . More recent work has showed that in some cases using L2-SP regularization for lower layers and L2 regularization for higher layers can improve performance plested2021non .

2. 2.

   DELTA li2019delta is an example of knowledge distillation or feature map based regularization. It is based on the idea of re-using CNN channels that are not useful to the target task while not changing channels that are useful. Training on the target task is regularized by the attention weighted L2 loss between the final layer feature maps of the source and target models:

   	$\displaystyle\Omega(w,w^{0},x_{i},y_{i})=\sum_{j=1}^{N}(W_{j}(w^{0},x_{i},y_{i})$
   	$\displaystyle\cdot\left\|FM_{j}(w,x_{i})-FM_{j}(w^{0},x_{i}))\right\|_{2}^{2}$

   Where $FM_{j}(w,x_{i})$ is the output from the $jth$ filter applied to the $ith$ input. The attention weights $W_{j}$ for each filter are calculated by removing the model’s filters one by one (setting its output weights to 0), calculating the increase in loss. Filters resulting in a high increase in loss are then set with a higher weight for regularization, encouraging them to stay similar to those trained on the source task. Others that are not as useful in the target task are less regularized and can change more. This regularization resulted in performance that was slightly better than the L2-SP regularization in most cases with ResNet-101 and Inceptionv3 models, ImageNet 1K as the source dataset and a variety of target datasets. The original paper showed state of the art performance for DELTA on Caltech 256-30, however they used mostly the same datasets as the original L2-SP paper li2018explicit and for the two additional datasets used they showed that L2-SP outperformed the baseline L2 regularization. It has since been shown that like L2-SP, DELTA can also hinder performance when the source and target datasets are less similar kou2020stochastic ; chen2019catastrophic ; jeon2020sample .

3. 3.

   Wan et al. wan2019towards propose decomposing the transfer learning gradient update into the empirical loss and regularization loss gradient vectors. Then when the angle between the two vectors is greater than 90 degrees they further decompose the regularization loss gradient vector into the portion perpendicular to the empirical loss gradient and the remaining vector in the opposite direction of the empirical loss gradient. They remove the latter term, in the hopes that not allowing the regularization term to move the weights in the opposite direction of the empirical loss term will stop negative transfer. They show that their proposal improves performance slightly with a ResNet 18 on four different datasets. However, their results are poor compared to state of the art as they do not test on modern very deep models. For this reason, it is difficult to judge how well their regularization method performs in general.

4. 4.

   Batch spectral shrinkage (BSS) chen2019catastrophic introduces a loss penalty applied to smaller singular values of channelwise features in each batch update during fine-tuning so that untransferable spectral components are suppressed. They test this method using a ResNet50 pretrained on ImageNet 1K and fine-tuned on a range of different target datasets. The results show that their method never hurts performance on the given datasets and often produces significant performance gains over L2, L2-SP and DELTA regularization for smaller target datasets. They also show that BSS can improve performance for less similar target datasets where L2-SP hinders performance.

5. 5.

   Sample-based regularization jeon2020sample proposes regularization using the distance between feature maps of pairs of inputs in the same class, as well as weight regularization. The model was tested using a ResNet-50 and transferring from ImageNet 1K and Places365 to a number of different, fine grained classification tasks. The authors report an improvement over L2-SP, DELTA and BSS in all tests. Their results reconfirm that BSS performs better than DELTA and L2SP in most cases and in some cases DELTA and L2SP decrease performance compared to the standard L2 regularization baseline.

Further to regularization based methods, there are several recent techniques that attempt to better align fine-tuning in the target domain with the source domain. This is achieved by making adjustments to the standard batch normalization or other forms of normalization that are used between layers in modern CNNs.

1. 1.

   Sharing batch normalization hyperparameters across source and target domains has been shown to be more effective than having separate ones across many domain adaptation tasks wang2019transferable ; maria2017autodial . Wang et al. wang2019transferable introduce an additional batch normalization hyperparameter called domain adaptive $\alpha$. This takes standard batch normalization with $\gamma$ and $\beta$ shared across source and target domain and scales them based on the transferability value of each channel calculated using the mean and variance statistics prior to normalization. As far as we are aware these techniques have not been applied to the general supervised transfer learning case.

2. 2.

   Stochastic normalization kou2020stochastic samples batch normalization based on mini-batch statistics or based on moving statistics for each filter with probability hyperparameter p. At the start of fine-tuning on the target dataset the moving statistics are initialised with those calculated during pretraining in order to act as a regularizer. This is designed to overcome problems with small batch sizes resulting in noisy batch-statistics or the collapse in training associated with using moving statistics to normalize all feature maps ioffe2017batch ; ioffe2015batch . The authors results show that their methods improve over BSS, DELTA and L2-SP for low sampling versions of three standard target datasets and improve over all but BSS for larger versions of the same datasets. Their results again show that BSS performs better than DELTA and L2SP in most cases and in many cases DELTA and L2-SP decrease performance compared to the standard L2 regularization baseline.

Guo et al. guo2019spottune make two copies of their ResNet models pretrained on ImageNet 1K. One model is used as a fixed feature selector with the pretrained layers frozen and the other model is fine-tuned. They reinitialize the final classification layer in both. A policy net trained with reinforcement learning is then used to create a mask to combine layers from each model together in a unique way for each target example. They show that their SpotTune model improves performance compared to fine-tuning with an equivalent size single model (double the size of the two individual models within the SpotTune architecture) and achieves close to or better than state of the art in most cases. MultiTune simplifies SpotTune by removing the policy network and concatenating the features from each model prior to the final classification layer rather than selecting layers. It also improves on SpotTune by using two different non-binary fine-tuning hyperparameter settings plested2021non rather than one fine-tuned and one frozen model. The results show that MultiTune improves or equals accuracy compared to SpotTune in most cases tested, with significantly less training time.

Co-tuning for transfer learning you2020co uses a probabilistic mapping of hard labels in the source dataset to soft labels in the target dataset. This mapping allows them to keep the final classification layer in a ResNet50 and train it using both the target data and soft labels from the source dataset. As with many other recent results, they show that their algorithm improves on all others, including BSS, DELTA and L2SP, but their results are significantly below state of the art for identical model sizes, source and target dataset. They do show the same ordering for the target datasets, using BSS improves on DELTA which improves on L2SP.

Further to advances in techniques and models, there has been a large body of recent research that extends the early work on best practice for deep transfer learning for image classification described in Section 5.1. These studies give insights on the following decisions that need to be made when performing deep transfer learning for image classification:

• •

  Selecting the best model for the task. Models that perform better on ImageNet were found to perform better on a range of target datasets in kornblith2019better , however this effect eventually saturates abnar2021exploring . Given a set of models with similar accuracy on a source task, the best model for target tasks can vary between target datasets abnar2021exploring .

• •

  Choosing the best data for pretraining. In many cases pretraining with smaller more closely related source datasets was found to produce better results on target datasets than with larger less closely related source datasets mensink2021factors ; ngiam2018domain ; mahajan2018exploring ; cui2016fine ; cui2018large ; puigcerver2020scalable . For best results the source dataset should include the image domain of the target dataset mensink2021factors . For example ImageNet 1k contains more classes of pets than Oxford Pets making them an ideal source and target dataset combination. There are various measures of similarity used to define closely related that are outlined in Section 5.3.1.

• •

  Finding the best hyperparameters for fine-tuning. Several studies include extensive hyperparameter searches over learning rate, learning rate decay, weight decay, and momentum kornblith2019better ; li2018explicit ; mahajan2018exploring ; li2020rethinking ; plested2021non . These studies show the relationship between the size of the target dataset and its similarity to the source dataset with fine-tuning hyperparameter settings. Optimal learning rate and momentum, are both shown to be lower for more related source and target datasets li2020rethinking ; plested2021non . Also the number of layers to reinitialise from random weights is strongly related to the optimal learning rate neyshabur2020being ; plested2021non .

• •

  Whether a multi-step transfer process is better than a single step process. A multi-step pretraining process, where the intermediate dataset is smaller and more closely related to the target dataset, often outperform a single step pretraining process when originating from a very different, large source dataset mensink2021factors ; ng2015deep ; puigcerver2020scalable ; gonthier2020analysis . Related to this, using a self-supervised learning technique for pretraining on a more closely related source dataset can outperform using a supervised learning technique on a less closely related dataset zoph2020rethinking .

• •

  Which type of regularization to use. L2-SP or other more recent transfer learning specific regularization techniques like DELTA, BSS, stochastic normalization, etc, improve performance when the source and target dataset are closely related, but often hinder it when they are less related li2018explicit ; li2019delta ; wan2019towards ; plested2021non . These regularization techniques are discussed in more detail in Section 5.2.1.

• •

  Which loss function to use. Alternatives to the cross-entropy loss function are shown to produce representations with higher class separation that obtain higher accuracy on the source task, but are less useful for target tasks in kornblith2021better . The results show a trade-off between learning features that perform better on the source task and features relevant for the target task.

In an attempt to generalize hyperparameters and protocols when pretraining with source source datasets that are larger than ImageNet 1K, Kolesnikov et al. created Big Transfer (BiT) kolesnikov2019big . They pretrain various sizes of ResNet on ImageNet 1K and 21K, and JFT and transfer them to four small to medium closely related image classification target datasets as well as the COCO-2017 object detection dataset lin2014microsoft . Based on these experiments they make a number of general claims about deep transfer learning when pretraining on very large datasets including:

1. 1.

   Batch normalization (BN) ioffe2015batch is detrimental to BiT, and Group Normalization wu2018group combined with Weight Standardization performs well with large batches.

2. 2.

   MixUp zhang2017mixup is not useful for pretraining on large source datasets and is only useful during fine-tuning for mid-sized target datasets (20-500K training examples)

3. 3.

   Regularization (L2, L2-SP, dropout) does not enhance performance in the fine-tuning phase, even with very large models (the largest model used for experiments has 928 million parameters). Adjusting the training and learning rate decay time based on the size of the target dataset, longer for larger datasets, provides sufficient regularization.

The authors use general fine-tuning hyperparameters for learning rate scheduling, training time and amount/usage of MixUp that are only adjusted based on the target dataset size, not for individual target datasets. They achieve performance that is comparable to models with selectively tuned hyperparameters for their model pretrained on ImageNet and state of the art, or close to in many cases, for their model pretrained on the 300 times larger source dataset JFT. However their target datasets, ImageNet, CIFAR 10 & 100, and Pets are very closely related to their source datasets making them easier to transfer to. Their final target dataset, Flowers, is also known to be better suited to transfer to from their source datasets. See section 5.6 for further discussion of which target datasets are easier to transfer to.

We expect that best practice recommendations developed for closely related datasets will not be applicable to less closely related target datasets as has been shown for many other methods and recommendations li2018explicit ; li2019delta ; wan2019towards ; plested2021non ; li2020rethinking ; chen2019catastrophic ; kou2020stochastic . To test this hypothesis we reran a selection of the experiments in BiT using Stanford Cars as the target dataset which is very different from the source dataset ImageNet 21K and known to be more difficult to transfer to kornblith2019better ; plested2021non . We first confirmed that we could reproduce their state of the art results for the datasets listed in the paper, then produced the results in Table 1 using Stanford Cars. These results show that BiT produces far below state of the art results for this less related dataset. The first column shows the results with all the recommended hyperparameters from the paper. While the performance can be improved with increases in learning rates and number of epochs before the learning rate is decayed, final results are still well below state of the art for a comparable model, source and target dataset. The fine grained classification task in Stanford Cars is known to be less similar to the more general ImageNet and JFT datasets. Because of this it is not surprising that recommendations developed for more closely related target datasets do not apply.

Here we review works that give more general insight as to what is happening with model weights, representations and the loss landscape when transfer learning is performed as well as measures of transferability of pretrained weights to target tasks.

Several methods for analysing the feature space were used in neyshabur2020being . They found that models trained from pretrained weights make similar mistakes on the target domain, have similar features and are surprisingly close in $\ell_{2}$ distance in the parameter space. They are in the same basins of the loss landscape. Models trained from random initialization do not live in the same basin, make different mistakes, have different features and are farther away in $\ell_{2}$ distance in the parameter space.

A flatter and easier to navigate loss landscape for pretrained models compared to their randomly initialized counterparts was also shown in liu2019towards . They showed improved Lipschitzness and that this accelerates and stabilizes training substantially. Particularly that the singular vectors of the weight gradient with large singular values are shrunk in the weight matrices. Thus, the magnitude of gradient back-propagated through a pretrained layer is controlled, and pretrained weight matrices stabilize the magnitude of gradient, especially in lower layers, leading to more stable training.

Several recent techniques have been proposed for measuring the transferability of pretrained weights:

1. 1.

   H-score bao2019information is a measure of how well a pretrained model $f$ is likely to perform on a new task with input space $X$ and output space $Y$ based on the inter-class covariance $cov(\mathbb{E}_{P_{X|Y}}[f(X)|Y])$ and the feature redundancy $tr(cov(f(X)))$

   $$\mathcal{H}(f)=tr(cov(f(X))^{-1}cov(\mathbb{E}_{P_{X|Y}}[f(X)|Y])$$

   the H-score increases as interclass covariance increases and feature redundancy decreases. The authors show that H-score has a strong correlation with target task performance. They also show that it can be used to rank transferability and create minimum spanning trees of task transferability. The latter may be useful in guiding multi-step transfer learning for less related tasks as discussed in Section 5.2.4.

2. 2.

   Transferability and negative conditional entropy (NCE) for transfer learning tasks where the source and target datasets are the same, but the tasks differ, are defined in tran2019transferability . The authors define transferability as the log-likelihood $l_{Y}(w_{Z},k_{Y})$, where $w_{z}$ is the weights of the model backbone pretrained on the source task $Z$ and $k_{Y}$ is the weights of the classifier trained on the target task. They then define conditional cross-entropy (NCE) as another measure of transferability, defined as being the empirical cross entropy of label $\bar{y}$ from the target domain given a lablel $\bar{z}$ from the source domain. To empirically demonstrate the effectiveness of the NCE measure a ResNet18 model as the backbone was paired with an SVM classifier. NCE was demonstrated to have strong correlation with accuracy on the target tasks for combinations of 437 source and target tasks.

3. 3.

   LEEP nguyen2020leep is another measure of transferability. Using the pretrained model, the joint distribution over labels in the source dataset and the target dataset labels is estimated to construct an empirical predictor. LEEP is the log expectation of the empirical predictor. LEEP is defined mathematically as:

   $$T(\theta,D)=\frac{1}{n}\sum_{i=1}^{n}log\left(\sum_{z\epsilon Z}\hat{P}\left(y_{i}|z\right)\theta\left(x_{i}\right)_{z}\right)$$

   where $\theta\left(x_{i}\right)_{z}$ is is the probability of the source label $z$ for target input data $x_{i}$ predicted using the pretrained weights $\theta$, and $\hat{P}\left(y_{i}|z\right)$ is the empirical conditional probability of target label $y_{i}$ given source label $z$. LEEP is shown to have good theoretical properties and empirically it is demonstrated to have strong correlation with performance gain from pretraining weights on the source tasks. This is shown on source tasks ImageNet 1K and CIFAR10 and 200 random target tasks taken from the closely related CIFAR100 and less closely related FashionMNIST. The authors expand NCE to the case where the source and target datasets are different by creating dummy labels for the target data based on the source task using the pretrained model $\theta$. They show that LEEP has a stronger correlation with performance gain than the expanded NCE measure and H-score.

Recent works have tested the limits of the effectiveness of transfer learning with large source and target datasets by pretraining on datasets that that are 6$\times$he2017mask , 300$\times$ sun2017revisiting ; ngiam2018domain ; xie2020self , and even 3000$\times$ mahajan2018exploring larger than ImageNet 1K and target datasets that are up to 9$\times$ larger mahajan2018exploring .

In general, increasing the size of the source dataset increases the performance on the target dataset. This occurs even when the target dataset is large such as ImageNet 1K (1K classes, 1.3M training images) and ImageNet 5k (5K classes, 6.6M training images) or 9k (9K classes 10.5M training images) ngiam2018domain ; mahajan2018exploring ; kolesnikov2019big . However, source data that is carefully curated to match target data more closely can perform better than pretraining on larger more general source datasets ngiam2018domain ; mahajan2018exploring .

Early work in huh2016makes showed that additional pretraining data is useful only if it is well correlated to the target task. In some cases adding additional unrelated training data can actually hinder performance.

In more recent work a ResNeXt-101 32×16d was pretrained on various large Instagram source datasets. When using ImageNet 1K as the target dataset, the model was found to have the same performance when pretrained on a source dataset of 960M images with hashtags that most closely matched the ImageNet 1K classes as when the model that was pretrained with 3.5B images with 17K different hashtags mahajan2018exploring . However, pretraining with the smaller source dataset produced significantly worse performance when the target dataset was ImageNet 5K or 9K, or the more task specific Caltech Birds wah2011caltech and Places365 zhou2017places .

A similar result was found in ngiam2018domain . Pretraining using data from subgroups of classes that closely matched the target classes, consistently achieved better performance than using the entire JFT dataset with 300 million images and 18,291 classes. Similar performance was achieved using their technique of reweighting classes during source pretraining so that the class distribution statistics more closely matched the target class distribution. Interestingly though, applying the same technique to pretraining with the much smaller ImageNet 1K dataset resulted in minimal performance gains in most cases and a significant decrease in performance for one target dataset. This suggests a minimum threshold for the number of training examples where better matched training data is better than more training data. In contradiction to mahajan2018exploring these results showed that pretraining with the entire JFT dataset resulted in worse performance than pretraining with just ImageNet 1K for most target classes. In some cases performance was worse with pretraining on the entire JFT dataset rather than initialising with random weights without pretraining. This may be an indication of issues with the suitability of the JFT dataset, where image labels are non-mutually exclusive and on average, each image has 1.26 labels, as a source task for a mutually exclusive target classification task.

With a similar set up to ngiam2018domain , Puigcerver et al. puigcerver2020scalable produced a large number of different “expert” models by starting from one model pretrained on all of JFT then fine-tuning copies of this model on different subclasses. Performance proxies such as K nearest neighbours were then used to select the relevant expert for each target task. They found that selecting the best expert pretrained on the whole of JFT then fine-tuned on just a subset of classes resulted in better performance than just pretraining on the whole of JFT. These results are consistent with those from ng2015deep outlined in Section 5.7.2 in showing that a multi-stage fine tuning pipeline with an intermediate dataset that is more closely matched to the target dataset can produce better performance than transferring straight from a larger, less related source dataset.

Sun et al. sun2017revisiting found that pretraining a ResNet 101 he2016deep with the 300 million images from the full JFT dataset resulted in significantly better performance on ImageNet 1K classification compared to random initialisation. When the target task was object detection on the COCO-2017 dataset lin2014microsoft pretraining with the full JFT dataset or JFT plus ImageNet 1K produced a much larger increase in performance than pretraining with only ImageNet 1K as the source dataset. Both sets of results seem to indicate that the large ResNet 101 model generally overfits to the ImageNet 1K classification task when trained from random initialization, despite the dataset having over 1 million labelled training examples. They found that larger ResNet models produced greater performance gains than smaller models on the COCO object detection task with ResNet 152 outperforming both ResNet 101 and 50. Given that the difference between the mAP@[0.5,0.95] of the ResNet 101 and 152 was not significant when they were both pretrained on ImageNet 1K, but was when they were pretrained on the 300 times large JFT dataset it again indicates that larger models may overfit to ImageNet 1K.

Yalniz et al. yalniz2019billion created a multi-step semi-supervised training procedure that consisted of:

1. 1.

   training a model on ImageNet 1K

2. 2.

   using that model to label a much larger dataset of up to one billion social media images with hashtags related to the ImageNet 1K classes.

3. 3.

   using these weak self-labelled examples to train a new model

4. 4.

   finally, fine-tuning the new model with the ImageNet 1K training set.

They showed a significant increase in ImageNet 1K performance across a range of smaller ResNet architectures.

Xie et al. xie2020self extended the training procedure outlined above yalniz2019billion to iterate between training a large EfficientNet tan2019efficientnet model on ImageNet 1K, then using that model to label the 300 million images from the full JFT dataset and training the model using both the weak self-labelled images from JFT and real labelled images from ImageNet 1K. This resulted in state of the art performance on ImageNet 1K. They also found that testing this model on more difficult ImageNet test sets being ImageNet-A (particularly difficult ImageNet 1K test examples) hendrycks2019benchmarking , as well as ImageNet-C and ImageNet-P (test images with corruptions and perturbations such as blurring, fogging, rotation and scaling) hendrycks2021natural resulted in large increases in state of the art performance. State of the art accuracy was doubled on two out of the three more difficult ImageNet test sets.

In light of the findings in the previous section the question of how to measure similarity between source and target datasets is important. In some cases source and target domain similarity can be effectively estimated through human intuition and/or similarity between class labels mahajan2018exploring ; ngiam2018domain ; puigcerver2020scalable . Further to this there are many methods for using calculations of domain similarity in the hopes of finding the best source data for pretraining:

• •

  Cui et al. cui2018large showed that performance on the target dataset increases as a measure of similarity based on the earth mover distance between the source and target domains increases.

• •

  Domain adaptive transfer learning (DATL) ngiam2018domain uses importance weights based on the ratio $P_{t}(y)/P_{s}(y)$ to reweight classes during source pretraining so that the class distribution statistics match the target statistic $P_{t}(y)$. Where $P_{t}(y)$ and $P_{s}(y)$ describe the distribution of labels in the target and source datasets respectively. They show that using adaptive transfer or a subset of the source dataset that more closely matches the target dataset improves performance over using the entire unweighted 300 million JFT training examples.

• •

  Puigcerver et al. puigcerver2020scalable train a large selection of expert models on particular subsets of JFT source data based on class labels. They then determine the best model to use for each prediction based on performance proxies such as k nearest neighbours. This technique was shown to increase performance on several target datasets compared to training on the whole JFT dataset.

• •

  Using reinforcement learning to learn a weight for each class in the source dataset so as to rely more heavily on more effective training examples during pretraining zhu2019learning . Also using reinforcement learning to value individual training samples for a particular task and rely more heavily on more valuable examples during training yoon2020data .

• •

  Ge et al ge2017borrowing create histograms for images from the source and target domain using the activation maps from the first two layers of filters in a CNN. They then use nearest neighbour ranking to find the most similar images in the source dataset to those in the target dataset. Samples in the source domain are then weighted based on their KL-divergence from a given target sample.

Similarly to large target tasks, better matched pretraining data has been shown to produce better performance on the target task than more pretraining data.

Pretraining on the scene classification task Places365 achieved considerably better performance on the scene classification dataset MIT Indoor 67 as compared to pretraining with the smaller and less related ImageNet 1K in li2018explicit . However the reverse was true for the target tasks that were more related to ImageNet 1K, being Stanford Dogs and Caltech256-30 and 60.

Source and target data classes were matched using the Earth Mover’s Distance peleg1989unified ; rubner2000earth in cui2018large . Pretraining with well matched subsets was shown to perform better than with the largest source dataset on a variety of fine-grained visual classification (FGVC) tasks. They also showed a strong correlation between source and target domain similarity as calculated by the Earth Mover’s Distance and performance on the target task for all but one target task.

Ge and Yu ge2017borrowing found model performance was improved by fine-tuning a model on the target task along with the most related data from the source task. Related data was calculated using nearest neighbour on the activations of Gabor filters. Sabatelli et al. sabatelli2018deep found that performance was improved when pretraining on a smaller art classification source dataset rather than the larger unrelated ImageNet 1K when the target task was art classification. In the same domain Gonthier et al. gonthier2020analysis used a two step fine-tuning process consisting of:

1. 1.

   pretraining on ImageNet 1K

2. 2.

   fine tuning on an art classification dataset that was an order of magnitude larger than the target task

3. 3.

   fine-tuning on the final target art classification dataset.

They found that this improved performance over pretraining on only ImageNet 1K or only the intermediate art classification dataset.

Face recognition often relies on pretraining of deep CNN architectures on general image classification datasets like ImageNet 1K. However this area of research presents its own unique challenges masi2018deep :

1. 1.

   With each class representing only one individual there are often only slight differences between classes and a small number of training images per class.

2. 2.

   There can be many more classes than is common for an image classification task with common face recognition datasets having 10’s or 100’s of thousands or even millions of different subjects.

Related to Point 1, a common challenge for facial recognition models is to pretrain them on a large number of publicly available faces of celebrities then use them to quickly learn to classify new faces with limited examples via transfer learning techniques. Several additions to the typically used cross-entropy loss have been proposed to help with this challenge they:

• •

  explicitly minimize the intraclass variance wen2016discriminative ,

• •

  increase the margin between classes liu2016large ; liu2017sphereface ,

• •

  encourage features to lie on a hypersphere ranjan2017l2 ; zheng2018ring .

These methods are all designed to produce a well defined feature space during pretraining so that there is likely to be a good separation between subjects for both source and target subjects. When the number of subjects in face recognition datasets get very large, deep metric learning losses are generally used instead of classification losses schroff2015facenet ; wang2018cosface ; deng2019arcface . It has recently been shown that metrics designed specifically for face recognition such as CosFace wang2018cosface and ArcFace deng2019arcface can be more successful when applied to common deep metric learning benchmark datasets or small fine-grained image classification tasks musgrave2020metric . Deep metric learning is discussed further in Section 6.3.

Like many fine-grained classification problems facial expression recognition (FER) datasets are often challenging because they are small. Most well known FER datasets have less than 10,000 images or videos. Even the larger ones often have only around 100 different subjects, making the individual images highly correlated. An additional challenge unique to facial expression recognition is that high intersubject (intraclass) variations exist due to different personal attributes, such as age, gender, ethnic backgrounds and level of expressiveness li2020deep .

Again, for this task it has been shown that pretraining with source data more closely matched to the target dataset results in better performance. pretraining on a large facial recognition dataset has been shown to perform better than pretraining on the more general and less closely related ImageNet 1K imagenet_cvpr09 . A multi-stage pretraining pipeline, using a larger FER dataset for interim fine-tuning prior to the final fine-tuning on the smaller target dataset, was shown to improve performance in ng2015deep .

The two major problems presented when using deep neural networks for medical imaging tasks that are most relevant to this review are:

1. 1.

   Scarcity of data. Training data in medical image datasets often numbers in just the 100’s or 1,000’s of examples as opposed to the 100,000s, millions or even billions of examples often available in more general image datasets mazurowski2019deep .

2. 2.

   Severely unbalanced classes. There are often many more examples of healthy images as opposed to those with a rare disease mazurowski2019deep .

Training very deep networks from scratch is problematic in many medical imaging cases due to the problems listed above. Deep transfer learning is adopted in almost every modality of medical imaging, including X-rays, CT scans, pathological images, positron emission tomography (PET), and MRI mazurowski2019deep . Despite this there has been limited work addressing best practice for deep transfer learning in the medical imaging classification domain.

Early experiments tajbakhsh2016convolutional compared an AlexNet pretrained on ImageNet 1K with and without fine-tuning, an AlexNet trained from scratch, and traditional models with hand crafted features. They demonstrated that:

1. 1.

   The use of a pretrained AlexNet CNN with adequate fine-tuning consistently improved on or matched training from random initialisation and traditional methods.

2. 2.

   While the increase in performance using a pretrained and fine-tuned AlexNet was marginal for relatively larger target datasets the performance improvement was much more significant as the size of the target datasets was reduced.

More recent experiments in deep transfer learning for digital pathology classification using a range of modern models with residual connections and four different target datasets were performed by Mormont et al. mormont2018comparison . The models were pretrained on ImageNet 1K. When using the pretrained models as fixed feature extractors the last layer features were always outperformed by features taken from an inner layer of the network. In keeping with previous results in the field they found that fine tuning improves on fixed extraction. However, they did not combine fine tuning with testing different layers in the network for optimal performance, which we suggest for future work.

There have been a number of studies in the last two years examining whether a very large less related source dataset is better, for pretraining for image classification in the medical domain, when compared to a within domain dataset that is orders of magnitude smaller raghu2019transfusion ; heker2020joint . Both these scenarios have also been compared to self-supervised pretraining on a medium sized unlabelled within domain dataset azizi2021big .

Previous results showing that pretraining with much larger source datasets increases performance on the target task were extended to the medical image domain in mustafa2021supervised . Performance on three well known small medical image classification target tasks was shown to improve as the size of the source dataset increased from ImageNet 1K with 1.3 million training examples to JFT with 300 million training examples. All pretraining produced better performance than random initialization. There has also been some work showing that performing supervised pretraining on a moderately sized medical image dataset and transferring to a smaller one can increase performance compared to training from random initialisation kraus2017automated ; heker2020joint , other shallow machine learning methods kraus2017automated , and even pretraining with ImageNet 1K heker2020joint .

Raghu et al. raghu2019transfusion showed that when medical image datasets are large (around two hundred thousand training examples), pretraining on ImageNet 1K results in limited improvements over random initialisation. This applies to both the large CNN models large ResNet50 and Inception v3 and small CNN models designed for better performance on medical datasets. When the number of training examples was reduced to a small dataset size of 5,000, pretraining with ImageNet 1K resulted in small improvements over random initialisation. They also reaffirm that lower layers are more transferable than higher more task specific layers as per yosinski2014transferable , even when the source and target tasks are very different.

A two step self-supervised pretraining process was used on two different medical imaging classification target tasks in azizi2021big . This involved:

1. 1.

   self-supervised pretraining on ImageNet 1K

2. 2.

   followed by self-supervised fine-tuning on a large unlabelled medical dataset from same source as the target task

3. 3.

   then fine-tuning on the final labelled medical image classification task.

This was found to produce significantly better results on the target task compared to self-supervised pretraining on only ImageNet and slightly better results than pretraining on the unlabelled medical dataset only or pretraining on ImageNet 1K with supervised methods.

Using a multi-stage supervised pretraining pipeline such as in ng2015deep does not appear to have been applied to classifying medical images with deep CNNs. This could be a useful area of further research.

Although it is somewhat beyond the scope of this paper, we briefly review the evidence on whether pretraining on image classification datasets is beneficial for object detection tasks.

Starting with girshick2014rich many studies have shown improved performance on object detection tasks when ImageNet pretraining is used ren2015faster ; redmon2016you . It has become the conventional wisdom that pretraining is needed to achieve top results on object detection tasks as the datasets tend to be smaller than image classification datasets like ImageNet imagenet_cvpr09 . However, a number of recent results have challenged this idea.

Comparable results on the COCO object detection task lin2014microsoft are shown with random initialisation of weights compared to pretraining on ImageNet 1K imagenet_cvpr09 when training protocols are adjusted to be optimal for training from random initialization he2018rethinking . This result is repeated even when the target dataset is reduced to be on 10K images total, or 10% of the full COCO size. A number of other experiments show similar results with and without pretraining shen2017dsod ; shen2017learning ; zhu2019scratchdet and that performance is often better without pretraining when more exact measures of bounding box overlap are used he2018rethinking ; zhu2019scratchdet .

Mixed results from pretraining on the 300 $\times$ larger Instagram hashtag dataset for the same COCO task are reported in mahajan2018exploring . The improvements were marginal at best, whereas the improvements reported on the ImageNet and CUB2011 classification tasks are larger. However in sun2017revisiting performance on COCO was significantly improved when pretraining on the large JFT source dataset with 300 million training examples, compared to ImageNet 1K. Again, these mixed results seem to indicate that deep transfer learning training hyperparameters can have a large influence on results when target datasets are smaller and less similar to source datasets.

A multi-stage pretraining pipeline ng2015deep may again be useful to consider in this domain. Developing more domain specific self-supervised pretraining techniques could also be considered.

Due to the inherent difficulty of gathering and creating per pixel labelled segmentation datasets, their scale is not as large as the size of classification datasets such as ImageNet. For this reason semantic image segmentation models have traditionally been pretrained on ImageNet 1K and other image classification datasets such as JFT garcia2018survey ; ghosh2019understanding ; minaee2020image . The same pattern is noted here as with other transfer learning applications in that more training data tends to produce better results chen2018encoder . Recently it has been shown that self-training on unlabelled but more closely related data consistently improves performance over training the same model from scratch. Whereas pretraining on ImageNet 1K can reduce performance (negative transfer), particularly when the target dataset is larger zoph2020rethinking .