Investigating a Baseline Of Self Supervised Learning Towards Reducing Labeling Costs For Image Classification

Considering the requirement of a deep neural network for large amounts of annotated data, it is necessary to come up with well-performing unsupervised learning techniques. Although fully unsupervised learning remains an unresolved challenge in computer vision research, the discovery of self-supervised learning opens a potential for better performance with a small amount of labelled training data. Before the discovery of self-supervised learning, several unsupervised visual feature representation methods were proposed. Singh et al. [2] proposed an unsupervised visual representation technique, which aims at discovering desired patches from a set of unlabelled images.Young et al. [3] proposed an algorithm that learns the features of ConvNet and clusters iteratively with the help of a recurrent framework. The framework comprises a clustering algorithm with successive operations that are expressed as a recurrent process. This clustering algorithm is placed on top of a representation output of a Convolutional Neural Network. The image cluster and representations are jointly updated during the forward pass of the training stage whereas representation learning is conducted in the backward pass. By doing so, the framework achieves a powerful representation and precise image cluster.

Other augmentations were also successful in transferring valued features to the downstream task, such as jigsaw puzzle that predicts the correct arrangement of a jigsaw puzzle generated from the input image to learn the high-level image representations. Noroozi found that using jigsaw in self-supervised learn both the feature mapping and the proper spatial arrangement of object elements[4]. The scalability of self-supervision learning, specifically the Jigsaw and Colorization, is investigated by Priya Goyal et al. The aim of this paper is to see whether scaling the data, the model, or the problem complexity improves performance[5]. The two previous groups, in general, split input samples along a temporal line, predicting one provided the other. That could make it better suited to related tasks, such as semantic segmentation. According to Zhang et al, who used AlexNet to demonstrate comparative outcomes as compared to recent supervised methods, colorization is a pioneer of self-supervision [6]. The value of the failure, network design, and training information in achieving competitive results is demonstrated by Gustav Larsson’s et al. thorough investigation into self-supervised colorization [7]. Spyros Gidaris et al. [8] demonstrated that predicting image rotation is a powerful method to train convolutional neural network to learn semantic features of input images without labels. Therefore, self-supervised feature learning enhances the performance of object detection, image classification and image segmentation considering limited amount of data labels. Spyros experimented image rotation in self-supervised learning setting in many benchmarks’ problems achieving state-of-the-art performance using different well-known datasets. Dosovitskiy et al. [9] apply ConvNet model training to achieve representations that are invariant on geometric transformations while discriminative between images. However, in image rotation pretext, a ConvNet model learn to recognize the geometric transformation applied to an image to predict one out of four [0, 90, 180, 270] image orientations as demonstrated in Figure 1 . Unlike previous self-supervised approaches that rely on finite space of supervision labels and auxilary loss functions, Zhirong et al. [10] devised a novel self-supervised learning method in which each image instance is interpreted as a separate class, with a classifier trained to differentiate between individual input images. Nonetheless, no baseline has been established in the literature that can determine how many labelled data are required to achieve adequate image classification accuracy. This research aims to find such a baseline, if one exists, that can be applied to a variety of datasets and convolutional neural network architectures.

The following is the rest of the paper: The methodology of self-supervised learning, picture categorization, and model architecture are illustrated in section 2. Section 3 summarizes the findings of the ethe experiment and discusses the key foundations, while Section 4 wraps up the article with the most significant aspects.

The Residual Network (ResNet) developed by Kaiming He et al. [11] has an extremely deep CNN (up to 152) layer while keeping the number of parameters modest. The use of skip connections and the signal feeding into a layer being added to the output of a layer deeper upstream is the key trick to being able to train such a deep network. The ResNet architecture has proven to be effective in image classification and a variety of other computer vision tasks. The ResNet architecture with 20 layers is incorporated into the pretext and downstream tasks in this paper study. Figure 3 illustrates the main layers used in ResNet 20.

To compensate for the lack of labeled data, the pretext method in SSL applies a rotational transformation to the available inputs while simultaneously labeling them with a proxy label. The new pseudo labelled are four correspond to four angles of rotations (0 90, 180 and 270). The generted data can be thought of as a new classification task, in which training a convolutional neural network (CNN) using ResNet archetizture with twenty layers (ResNet-20) to predict the pseudo label can benefit in learning the feature representations of the data.

The task’s objective is to minimize the cross entropy losses incurred by predicting the four output neurons against the ground truth in the pdesuo labels. If we denote the rotation applied to an image by $T(.|y^{t})$ where $y^{t}$ is a binary variable activiate for every rotation angle t (four options) then a set of four rotations (classes) is created that the model has to predict from $R=\{T(.|y^{t})\}_{t=1}^{4}$. The classifier so generate a probablitc output among all possible classes $M(X|\theta)=\{M^{t}(X|\theta)\}_{t=1}^{4}$ and so the loss of single example become as shown in equation 1.

The question that this paper attempts to address is how much unlabeled data is needed to learn a visual representation that can compete with standard supervised learning with equivalent label data in the downstream task (image classification). The response is discovered by running two simultaneous training sessions on ten separate data volumes, one with SSL and the other without as illustrated in algorithm 1 . After the data is divided into ten stratified kfolds, and the algorithm feeds it to the pre-trained model resulted from the pretext task, with the last two layers fine-tuned. Simultaneously, the same data volume is fed to a ResNet model that has not been trained as illustrated by figure 2. THe resulted accuracy of image classifier is compared in both scenarios to draw a conclusion about the number of labelled data influencing the downstream task.

The basic concept behind using image rotations as a pre-training task before the main classification task is that any CNN model trained on any task would attempt to identify and become familiar with the data features and semantic parts in images that lead to the representation acquired. In order to predict effectively, the pretext model must relate the sections of images, hidden features, and the orientation of its background. Figure 4(a) shows the feature maps created by the conventional classification model without the pre-training component. Despite the fact that the pretext model’s goal is rotation predictions and not the background component, the feature maps in figure 4(b) are produced by the pretext model trained on the rotation recognition task, where they appear to be similar to the attention maps in figure 4(a) if we assume high density corresponds to the main parts of the object being identified.

Task generalizability on MNIST data: We further explore the generalizability of the framework with MNIST dataset, consisting of 60,000 training images and 10,000 test images of size 28x28x1. We use ResNet20 and follow exactly the same design used for Cats and dogs experiment. We fix all parameters and remove the last layer of the model substitute it with a densely connected NN layer with SoftMax activation. The model now performs classification for 10 categories instead of 4. In the downstream we compare the SSL model with the supervised model using stratified k-fold cross validation of k = 10 and then we choose to create 10 models. Each of the 10 models will be trained on training data starting from from 6,000 labelled input for model 1 to 60,000 labelled input for model 10. As shown from Figure 8 the accuracy gap between the supervised and self supervised model relativly smaller compared to the cats and dogs data.

Following the same trend from cats and dogs experiment, the highest advantage from self supervision is gained when the both models are trained on lowest fraction of data. Here 6,000 images only (10% of training data) provided an advantage for an increase in classification accuracy for SSL with a gap of around 4%. When provided with full data, the SSL downstream model accuracy the supervised model accuracy is almost similar with 0.990 and 0.989 respectively.

Task generalizability on Fashion-MNIST data: In Fashion-MNIST we repeat the same experiment done for the MNIST dataset. The training set consists of 60,000 grey scale images and 10,000 test images of size 28x28x1. The downstream model performs image classification for 10 classes similar to the MNIST classification. As shown from Figure 9(a) the accuracy gap between the supervised and self supervised model is largest for smaller fraction of training data. Similar to the results of the previous experiments, using 6,000 images only (10% of training data) provided an advantage for an increase in classification accuracy for SSL with a gap of around 0.05.

Discussion: In Table 1 we summarize the results for our experiments in a single table showing how different datasets benefit from different fraction of labelled training data. For each dataset, we see how much gain the self supervision provides to the task accuracy given small fraction of labelled training data (i.e., 2% and 10%).

Self-supervised learning allows deep neural network model to be trained utilizing less amount of labelled data. Random rotation pretext task allowed observable sematic feature learning for input images without labels. These learned visual features boosted the downstream image classification task remarkably. This paper attempts to draw performance baseline between SL and SSL models based on amount of labelled data trained on. Results showed that SSL outperformed SL in image classification task when both models are trained on different sizes of labelled data. Highest advantage of downstream task accuracy is gained from training with low fraction of labelled data, which highlight the advantage of SSL dealing with less labelled data. Extending the experiment to Fashion and MNIST datasets, SSL by image rotation generalizes very well to other datasets, and highest gain is noticed with more challenging datasets. Further experiments to random rotation pretext showed that high pretext task accuracy in SSL, resulted in more semantic feature learning, which translate to higher downstream task accuracy.