A Multi-view Perspective of Self-supervised Learning

Due to the powerful learning representation ability, deep convolutional neural networks (CNNs) (LeCun et al., 1998) have made breakthroughs in such areas as computer vision and natural language processing. At present, training a powerful CNNs often requires a large number of annotated/labeled instances, however, collecting such data in numerous practical tasks is time-consuming and expensive. In contrast, unlabeled data is cheap and easily accessible. Therefore, research on the effective use of such data is already a hot issue (Denton and Birodkar, 2017). In recent years, a new learning paradigm that automatically generates supervisory signals according to some attributes of data to guide feature learning—self-supervised learning (SSL) (Tung et al., 2017) is receiving increasingly more attention.

In SSL, a pretext with pseudo labels is first designed for CNNs to solve, where the pseudo labels are automatically generated based on some attributes of data without manual annotation. After the pretext task finished, the learned feature representation can be further transferred to downstream tasks as pre-trained models, making them obtain better starting points of solution (Jing and Tian, 2019). Thus the pretext task plays a key role, and many pretext tasks have been proposed such as in-painting (Pathak et al., 2016), patch context and jigsaw puzzles (Doersch et al., 2015; Noroozi and Favaro, 2016), rotation (Gidaris et al., 2018), color permutation (Lee et al., 2019), and so on. In particular, due to easy implementation and computational efficiency, the rotation pretext task has been more widely extended to multiple learning scenarios like domain adaptation (Sun et al., 2019), generative adversarial networks (GANs) (Chen et al., 2019), out-of-distribution detection (Hendrycks et al., 2019). Besides, as an auxiliary learning task, some researchers recently also have applied it to other learning tasks such as cross-modal learning task (Song et al., 2019) and face representation task (Röthlingshöfer et al., 2019), etc.

Although the effectiveness of SSL has been verified in most tasks, the essence of such effectiveness has not been quite clear yet. In fact, almost all existing pretext tasks are designed based more on intuitive and heuristic, which restricts its further development. It is generally believed that the process of solving the pretext task captures the feature representation beneficial for downstream tasks (Jing and Tian, 2019). However, it is a fact that the pseudo labels of pretext tasks are usually not explicitly associated with the supervised lable/signal of downstream tasks. Thus a natural question is: how does it work effectively? From a multi-view perspective, we can find that existing pretext tasks are mainly to perform explicit linear or nonlinear transformation (i.e., self-supervised signals) on the given original (single view) data, thus forming multiple transformed views (i.e., view augmentation of the data), which interestingly falls just right into the multi-view learning framework of single view data (Wang et al., 2007, 2011). However, unlike the traditional multi-view learning (MVL), SSL mainly introduces homogeneous view data with same dimension while exploits the views’ own labels as proxy supervised signals to guide feature learning. Furthermore, though some researchers (Federici et al., 2020) also conceptually involved this perspective, they have neither discriminated the difference between MVL and SSL mentioned above nor explored the essence of SSL in depth. As a preliminary attempt, this paper therefore borrows such a perspective to explore the essence so as to provide some insights into the design of pretext task.

Specifically, we first decouple a class of popular pretext tasks (e.g., rotation, color permutation) into a combination of view augmentation of the data (VAD) and view label classification (VLC). Taking the rotation as an example, such a pretext task essentially 1) first performs a rotation transformation {$0^{\circ},90^{\circ},180^{\circ},270^{\circ}$} on the corresponding data, so as to generate multiple groups of transformed data, each group of data can be regarded as one-view new data generated from original data, all the groups form a set of so-called augmented multi-view data; 2) then predicts the rotation transformation of these data, i.e., view labels.

Further, a simple yet effective verification framework is designed to explore the impacts of VAD and VLC on SSL’s performance as shown in Figure 1. Concretely, we specially design a simple multi-view learning framework for SSL (SSL-MV), which assists the feature learning of the downstream task (original view) through the same tasks on the augmented views (i.e., VAD tasks), making it focus on VAD rather than VLC. Note that thanks to replacing VLC with VAD tasks, SSL-MV also enables an integrated inference combining the predictions from the augmented views, further improving the performance. In addition, as a contrast, a multi-task learning framework of SSL (SSL-MT) is also provided, which focuses on both VAD and VLC.

The rest of this paper is organized as follows. Section 2 simply reviews the recent SSL works. In Section 3, we focus on a class of popular pretext tasks (such as rotation, color permutation), and decouple them into a combination of view augmentation of the data (VAD) and view label classification (VLC) from a multi-view perspective. Then a simple yet effective verification framework is designed to help us to explore the essence of such SSL tasks. Experimental results and analyses on several benchmark datasets are reported in Section 4. Section 5 is a brief discussion, while Section 6 concludes this paper.

As the key to SSL is the design of the pretext task, much of the research currently focuses more on designing new pretext tasks, thus developing a number of SSL methods. These methods use various cues from images or videos, etc. For example, some methods learn feature representation by solving the generations involving images or videos, such as image colorization (Zhang et al., 2016), image inpainting (Pathak et al., 2016), and video generation with GANs (Tulyakov et al., 2018), video prediction (Srivastava et al., 2015), etc. Some methods mainly adopt the image or video context information, such as jigsaw puzzles (Noroozi and Favaro, 2016), patch context (Doersch et al., 2015), geometry (Dosovitskiy et al., 2015; Li et al., 2019), counting (Noroozi et al., 2017), clustering (Caron et al., 2018), rotation (Gidaris et al., 2018), color permutation (Lee et al., 2019), and temporal order verification (Misra et al., 2016; Wei et al., 2018), sequence prediction (Lee et al., 2017a), etc. Furthermore, there are also some methods related to the automatic generation of free semantic label by traditional hardcode algorithms (Faktor and Irani, 2014) or by game engines (Ren and Jae Lee, 2018), such as semantic segmentation (Wang et al., 2020), contour detection (Ren and Jae Lee, 2018), relative depth prediction (Jiang et al., 2018), etc. For more details about these methods mentioned above, we refer the reader to (Jing and Tian, 2019).

In addition to exploring the design of pretext tasks, some researchers have also attempted to analyze the inherent advantages of SSL from other perspectives. From the perspective of CNNs’ structure, Kolesnikov et al. (Kolesnikov et al., 2019) studied the combination of multiple CNNs’ structures and multiple SSL pretext tasks, where they found that unlike supervised learning, the performance of SSL tasks depends significantly on the structure of CNNs used. Furthermore, they also experimentally show that the performance advantages obtained by the predecessor tasks do not always translate into the advantages in the feature representation of the downstream tasks, sometimes even worse. From the robust learning perspective, Hendrycks et al. (Hendrycks et al., 2019) found that SSL can improve robustness in a variety of ways, such as robustness to adversarial examples, label corruption, common input corruptions, while it also greatly benefits out-of-distribution detection (Hendrycks and Gimpel, 2016; Lee et al., 2017b) on difficult, near-distribution outliers.

From the data augmentation perspective, Lee et al. (Lee et al., 2019) reorganized various SSL strategies, where they mainly regarded SSL as a data augmentation technique and modeled the joint distribution of the original and self-supervised labels. Further, Federici et al. (Federici et al., 2020) empirically demonstrated the effectiveness of this kind of data augmentation under the multi-view information bottleneck framework they proposed. Besides, Keshav and Delattre (Keshav and Delattre, 2020) explored the crucial factors to avoid learning trivial solutions when constructing pretext tasks, while Kumar et al. (Kumar et al., 2020) investigated the SSL task combined with clustering, and provided the insight from data transformation perspective.

In this part, we focus on a class of popular pretext tasks such as rotation and color permutation, where we decouple them into a combination of view augmentation of the data (VAD) and view label classification (VLC), so as to explore whether VAD or VLC dominates the performance of such SSL. Specifically, two implementation pipelines of SSL, i.e., SSL-MT and SSL-MV, are designed, respectively, where SSL-MT focused on both VAD and VLC, while SSL-MV concentrates only on VAD. The more details about them will be provided in the following parts.

Notation. Let $\mathcal{D}=\{(\bm{x}_{i},y_{i})\}_{i}^{N}\subseteq\mathbb{R}^{N\times d}$ denote the training set, where $y_{i}\in\{1,2,...,K\}$ is the corresponding label and $K$ is the number of classes. Let $f(\bm{x})$ represent the shared feature extractor, $\mathcal{L}_{\text{CE}}$ be the cross-entropy loss function, and $\phi_{i}(\cdot)$ be the individual classifier. Furthermore, let $\bm{x}^{(j)}$ denote the augmented sample using a transformation $T_{j}$ (self-supervision label), where $j=0,1,2,...,M$, and $T_{0}$ denotes the identical transformation.

From the multi-view perspective, this paper revisits a class of popular SSL methods as an attempt to preliminarily explore the essence of SSL, where we find it is VAD rather than generally considered VLC plays a crucial role in boosting performance for downstream tasks. Please note that unlike traditional data augmentation and multi-view learning, SSL also considers the so-attached self-supervised labels. Furthermore, to our best knowledge, the theoretical exploration of SSL currently is almost blank as well. Thus, from the above explorations, it seems that we can draw on the existing multi-view learning theory (Wang and Zhou, 2017) to compensate this issue, and further expand SSL by the following typical strategies:

1. (1)

   Generate more diverse self-supervised signals/labels. For example, through richer transformation compositions: composed/nested/hierarchical operations on data examples.

2. (2)

   Establish a unified learning framework of self-supervised learning and multi-view learning.

Additionally, inspired by the effective use of self-supervised signals in SSL, exploring the use of view labels in existing multi-view learning is also worth further in-depth discussion.

In this paper, we revisit SSL from the multi-view perspective as an attempt to explore its essence, where we decouple a class of popular pretext tasks into a combination of view data augmentation (VAD) and view label classification (VLC), empirically revealing that it is VAD rather than generally considered VLC that dominates the performance of such SSL. Furthermore, we also give some insights about SSL, hoping to benefit the researchers in this community.