Mask Hierarchical Features For Self-Supervised Learning

In computer vision, self-supervised learning has attracted unprecedented attention for its impressive representation learning ability. Contrastive learning and Masked Image Modeling(MIM) are the two most effective methods.

Contrastive learning methods are first adopted to learn invariant image-level global representation. Concretely, the network is typically trained to maximize the similarity between two augmentations of one image, such as MoCo v1/v2 [13, 6], BYOL [10], and SimSiam [7]. These image-level methods can learn distinguished visual representation, such achieve high performance on classification tasks. Other contrastive learning methods[25, 30, 26] attempt to model local information by aligning the local representations from the same location in different augmented image views. For instance, SoCo[26] first generated object proposals via selective search [21] and then aligned the pre-proposal object features of the different views. These object-level methods achieve high performance on dense prediction tasks but show a distinct disadvantage on classification tasks.

Recently, Masked Image Modeling(MIM)[12, 2, 29, 31, 1] methods have aroused great interest in the community. The core idea of MIM is to reconstruct the masked source image with sparse unmasked patches. Take MAE[12] for example, it masks patches of the source images, feeds the masked image into an encoder model, and then utilizes visible patch features to predict the masked pixels by a decoder model. Other methods[2, 29] concern the high-frequency noise of predicting low-level pixels. Therefore, they predict high-level semantics of the patches embedded by a tokenizer, instead of predicting low-level pixels directly. These MIM methods have made a breakthrough for the pre-training of the vision transformer model. We attribute part of the success of the MIM methods to the relationship probing among sparse patches. It proves that reasoning invisible context from sparse local patches is an efficient pretext task.

In this paper, We follow the idea of learning knowledge from reasoning the relationship among sparse local patches. Distinctively, to avoid the consumption of thousands of training epochs, we extract the embedding of sparse local patches from a well-designed feature space several times, instead of feeding in masked images repeatedly, like MIM methods. Concerning the hierarchical property of visual representations, we incorporate an FPN-type module to make patch embedding hierarchical. We assume that each point feature in the spatial feature represents the high-level semantics of a specific region in the source image. Such, We can construct the group of sparse visible patch embedding form an implicit representation space directly. We called the approach as MaskDeep.

To learn knowledge from the groups of sparse local patches, we reason from sparse patch embedding to predict the global semantics of an image. The intuition lies in the fact that the local objects contain fine-grained object semantics, while the global image demonstrates high-level semantic information across all spatial locations. Integrating local object features to represent global image semantics can be an effective self-supervised objective. As the Figure 2, the sparse patch features, including ’human head’, ’dog leg’ and ’tree’ etc., are integrated to predict the image contents that ’A person is walking a dog in the park’. In this way, both local object understanding and global context information reasoning are concerned. This property promises MaskDeep a unified self-supervised learner, which is friendly for both image-level and object-level downstream tasks. Concretely, in MaskDeep, an online branch is used to construct patch features and a momentum branch is used to extract the information of the whole image. Only about ten sparse patches are used to predict the information of the whole image. The prediction is supervised by maximizing the cosine similarity between the joint-patches representation and global representation. Different from the previous contrastive learning method [10], MaskDeep needs no projector after the momentum encoder.

Extensive experiments on both classification and dense prediction downstream tasks show the effectiveness and generality of our MaskDeep. As the Figure 1, trained on ResNet50 with 200 epochs, MaskDeep achieves 71.2% Top1 accuracy linear classification on ImageNet, and 41.5% bbox mAP on COCO object detection tasks, which surpasses current self-supervised methods. In the Table 7, MaskDeep outperforms the SoCo, which specifically designed for object detection, by 0.1 mAP.

Our contributions can be summarized into three-folds.

• $\bullet$

  We propose a novel self-supervised learning approach, i.e., MaskDeep, in which views are masked from an well-designed implicit representation space.

• $\bullet$

  We align the joint-patches representation and the global representation directly without projector. Both local object understanding and global context information reasoning are concerned.

• $\bullet$

  MaskDeep achieves the state-of-the-art performance on both classification and dense prediction tasks, i.e., classification (71.2% Top1 ACC), object detection (41.5% box AP), and instance segmentation (36.1% mask AP) when trained 200 epochs.

To evaluate the generality of MaskDeep, we evaluate MaskDeep on both classification and dense prediction tasks, including ImageNet linear classification[8], PASCAL VOC object detection[9], COCO object detection and instance segmentation[18].

We conduct a series of ablation studies step by step to demonstrate the effectiveness of each component in MaskDeep. Unless specified, all the ablation experiments are constructed using a default batch size of 1024 and 100 pre-training epochs. The configuration for downstream tasks remains the same. The ResNet is evaluated on ImageNet linear-probing benchmark.

Because we follow the main paradigm of contrastive learning methods and use the cosine similarity loss function, we regard the BYOL as a baseline. The ablation results are shown in the Table 2. The first line represents the performance of the BYOL when trained 100 epoches. The second line represents a naive idea for masking patches in the representation space. In the naive idea, the group of point features are sampled from the last layer of the encoder directly. As shown in the Table 2, the naive idea achieves 64.9% top-1 acc, while SimCLR achieves 64.8% and BYOL achieves 66.5% with the same 100 training epochs. The baseline shows a little poor performance, but it works and learns representation from the naive pretext task. It lags behind image-level self-supervised methods, but surpasses object-level self-supervised methods, such as DenseCL which only achieve 63.6%, as shown in the Table 7.

After concerning hierarchical property by the Hierarchical Deep-Masking module, our MaskDeep obtains an improvement of +3.0%. It achieves 67.9% top-1 acc and surpasses the BYOL by 1.4%. With the Hierarchical Deep-Masking module, it is comparable with the current self-supervised learning methods. Further, we introduce a multi-group strategy, which is specific to our method. Intuitively, the multi-group strategy increase the number of samples, such obtains an improvement of +0.9%. Finally, we introduce a multi-target strategy to provide more supervision signals. The strategy makes the training process more stable. As shown in the Table 2, an improvement of +0.8% for ResNet is found.

In this section, we further examine different hyperparameters for each component in MaskDeep.

In the Table 3 (a), the level 1 represents only using the output of the last stage $\{P_{5}\}$, the level 2 represents using the output of the last two stages $\{P_{4},P_{5}\}$, the level 3 represents using the output of the stages $\{P_{3},P_{4},P_{5}\}$, the level 4 represents using the output of the all stages $\{P_{2},P_{3},P_{4},P_{5}\}$,. The results show that when only using the output of the last stage $\{P_{5}\}$, the performance drops about 1.4%. It indicates the indispensable role of the Hierarchical Deep-Masking module. Except only using the $\{P_{5}\}$, our method is robust to the other levels. The results show that level 4 is a little harmful. This could be caused by the large resolution of $\{P_{2}\}$, the semantic information of each point is insufficient. Our method achieves the best performance with the output of the stages $\{P_{3},P_{4},P_{5}\}$.

Table 3 (b) shows the influence of the dimension of the Hierarchical Deep-Masking module. While the output dimension of ResNet50 is $[256,512,1024,2048]$, the dimension 512 performs best. We summarize that the dimension should be close to the output dimension of the encoder. Small dimension will limit the representation capacity of each patch feature, while large dimension will decrease the training speed and increase learning difficulty.

The implicit representation space from the Hierarchical Deep-Masking module is an important part in our method. We sample all patch features from the implicit representation space. We assume that each patch features contains local semantic corresponding to specific image region, such we can infer from several patch features to the global image content. Therefore, we visualize the implicit representation space to verify our idea with Grad-CAM [22] and clustering map. In the visualization process, we only apply the resize preprocessing method for view1. When applying Grad-CAM, we only use one local representation $q(\mathcal{H}^{1})$ and one global representation $\mathcal{G}^{2}$ to calculate the cosine similarity loss. We use the gradient at each scale level as the weight for the output of the Hierarchical Deep-Masking module. For clustering map, we select 5 point features as the initial features. Then, we calculate the similarity between all point features across space dimension with the 5 initial features. Finally, we cluster the points with highest similarity and show a cluster with the same color.

Fig. 4 demonstrates the visualization results. The Grad-CAM results show that all scale level of the Hierarchical Deep-Masking module can attention the main object in the image successfully. Regions covering the objects have higher activation, while background regions (e.g., ground or sky) receive lower attention. For different scale level, the granularity of attention is different. $P_{3}$, with largest resolution, shows the fine-grain attention. In the first image, the contours of pig head are accurately perceived. The clustering maps show that patch features with similar context can be clustered together. The background and the subject are distinguishable. The distinguish is shown more clearly in fine-grain $P_{3}$ level. Take the first pig picture for example, the background is clustered into the same cluster and the most pig body have high semantic similarity. Further, among different scale level, the clustering results are consistent. The visualization results support our assumption that each patch features contains specific local semantic. The process of aligning implicit local patch representation and global image representation is meaningful.

In this paper, we propose a simple but strong self-supervised learning method, i.e., MaskDeep. We aligns local and global representations in a single online-momentum framework. We show that reasoning from sparse patch features to infer the global image content is a strong self-supervised learning task. Furthermore, we first construct the patch features from an implicit representation space instead of masking the source image. We visualize the implicit representation space. The results prove that each point contains the semantic information of the corresponding context. We propose three novel designs: the hierarchical deep-masking module, the multi-group strategy and the multi-target strategy. These designs improve the efficiency of our method. Our MaskDeep outperforms current self-supervised methods on classification, object detection and segmentation downstream tasks. Trained on ResNet50 with 200 epochs, MaskDeep achieves state-of-the-art results of 71.2% top-1 accuracy linear classification on the ImageNet dataset. On COCO object detection tasks, MaskDeep achieves state-of-the-art results of 41.5% bbox mAP and 36.1 mask mAP, which outperforms most object-level self-supervised method.