Self-supervised Image Clustering from Multiple Incomplete Views via Constrastive Complementary Generation

Contrastive learning Chen et al. [2020]; Deng et al. [2018]; Tian et al. [2020]; Tsai et al. [2020] is an essential method for unsupervised learning Bengio et al. [2013]. Its major goal is to maximize feature space similarity between positive samples while reducing the distance between negative samples. Most contrastive learning methods Yuan et al. [2021]; He et al. [2020] are built to handle various data augmentations or modal augmentations. Especially in the field of computer vision, contrastive learning methods have produced excellent results Van Gansbeke et al. [2020]. As an example, methods such as SimClR or MoCo Xu et al. [2021]; He et al. [2020]; Girshick and He minimize the InfoNCE loss function van den Oord et al. to maximize the upper bound of mutual information. Because dealing with negative samples is inconvenient, later contrastive learning algorithms Grill et al. [2020]; Chen and He [2021] have successfully replaced the contrast task with the prediction task without the need for negative samples. More tools can be used in contrastive learning. As the first comparative hashing method, CMH Hu et al. [2022] solves the problem of the negative impact of false negative samples.

Almost all existing contrastive learning methods Xu et al. [2021]; He et al. [2020]; Girshick and He ; Grill et al. [2020]; Chen and He [2021] are designed to handle single-view data, exhaustively exploring various data augmentations to build different views/augments. To the best of our knowledge, this is the first time double contrastive learning has been used on the IMC problem to handle the challenge of mutual promotion of consistency learning and data recovery from a different perspective. The introduced double contrastive learning enhances consistency in learning while alleviating inconsistency. We use latent representations to learn the consistency between distinct views and a BYOL Grill et al. [2020] based prediction network to attenuate the inconsistency between different views. These two are integrated and mutually promote each other, which greatly improves the clustering performance. As far as we know, this is the first work that uses the BYOL structure for data recovery.

According to the utilization method of multimodal data, traditional IMC methods can be roughly divided into four categories: IMC based on kernel learning Liu et al. [2020]; Rai et al. [2010], IMC based on matrix factorization, IMC based on graph, and IMC based on spectral clustering. Trivedi et al. proposed a kernel-based method Rai et al. [2010] that recovers the kernel matrix of incomplete views from the kernel matrix of complete views but requires one of the views to be all complete. To address this limitation, matrix factorization-based IMC methods Liu et al. [2013]; Zhao et al. [2017]; Yang et al. [2020a]; Hu and Chen [2018]; Li et al. [2014] use lower ranks to project parts of the data into a common subspace, formally similar to the K-means relaxation method MacQueen et al. [1967]. In the follow-up work, Linlin et al. Zong et al. [2020] introduced an indicator matrix to map non-existing instances into 0-element vectors, and Zhiqi et al. Yu et al. [2022] introduced a fusion matrix while learning the self-representation in the subspace, which further developed the IMC method based on matrix decomposition. However, feature-based matrix factorization methods generally cannot explore nonlinear data structures, while graph-based methods Peng et al. [2019]; Wen et al. [2021] can effectively deal with nonlinear data structures. For example, PSIMVC-PG Li et al. [2022] proposes a parameter-free clustering based on a prototype graph framework to handle non-linear data structures with lower complexity. In addition, the spectral clustering-based method Wang et al. [2019] can also deal with the nonlinearity in the data by constructing a Laplacian graph. For example, FMDC Qiang et al. [2021] uses a very short time to construct an anchor graph on different views, which significantly reduces the time cost. IMSC-AGLWen et al. [2018] used graph learning and spectral clustering to learn incomplete public representations for the first time, and achieved better results.

However, traditional IMC methods have limitations in terms of representation ability and high complexity. In recent years, the self-supervised IMC method Jiang et al. [2019]; Wang et al. [2018]; Xu et al. [2019]; Lin et al. [2021]; Zhang et al. [2019]; Wang et al. [2015]; Andrew et al. [2013]; Wang et al. [2021] based on deep learning has attracted attention, and this method combined with deep neural networks shows strong generalization ability. The differences between this work and existing studies are as follows. CIMIC-GAN belongs to a deep IMC framework, which has better scalability than traditional IMC methods. Most existing IMC methods Liu et al. [2013]; Zhao et al. [2017]; Yang et al. [2020a]; Xu et al. [2019]; Lin et al. [2021]; Wen et al. [2019]; Andrew et al. [2013]; Wang et al. [2019] can only infer and generate missing data with consistency learned from complete data, but can’t properly use incomplete data, wasting the hidden information contained in the training set’s incomplete data. In contrast, we leverage GAN to fill in the missing views and train the prediction network with the new data generated. Therefore, we target the problem that incomplete data in IMC cannot be used or trained. It can even outperform other methods in extreme circumstances where the missing rate is exceedingly high.

Recently, there have been several approaches utilizing GANs Goodfellow et al. [2014] for multi-view learning, such as CycleGAN Zhu et al. [2017]; Yang et al. [2020b] and StarGAN Choi et al. [2018]. Wang et al. proposed a generative adversarial network (GAN) based deep IMC method Wang et al. [2018]; Goodfellow et al. [2014]. It uses one view to generate missing data for another view of recurrent GAN Zhu et al. [2017]. However, the accompanying challenge of cross-modal generation is how to narrow the differences between different modalities. To overcome this problem, CAD Hu et al. [2021] fills the heterogeneity gap between modalities by matching the synthesized data in the feature space with the help of two groups of specific GANs and a discriminative mechanism across modalities. However, GANs mostly adopt shallow models, which are difficult to capture deep semantic understanding. AIMC Xu et al. [2019] uses deeper networks to perform missing data inference while finding the common latent space of multiple view data.

Different from the above works, these methods mainly focus on cross-domain data generation. In our framework, the adversarial process only acts on a single view, which preserves the private information of the view to the greatest extent, thus effectively realizing incomplete data recovery in independent feature space. Moreover, the adversarial process is integrated with the target reconstruction, so that the pseudo-data distribution contains more complementary information, which is beneficial for the network to capture a better clustering structure. Furthermore, Consistent GAN Wang et al. [2018] for the two-view IMC problem can only handle dual-view data, it uses one view to generate missing data from the other and then performs clustering on the generated full data. While CIMIC-GAN can be extended and applied to the IMC problem with an arbitrary number of views.

Given an instance, it usually contains multiple views or multiple modalities, such as RGB images or depth maps, representing different perspectives of the same object. Regarding this phenomenon, the incomplete multi-view clustering problem tries to solve the premise that there are missing views in an instance and can have good clustering performance. One instance generally has multiple views, which may or may not be complete. As shown in Fig. 2, assuming that a dataset has two views, namely $A$=2, there are $(N+\widetilde{N})$ instances in total, such a dataset can be divided into two parts: complete data and incomplete data. In the set of complete instances, we use $X_{n}^{(v)}(v=1,\ldots,A)$ to denote the feature vector for the $v$-th view of the $n$-th instance. In the set of incomplete instances, we use $I_{\widetilde{n}}^{(v)}(v=1,\ldots,A)$ to denote the feature vector for the $v$-th view of the $\widetilde{n}$-th instance. $\forall X_{n}^{(v)},I_{\tilde{n}}^{(v)}\in\mathbb{R}^{d_{v}}$ , where $d_{v}$ is the dimensionality of the $v$-th view. Our goal is to group all the $N+\widetilde{N}$ instances into $K$ clusters.

Without loss of generality, we make $A=2$, i.e., two views are involved. Define a two-view dataset as a collection of $\left\{{X}^{1},{X}^{2},{I}^{1},{I}^{2}\right\}$ of $(N+\widetilde{N})$ instances, where $X^{1}$ and $X^{2}$ represent instances that exist and are aligned in two views, and where $I^{1}$ and $I^{2}$ represent instances that exist only in the first and second views, respectively.

Multi-View Clustering is an unsupervised learning problem. The key to solve this problem is to learn strong representations as well as the consistency of them. Considering that contrastive learning has a strong representation ability in the field of unsupervised learning, we incorporate a double contrastive learning module (Network1) into our model, CIMIC-GAN, to learn consistency of the representations of the views. On the other hand, considering the missing data of instances, we fill the incomplete data in a generative adversarial way to obtain more complete auto-encoding representations of views (Network2). The training process of CIMIC-GAN is divided into three steps:

Step 1: Train Network1 with $\left\{X^{1},X^{2}\right\}$. The aligned complete view data $X^{1}$ and $X^{2}$ is respectively used in the encoder $f_{1}$ and $f_{2}$ of Network1 to obtain the latent representations. $Z^{1}$ and $Z^{2}$ is the latent representations of the first view and the second view, respectively. Based on $Z^{1}$ and $Z^{2}$, there are three objective functions that need to be optimized in this step: i) The loss obtained by reconstructing different views through the autoencoder is denoted by $\mathcal{L}_{rec}$. ii) By using contrastive learning, we maximize mutual information between $Z^{1}$ and $Z^{2}$. The corresponding loss is denoted by $\mathcal{L}_{cc}$. iii) Through contrastive learning without negative samples, $Z^{1}$ and $Z^{2}$ are predicted by two symmetrical networks using predictors to alleviate the inconsistency between distinct viewpoints, and the loss function is denoted by $\mathcal{L}_{pre}$. Ultimately, we propose the overall objective function of network 1:

The parameters $\alpha$ and $\beta$ are the balanced factors on $\mathcal{L}_{rec}$ and $\mathcal{L}_{pre}$, respectively. We set these two parameters as 0.1, which will be proven in the following parameter analysis experiments.

Step 2: Train Network2 using $\left\{I^{1},I^{2}\right\}$. Feed the incomplete view data $I^{1}$ and $I^{2}$ into the encoder of Network2. Note that, the encoder $f_{1}$, $f_{2}$ and the decoder $g_{1}$, $g_{2}$ already have converged in step 1. The decoder $g_{1}$ and $g_{2}$ will be equivalent to being well initialized as generators in the GAN structure. Each decoder incorporates the corresponding discriminator $D_{(v)}$ to form a typical GAN network. In detail, $\widehat{I}^{(v)}$ is first generated according to $I^{(v)}$. Then, the discriminator $D_{(v)}$ will judge whether $\widehat{I}^{(v)}$ generated by the decoder is true. Till the discriminator cannot provide that correctly, the generator $g_{(v)}$ converges. The purpose of this step is to train a powerful generator to generate the missing data of imcomplete view as well as expand the training dataset. As shown in Figure 2, according to the incomplete view $I^{(v)}$, the corresponding missing data $\widehat{I}^{(v)}$ is generated through Network2 and fills it in the corresponding modality in pair, $\left\{I^{1}+\widehat{I}^{2},I^{2}+\widehat{I}^{1}\right\}$.

Step 3: Train Network1 again with $\left\{I^{1}+\widehat{I}^{2},I^{2}+\widehat{I}^{1}\right\}$ to achieve a consistent representation. Feed the pseudo-complete data of different views $\left\{I^{1}+\widehat{I}^{2},I^{2}+\widehat{I}^{1}\right\}$ to Network1, and perform consistency learning as step 1. The optimization objective is the same as the first step, and the balance factor has not changed, as follow Eq.(1). The role of this step is to get sufficient training data to make the Network1 model more generalized and robust. After Network1 has converged, we feed $\left\{{X}^{1},{X}^{2},{I}^{1},{I}^{2}\right\}$ to Network1. This will give us latent representations of all views, including views with missing data. Finally, a consistent representation is obtained by concatenating representations of different views (see 2), which is further input to a specific cluster (Kmeans or other adaptive deep clustering methods with enhanced performance, such as GATCluster Niu et al. [2020], SpiceNiu et al. [2021], etc.) to achieve qualified multi-view image clustering.

The loss functions used in the proposed model is provided in this section.

We evaluate CIMIC-GAN on four clustering problem datasets. 1) Caltech101-20 Li et al. [2015] consists of 2,386 images of 20 subjects, and we use two views of HOG and GIST features, with 1984 and 512 as the feature dimensions, respectively. 2) Scene-15 Fei-Fei and Perona [2005] consists of 4,485 images distributed over 15 Scene categories, and we use two views of PHOG and GIST features, 20D and 59D feature vectors, respectively. 3) LandUse-21 Yang and Newsam [2010] consists of 2100 satellite images from 21 categories , and we use two views of PHOG and LBP features, 59D and 40D feature vectors, respectively. 4) A large dataset, Noisy MNIST Wang et al. [2015], consists of 70,000 instances of 10 classes. We randomly select 15,000 original instances as view 1 and 15,000 Gaussian noise-added instances as view 2. We summarize the detailed statistics of the datasets in Table 1.

We conduct all the experiments on the platform of ubuntu 16.04 with Tesla P100 Graphics Processing Units (GPUs) and 32G memory size. Our model, method and baseline are built on the pytorch 1.7.0 framework. Based on extensive ablation studies, the bacth size is set to 256 and the epochs of the three steps of training are 300, 250, and 200, respectively. The missing rate is fixed to 0.5, the momentum $m$ is fixed to 0.6, the entropy parameter $\gamma$ is fixed to 9 and trade-off hyper-parameters $\alpha$ and $\beta$ are fixed to 0.1. We utilize Adam optimizer Kingma and Ba [2015] with default parameters and a learning rate of 0.0001. We set the dimension of the autoencoder to d-1024-1024-1024-128, where d is the dimension of the input data. We simply adopt a dense (i.e., fully-connected) network where each layer is followed by a batch normalization layer and a ReLU layer. The structures of the autoencoder for different modalities are the same. MLPs are used to implement the contrastive prediction module, and all MLPs use batch normalization after each linear layer. Each MLP has two linear layers, with the ReLU activation function added in the middle of each of them.

Compared methods and evaluate metrics: We compare CIMIC-GAN with several baselines, including Autoencoder in Autoencoder Networks (AE2Nets) Zhang et al. [2019], Unified Embedding Alignment Framework (UEAF) Wen et al. [2019], Doubly Aligned Incomplete Multi-view Clustering (DAIMC) Hu and Chen [2018], Efficient and Effective Regularized Incomplete Multi-view Clustering(EERIMVC) Liu et al. [2020], Deep Canonically Correlated Autoencoders (DCCAE) Wang et al. [2015], Partial Multi-View Clustering (PVC) Li et al. [2014], Binary Multi-view Clustering (BMVC) Zhang et al. [2018], Deep Canonically Correlated Analysis (DCCA) Andrew et al. [2013], Perturbation oriented Incomplete Multi-view Clustering (PIC) Wang et al. [2019], and COMPLETER  Lin et al. [2021]. The clustering performance is evaluated with three metrics: Accuracy (ACC), Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI). More details on these indicators can be found in Amigó et al. [2009]. A higher value of these evaluation indicators can reflect an advanced clustering performance.

Missing rate: To uniformly evaluate the performance of CIMIC-GAN on incomplete multi-view data, we randomly select $\widetilde{N}$ instances as incomplete data and randomly remove some views from each of them. The Missing Rate(MR) is defined as $\frac{\widetilde{N}}{N+\widetilde{N}}$. The larger the missing rate, the more incomplete data.

We design experiments with different methods at different missing rates to further demonstrate the efficiency of our method at high missing rates. On the Caltech101-20 dataset, we divide the missing rate into 10 cases ranging from 0 to 0.9 with a 0.1 interval. We adjust the batch size to 128 when the missing rate is 0.9. When the missing rate is 0, since there is no incomplete data, we will directly skip the first and second steps, and we use the complete data $\left\{X^{1},X^{2}\right\}$ to train CIMIC-GAN in an end-to-end manner.

Both DCCA and DCCAE learn nonlinear analysis between paired views through neural networks. BMVC needs paired views to encode and embed in binary space, and AE²Nets needs to first convert different views into a unified comprehensive representation. However, when the missing rate is 1, it means that there are no paired samples in the original data distribution, which makes the above four methods not work. COMPLETER relies heavily on the mutual information of complete sample pairs, so the phenomenon of model collapse occurs, and the accuracy rate is only 26.33% when all views are missing. The accuracies of UEAF, DAIMC, EEIMVC, PVC, PIC, and CIMIC-GAN are 32.15%, 26.77%, 35.00%, 33.24%, 37.12%, and 34.30%, respectively, in terms of the different number of missing views. The two traditional methods, PIC and EERIMVC, use the similarity of other views to fill in the missing similar items. Although the filling method is simple, it is effective when the missing rate is 1. However, our method is a deep method and infers missing data rather than missing similarities, so it enjoys higher interpretability. Even without paired samples, our method still approximates the original data distribution and achieves near-optimal performance.

As shown in Fig. 7, we observe the accuracy of different methods under different missing rates and can get the following results: 1. CIMIC-GAN outperforms almost all tested baselines in all missing rates, which validates the effectiveness of the proposed method. 2. Due to the pseudo-data distribution generated by GAN fitting more missing samples, CIMIC-GAN outperforms the suboptimal method by 3.68% with a missing rate of 0.9. This experiment also demonstrates that the model has strong robustness and generalization ability.

In the contrastive prediction module, if the parameters of the online network updated by the current step are directly copied to the target network, the target network’s training state will eventually be completely disordered, causing the incapacity to train to oscillate. Therefore, the coordination of states by introducing momentum $m$ is a natural solution. Consistency and complementarity represent the same semantics and different semantics of different views, respectively. If the network is updated faster, the learned representation will contain more consistency, otherwise, it can retain more complementary information. So the introduction of momentum $m$ can also help us find the most suitable balance point. We designed a set of experiments on the Caltech101-20 dataset with a missing rate (MR) of 0.5. As shown in Fig. 8, the clustering effect is observed by adjusting different values of the momentum $m$.

From the results obtained in Fig. 8, it can be seen that: 1) When the value of $m$ is small, the parameters of the target network change too quickly, ignoring the complementarity between hidden representations. 2) When $m$ is large ($m>0.6$), the gradient update is slow the consistent learning is hindered, and the accuracy will experience a significant drop again. Therefore, too fast or too slow parameter changes are not conducive to consistent learning, because the consistency and complementarity of different views are not balanced. So choosing an appropriate momentum as a balance point can promote the consistent learning of latent representations from different views. In the update of the target network, 60% of the parameters keep the same value, and the clustering performance is optimal at this time.

For methods that can only handle complete view data, we fill in the missing data with the mean of the same view. For a fair comparison, we use the default network structure and parameters for all methods. We test all methods with an MR of equal to 0.5. Experimental results on the above four databases are listed in Table 2. Compared with the ten clustering methods mentioned above, our CIMIC-GAN achieves state-of-the-art clustering performance. We can observe the following from the results: 1) On all four datasets, CIMIC-GAN outperforms other methods. 2) In most cases, DAIMC, PVC, EERIMVC, and PIC obtain worse IMC performance than the other methods. These methods all consider consistency learning and data recovery as two separate parts. The contrastive consistency module in CIMIC-GAN strengthens consistency learning, and the contrastive prediction module alleviates inconsistency. There is a trade-off between these two parts. We can conclude that using double contrastive learning to simultaneously mine complementary and consistent information from multiple perspectives can improve clustering performance. 3) Methods such as DCCAE, PVC, BMVC, and PIC only learn consistency, while DAIMC learns representations that only consider complementary information. Thus, CIMIC-GAN far outperforms the above methods. This shows that effectively utilizing the complementary and consistent information of images can improve clustering performance. 4) CIMIC-GAN outperforms methods with incomplete data that do not participate in training, such as UEAF, DCCA, PIC, COMPLETER, etc. This shows that it is necessary for us to use GAN to mine the hidden information in incomplete data.

To demonstrate that our method can in principle be extended to more than two views, we implement a set of comparative experiments on different number of views. A dataset Fashion Xiao et al. [2017] with three views is introduced here to support this set of experiments. We leverage this dataset to construct a dataset only with two views, Fashion-2V, whereas we construct Fashion-3V with three views that is missing two views, and Fashion-3V* with three views that is missing one view. See Table 3 for more detailed settings of these three datasets.

To handle instances with more than two views, CIMIC-GAN needs some extensions. Specifically, we involve another set of network2 to learn the latent representation $Z^{3}$ for the third view. Similarly, we need to introduce two additional sets of prediction networks to realize the mutual prediction of $Z^{1}$ and $Z^{3}$, and $Z^{2}$ and $Z^{3}$. Correspondingly, we maximize the mutual information between the latent representations of any two of these three views (Formula 12). Fig. 11 shows the results of training on Fashion-2V, Fashion-3V and Fashion-3V* for 200 epochs. As the number of instance views changed from two to three, more features are involved and each evaluation metric improved significantly. Among them, the performance of Fashion-3V* is slightly higher than Fashion-3V in an all-around way. This is because our method alleviates the negative impact of missing views, so their two curves are very close, which further proves that CIMIC-GAN has strong robustness and adaptability.

In this section, we analyze CIMIC-GAN on the Caltech101-20 dataset from two perspectives, i.e., parameter analysis and ablation studies.

Our method has two user-specified parameters, i.e., the reconstruction loss trade-off parameter $\alpha$, and the contrastive consistency loss trade-off parameter $\beta$. To evaluate the impact of $\alpha$ and $\beta$, we change their value in the range of 0.001, 0.01, 0.1, 1, 10 when training the model on dataset Caltech101-20. As shown in Fig. 9, when $\alpha$ and $\beta$ are set to 0.1 and 0.1, respectively, the experimental results are the best. Moreover, CIMIC-GAN was run with the same hyper-parameter setting on the four datasets and achieved good performance (see Section 4.4), which means CIMIC-GAN is not very sensitive concerning the hyperparameters.

We conduct ablation studies with four variants of CIMIC-GAN to show the effectiveness of its different modules. (1) We only use the reconstruction module multi-modal clustering. (2) We only use the contrastive prediction module multi-modal clustering. (3) We only use the contrastive consistency module multi-modal clustering. (4-6) We combine the above three modules in pairs. (7) We use the complete Network1 with three modules to learn consistent cluster predictions of multiple modalities. (8) Compared with setting “(7)”, we additionally added GAN in Network2.

Table 4 shows the loss components and experimental results corresponding to the four variants. In row (2), it can be seen that optimization alone with contrastive prediction loss $\mathcal{L}_{pre}$ may lead to trivial solutions or model collapse because $\mathcal{L}_{rec}$ is not optimized and thus the low-dimensional latent representation loses more complementary information. This also demonstrates that the key task of IMC is to mine complementary information to achieve consistent predictions across multiple modalities. Comparing row (6) with row (4) and (5), dual contrastive learning is more effective than a single consistency learning module. Comparing row (7) with row (1), it can be inferred that double contrast learning has a great improvement in the clustering performance, and shows that the improvement in consistency by the double contrastive learning module is huge. Comparing row (7) with row (8), the introduction of GAN in the encoding process makes the hidden information of incomplete data more fully utilized. Significantly, one can find that each module of CIMIC-GAN has improved the clustering performance, which further proves the effectiveness of our motivations and the proposed framework.

As shown in Fig. 10, we show t-sne Van der Maaten and Hinton [2008] visualizations of the obtained representations on four datasets. In the experiments, the missing rate is fixed to 0.5. The common representation itself is a fusion representation, which fuses a complete view representation and a missing view representation, where the missing view representation is filled by incomplete data through a contrastive prediction module. Therefore, Fig. 10 illustrates the learning process of Network1. With the increase of epochs, more advanced semantic information is achieved, the common representations learned by Network1 become more compact and independent, and the density of clusters is higher. The advanced semantic information enables better consistent learning. For all these four datasets, Network1 converges at around 200 epochs, indicating that our proposed model is very robust.