Preventing Dimensional Collapse of Incomplete Multi-View Clustering via Direct Contrastive Learning

As mentioned above, existing incomplete multi-view clustering methods can be divided into traditional incomplete multi-view clustering and deep incomplete multi-view clustering.

Traditional incomplete multi-view clustering methods can be divided into matrix factorization-based methods [29, 30], kernel learning-based methods [31, 32, 33], and graph-based methods [34, 35, 36]. Matrix factorization-based methods attempt to project incomplete data into a low-dimensionalal common subspace satisfying low-rank constraints by generating latent representations. In this category, partial multi-view clustering (PVC) [11] is one of the earliest works that used non-negative matrix factorization (NMF) [37] to obtain a consistent latent representation of incomplete data. Furthermore, MIC [30] used the average of the observed samples in the corresponding views to complete unobserved samples, and then performs joint weighted NMF to learn the shared subspace between views. Kernel learning-based methods attempt to recover kernel matrices that are consistent across multiple views in various ways. Liu et al. [38] proposed two multi-kernel k-means methods with incomplete kernels. They perform joint imputation on incomplete kernel matrices and combine them for clustering, which is a successful solution with good results. Graph-based methods aim to explore the original geometric structure of the data by exploiting different graphs of all sample relationships. Based on this idea, Wen et al. [13] proposed to construct graphs via subspace learning, which fused the partitions generated by each view into a unified matrix.

As far as we know, deep learning has attracted much attention due to its strong scalability and excellent ability to handle large-scale data, and many incomplete multi-view clustering methods have been proposed through various different deep learning frameworks. Based on generative adversarial networks, Wang et al. [39] developed a view-specific generative adversarial network with multi-view cycle consistency to generate missing data for one view based on shared representations. Based on the contrastive learning, Yang et al. [40] use a noise-robust contrastive loss, which mitigates the effect of false negatives and overcomes the incomplete information challenge.

Contrastive learning [28, 41, 42] is a discriminative representation learning framework based on the idea of contrast, which is mainly used for unsupervised representation learning. The main purpose of contrastive learning is to maximize the similarity between instances of the same class and minimize the similarity between instances of different classes. Compared with generative learning, contrastive learning does not need to pay attention to the tedious details of instances, but only needs to learn to distinguish data in the feature space at the abstract semantic level. As such, the generalization ability of the model based on contrastive learning is stronger. Due to its advantages of simple model and saving hardware resources, it has been widely used in the fields of image generation [43], image classification [44] and clustering [26].

Contrastive learning has currently achieved promising results in computer vision. In terms of image generation, Liu et al. [45] employed contrastive learning to encourage diverse image synthesisers. The proposed DivCo improves the performance of diverse image synthesis without sacrificing the visual quality of the generated images. In terms of representation learning, Kaveh et al. [46] developed a multi-view representation learning method to solve the problem of graph classification through contrastive learning. In terms of clustering, Li et al. [27] proposed a one-stage online clustering method called contrastive clustering, which explicitly performs instance-level and cluster-level contrastive learning. Contrastive learning has become the main research direction of unsupervised learning because of its excellent achievements in various fields.

We treat the dataset as the original feature, where the dataset has multiple views with $K$ common clusters. For multi-view data, the existing methods generally try to learn the salient features of the sample from the information contained in the original data. In particular, deep autoencoders can accomplish the above tasks and have been applied in many research fields due to their excellent performance. In this paper, we use $\left\{\mathbf{X}^{v}\in\mathbb{R}^{N\times M_{v}}\right\}_{v=1}^{V}$ to represent a multi-view dataset with $V$ views and $N$ samples, and define $\mathbf{x}_{i}^{v}\in\mathbb{R}^{M_{v}}$ to represent the $M_{v}$-dimensional samples from the $v$-th view. In order to obtain the feature information of each view, we use autoencoder to transform heterogeneous multi-view data into clustering-friendly features. Specifically, the feature representation $\left\{\mathbf{Z}^{v}\right\}_{v=1}^{V}$ of $v$-th view is learned by encoder $f_{ec}^{v}\left(\mathbf{X}^{v};\theta^{v}\right)$ and decoder $f_{dc}^{v}\left(\mathbf{Z}^{v};\phi^{v}\right)$, where $\theta^{v}$ and $\phi^{v}$ are network parameters, and $\mathbf{z}_{i}^{v}=f_{ec}^{v}\left(\mathbf{x}_{i}^{v}\right)\in\mathbb{R}^{D}$ as the $D$-dimensionalal latent feature of the $i$-th sample. Therefore, in the $v$-th view, the reconstruction loss between the encoder input $\mathbf{X}^{v}$ and the decoder output ${\mathbf{X}^{*}}^{v}\in\mathbb{R}^{N\times M_{v}}$ can be defined as $\mathcal{L}_{\mathrm{Z}}^{v}$, so the reconstruction loss of all views can be expressed as:

The representation $\mathbf{Z}^{v}$ is defined by $\mathbf{Z}^{v}=f_{ec}^{v}\left(\mathbf{X}^{v}\right)$. The deep multi-view clustering methods based on autoencoders generally aimed to learn cluster-friendly common representations by maximizing the consistency between latent features of different views. However, the existing multi-view clustering methods have the following two issues: (1) Most methods suffer from the dimensional collapse problem that the latent feature vector learned in the model collapses along some dimensions so that the dimension of the embedding vector cannot be fully utilized. Although some methods based on contrastive learning solve the above problems by adding additional projection heads, this approach will result in increased computation and many parameters in the projection heads are unnecessary. (2) Some multi-view clustering methods perform reconstruction learning and consistency learning on the same latent features. The purpose of reconstruction learning is to enable the learned latent features to contain more view-private information of the original features, while consistency learning hopes to mine useful common information between views, which will lead to poor quality of the learned latent features and affect the final clustering results.

To avoid the above problems, we design a multi-view contrastive learning module. Inspired by [47], the projector in contrastive learning is of great significance to prevent dimensional collapse, and a linear projector only need to satisfy the diagonal and low rank to effectively prevent dimensional collapse. The details can be found in [47]. Accordingly, we adopt a novel way to feed the sub-vectors of the latent representation $\mathbf{Z}^{v}$ directly into the spectral contrastive loss, which satisfies the above conditions and optimizes the representation space, thus preventing the dimensional collapse that occurs in the clustering model. At the same time, since we perform consistent learning on carefully selected sub-vectors, the remaining irrelevant information does not participate in the contrastive learning, which helps to filter out the view-private information of features and obtain higher-quality common representations. The specific content of this part is elaborated in the next subsection.

It is obvious that the latent feature $\left\{\mathbf{Z}^{v}\right\}_{v=1}^{V}$ of each view obtained from the original features does not separate the consistent information and private information of each view. Most existing methods directly perform contrastive learning on features $\left\{\mathbf{Z}^{v}\right\}_{v=1}^{V}$, which causes private information in features to interfere with consistent learning. To solve this problem, Xu et al. [26] avoid the above situation by adding MLP after the feature $\left\{\mathbf{Z}^{v}\right\}_{v=1}^{V}$ to obtain advanced features. Although this method can resolve the conflict between the reconstruction objective and the consistency objective, the addition of MLP increases the complexity of the model and reduces the efficiency of the model. Therefore, we adopt a simple way to use sub-vectors representing features and feed the sub-vectors directly into the contrastive loss to solve the above problem, where we utilize the $\left[0:d_{0}\right]$-dimensional vectors of latent features as sub-vectors for consistency learning.

Since the purpose of contrastive learning is to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs, the sub-vector $\mathbf{z}_{i}^{*v}$ defined for each feature in our model has $(VN-1)$ feature pairs, where $\left\{\mathbf{z}_{i}^{*v},\mathbf{z}_{i}^{*n}\right\}_{n\neq v}$ are $(V-1)$ positive pairs and the rest $V(N-1)$ pairs are negative pairs. Based on the spectral contrastive loss [48], we adopt the following loss to achieve the consistency objective, so that the features focus on learning the common semantics of all views:

where $C_{N}^{2}$ denotes the combinatorial operations.

The feature representation $\mathbf{Z}^{v}$ obtained by our designed network avoids model collapse through direct contrast learning on its sub-vector and retains more useful information by further optimizing the representation space. In the case of complete data, we can get a cluster-friendly representation through the above steps and run k-means on the feature representation to get good results. Whereas, most of the data in practical applications are incomplete. Such incomplete data will greatly affect the quality of the learned latent representation and make multi-view clustering methods limited. Accordingly, researchers have proposed incomplete multi-view clustering methods to deal with incomplete data. Most of the current incomplete multi-view clustering methods use the mean filling method to solve the problem of missing information. But this method introduces noise information, and the learned representations cannot fully utilize the consistent information of views. To address this issue, we attempt to use a missing view recovery module to predict and recover missing views from the existing available views, so that our method can reduce the impact of missing information on clustering performance.

In this module, for the latent feature structure of the original data obtained by the autoencoder, we encourage that $\mathbf{Z}^{p}$ can be determined by $\mathbf{Z}^{q}$ by minimizing the conditional entropy $H\left(\mathbf{Z}^{p}\mid\mathbf{Z}^{q}\right)$. This mechanism enables the missing views to be determined by the available views, which can not only achieve the representation recovery of the missing views but also cleverly discard the inconsistent information between different views to obtain a better consistent representation. We represent the conditional entropy by the following equivalent log conditional likelihood transformation:

However, it is difficult to directly estimate the expected value in equation (3). A general approach to approximate this objective is by introducing a variational distribution $\mathcal{Q}\left(\mathbf{Z}^{p}\mid\mathbf{Z}^{q}\right)$ and maximize the lower bound of the expectations, i.e., $\mathbb{E}_{\mathcal{P}_{\mathbf{Z}^{p},\mathbf{Z}^{q}}}\left[\log\mathcal{Q}\left(\mathbf{Z}^{p}\mid\mathbf{Z}^{q}\right)\right]$.

The above $\mathcal{Q}$ has no strong restrictions and can be implemented in various ways, such as Gaussian distribution or through neural networks. For simplicity and directness, we use Gaussian distribution and utilize $G(\cdot)$ as a parameter function to map $\mathbf{Z}^{p}$ to $\mathbf{Z}^{q}$, i.e., $\mathcal{N}\left(\mathbf{Z}^{p}\mid G^{(q)}\left(\mathbf{Z}^{q}\right),\sigma\mathbf{I}\right)$, the $\sigma\mathbf{I}$ appearing here is a diagonal matrix. By ignoring the constants derived from the Gaussian distribution, the objective function in our data recovery can be defined as:

In order to visually observe the objective function of the data recovery part, for the data of the two views, the loss function has the following form:

Through the above loss optimization model, the missing view data recovery module can better recover complete information from another view, further reducing the impact of missing data on the model. The missing information can be recovered from such a way, i.e., $\mathbf{Z}^{2}=G^{(1)}\left(\mathbf{Z}^{1}\right)=G^{(1)}\left(f_{ec}^{1}\left(\mathbf{X}^{1}\right)\right)$ and $\mathbf{Z}^{1}=G^{(2)}\left(\mathbf{Z}^{2}\right)=G^{(2)}\left(f_{ec}^{2}\left(\mathbf{X}^{2}\right)\right)$.

In our experiments, we employ randomly generated incomplete multi-view datasets to validate our algorithm. For each dataset, we generate incomplete samples by randomly removing partial views across all samples. The missing rate $\eta$ is defined as $\eta=(n-m)/n$, where $m$ is the number of complete examples, and $n$ is the number of the whole dataset, which ranges from 0.1 to 0.7 with an interval of 0.2.

For all compared algorithms, we use default parameters or iteratively search in a given parameter set to find the best parameters for these methods and report their best clustering results. For the clustering method that can only be applied to the complete multi-view data (AE²-NET and MFLVC) adopt the same way as [24], we use the average value of the feature values of the corresponding views to fill in the missing views, and cluster on the completed data set obtained after filling. For each dataset, all methods were performed on the same randomly formed incomplete cases and their best results were reported for fair comparison. The clustering metrics (ACC, NMI and ARI) comparison is presented in Table II-VI. From these tables, we obtain the following observations:

1) From the Table II-VI, it can be observed that the clustering metrics (ACC, NMI and ARI) gradually decrease as the missing rate increases. The performance of other incomplete multi-view clustering methods is better than the complete multi-view clustering method when the missing rate is large, which indicates that the imputed views inferred by other incomplete multi-view clustering methods contain more semantic information than the average vectors and thus alleviate the risk of degraded clustering performance due to the adverse effects of noise introduced by mean value filling.

2) The clustering results of the proposed method on all data sets are obviously better than other methods, especially when dealing with large-scale data sets and high miss rate. For example, on Noisy MNIST with a missing rate of 0.7, our method exceeds the second best value by approximately 7.5%, 5.2% and 7.1% in terms of ACC, NMI and ARI, respectively. And since our method can effectively prevent the effect of dimensional collapse by using direct contrastive learning, it can more explicitly capture the useful information contained in latent features, thus greatly improving the clustering performance.

3) The clustering performance of all algorithms decreases with increasing missing rate, but the least decrease is clearly observed for our proposed method. Especially on the MNIST-USPS dataset, the performance drop of the comparison algorithm is larger than ours, and the ACC of our method only drops 3.46% in the case of missing rate 0.1-0.7. These observations proved that our algorithm in the case of a larger missing rate can still normal processing data. This reflects that our recovered views are more conducive to clustering, and our method discards inconsistent information between views so that it can better utilize views to obtain consistent information. This advantage allows our method to perform clustering tasks well even when many view data are missing.

In this section, we will analyze the ablation studies of the proposed method on the MNIST-USPS dataset. In the evaluation, the missing rate is fixed at 0.5.

Loss function component analysis. To further verify the importance of each module of our method in the overall algorithm framework, we evaluate each module of the algorithm through the following ablation studies. In detail, as shown in Table VII, the following seven experiments are designed to isolate the effect of the contrastive loss $\mathcal{L}_{\mathrm{C}}$, the reconstruction loss $\mathcal{L}_{\mathrm{Z}}$, and the prediction loss $\mathcal{L}_{\mathrm{R}}$. It can be observed from the table that the loss of each part is very important for the model, no matter which part of the loss is missing will greatly affect the performance of the algorithm.

Contrastive learning on different representations. To further illustrate the effectiveness of our direct contrastive learning model, we target and perform contrastive learning on different latent representations. As shown in the Table VIII, we can find that the model performance is poor if both the reconstruction loss and the contrastive loss are applied to $\mathbf{Z}$. The main reason for the above situation is that the dataset contains a large amount of private information, and this method cannot handle the conflict between private information and common semantics, thus leading to performance degradation. And the performance advantage of our structure can be clearly observed, which demonstrates the effectiveness of our method on dealing with incomplete multi-view clustering problems.

In this section, we verify whether our method has the ability to restore the view through visual experiments. Experiments are carried out on the Noisy MNIST and Fashion datasets. By visualizing the recovered views, we can observe the effectiveness of the algorithm more intuitively. In the experiments, the missing rate $\eta$ is fixed at 0.5.

Figure 4 demonstrates the data recovery ability of our method on the dataset Noisy MNIST and Fashion, and our method has the ability to infer missing view representations compared to existing state-of-the-art methods. As shown in the figure, we can observe that in the first three rows of data, the recovered images are very similar to the complete images in the first row, but the background of our newly acquired data is consistent with the lost view, so it can be concluded that our method can recover the important information of the missing view through the information of other views, and can discard the blurred noise in the original view.

In addition to the above visualizations, we also show t-sne visualizations of common representations learned by the model. As shown in the Figures 2 and 3, as the epoch increases, the learned representation becomes more compact and discriminative.

In this section, we investigate the convergence of our algorithm by reporting the loss value and the corresponding cluster performance over time. As shown in Figure 5, the x-axis represents training epochs, and the left and right y-axes represent clustering performance and corresponding loss values, respectively. It can be observed from the figure that the clustering effect of the model increases as the loss value decreases after several epochs, indicating that the proposed method has good convergence performance.

In the parameter sensitivity analysis, we studied the loss components in two hyper-parameters balance loss components, i.e., $\mathcal{L}_{\mathbf{Z}}+\lambda_{1}\mathcal{L}_{\mathbf{C}}+\lambda_{2}\mathcal{L}_{\mathbf{R}}$. Figure 6 shows the average value of ACC and NMI for 5 independents runs, as shown in the figure, our method is robust to the choice of $\lambda_{2}$. In addition, a good choice of $\lambda_{1}$ will significantly improve the performance of our model.