Partially Shared Semi-supervised Deep Matrix Factorization with Multi-view Data

In practical applications, real-world data can be described from different views, that is the so called multi-view data. For instance, an image can be described by several characteristics, e.g., shape, color, texture and so on. Because multi-view representation learning can exploit the implicit-dependent structure of multiple views and improve the performance of the learning tasks. In recent years, multi-view representation learning has attracted increasing research attention in machine learning [1, 2, 3].

Over the past decade, lots of researches on multi-view learning have emerged. In particular, Non-negative Matrix Factorization (NMF) [4, 5] as one of the most popular high-dimensional data processing algorithms, have been widely used for clustering multi-view data [7, 6]. NMF-based multi-view clustering methods have shown to generate superior clustering results which are easy to interpret [8, 9]. Considering the relations between view-specific (uncorrelated) and common information (correlated), partially shared NMF-based multi-view learning approaches are proposed [10, 11].

Nevertheless, most existing methods are single layer structures, which are hard to extract hierarchical structural information [12] of multi-view data. With the development of deep learning, Wang et al. proposed DCCA to extract the hidden hierarchical information in 2-view data [13]. Zhao et al. [14] extended single-view deep matrix factorization [15] into multi-view and proposed graph regularized deep multi-view clustering, which can eliminate the interference in multi-view data. Huang et al. [16] designed a novel robust deep multi-view clustering model to learn the hierarchical semantics without hyperparameters. For clarity, the general deep matrix factorization framework for multi-view data as illustrated in Fig. 1. However, most existing deep matrix factorization multi-view methods are merely considering the common information and ignoring the effect of view-specific information in each individual view. Besides, they are formulated as an unsupervised learning problem and inapplicable when partially labeled data is available. Actually, researchers have found that integrating such label information can produce a considerable improvement in learning performance [19, 18, 17].

To this end, we propose a novel deep multi-view clustering methods called Partially Shared Semi-supervised Deep Matrix Factorization (PSDMF). In our method, correlated and uncorrelated features of multi-view data are both considered by partially shared approaches, in which the latent representation of each view is divided into the common part and the view-specific part. A robust sparse regression term with $L_{2,1}$ norm is adopted to integrate the partial label information of the labeled data. Besides, to respect the intrinsic geometric relationship and avoid the parameters problem in different views, we apply graph regularization and auto-weighted strategy to PSDMF. An efficient iterative updating algorithm with pre-training scheme is designed for PSDMF. We summarize our major contributions as follows:

• •

  This paper proposes an semi-supervised deep multi-view clustering model to improve the performance of traditional unsupervised deep matrix factorization methods by using label regression learning approach.

• •

  To respect the common information and view-specific features of multi-view data, we propose a partially shared deep matrix factorization method to jointly exploit two kinds of information and learn a comprehensive final layer representation of multi-view data.

• •

  Local invariant graph regularization and the auto-weighted strategy are introduced to preserve the intrinsic geometric structure in each view and further boost the quality of output representation.

The rest of this paper is organized as follows. In Section II, we give a brief review of related works and describe the proposed PSDMF. Section III present an efficient algorithm to solve the optimization problem. In Section IV, we report the experimental results on five real-world datasets. Finally, Section V concludes this paper. Table. I summarizes the general notations in this article for reader’s convenience.

To expedite the approximation of the matrices in our proposed model, we conduct the pre-training [24] by decomposing each view $\mathbf{X}^{p}\approx\mathbf{U}_{1}^{p}\mathbf{V}_{1}^{p}$ by minimizing the Semi-NMF $\left\|\mathbf{X}^{p}-\mathbf{U}_{1}^{p}\mathbf{V}_{1}^{p}\right\|_{F}^{2}$, where $\mathbf{U}_{1}^{p}\in\mathbb{R}^{M^{p}\times k_{1}}$ and $\mathbf{V}_{1}^{p}\in\mathbb{R}^{k_{1}\times N}$. Then the factor matrix $\mathbf{V}_{1}^{p}$ is further decomposed as $\mathbf{V}_{1}^{p}\approx\mathbf{U}_{2}^{p}\mathbf{V}_{2}^{p}$ by minimizing the $\left\|\mathbf{V}_{2}^{p}-\mathbf{U}_{2}^{p}\mathbf{V}_{2}^{p}\right\|_{F}^{2}$, where $\mathbf{U}_{2}^{p}\in\mathbb{R}^{k_{1}\times k_{2}}$, $\mathbf{V}_{1}^{p}\in\mathbb{R}^{k_{2}\times N}$ and $k_{1},k_{2}$ denote the dimensions for layer 1 and layer 2. The process will be repeated till all of the layers are pre-trained. The optimization of Semi-NMF can be derived following a similar process as described in [20]. To save the space, we omit the updating rules here.

Update rule for hidden matrix $\mathbf{U}_{i}^{p}$: By fixing $\mathbf{V}_{m}^{p}$ and $\mathbf{W}$, we minimize the objective function (8) with respect to $\mathbf{U}_{i}^{p}$. Letting the partial derivative $\partial O/\mathbf{U}_{i}^{p}=0$, we can obtain

where $\Phi_{i-1}=\mathbf{U}_{1}^{p}\mathbf{U}_{2}^{p}\ldots\mathbf{U}_{i-1}^{p}$ and $\mathbf{U}_{im}=\mathbf{U}_{i+1}^{p}\ldots\mathbf{U}_{m}^{p}\mathbf{V}_{m}^{p}$.

Update rule for regression weight matrix $\mathbf{W}$: Following [27], the derivative of objective function (8) with respect to $\mathbf{W}$ is as follows:

where $\mathbf{E}$ is a diagonal matrix with $e_{ii}=1/2\|\mathbf{w}_{i}\|_{2}$. According to the optimization theory, set $\partial O/\mathbf{W}=0$, then we can obtain the update rule for $\mathbf{W}$ is

Update rule for partially shared factor matrix $\mathbf{V}_{m}^{p}$: As illustrated in Fig. 2, $\mathbf{V}_{m}^{p}$ can be divided into four parts: $\mathbf{V}_{sl}^{p},\mathbf{V}_{su}^{p},\mathbf{V}_{cl}$ and $\mathbf{V}_{cu}$. Similarly, each data matrix $\mathbf{X}^{p}$ is divided into two parts $\mathbf{X}_{l}^{p}$ and $\mathbf{X}_{u}^{p}$; each Laplacian Graph matrix $\mathbf{L}^{p}$ is divided into two parts $\mathbf{L}_{l}^{p}$ and $\mathbf{L}_{u}^{p}$.

For the constraint $\mathbf{V}_{m}^{p}\geq 0$, we introduce the Lagrangian multiplier $\eta$ as follows:

where $\Phi_{m}=\mathbf{U}_{1}^{p}\mathbf{U}_{2}^{p}\ldots\mathbf{U}_{m}^{p}$. Accordingly, $\Phi_{m}$ is divided into two parts $\Phi_{ms}$ and $\Phi_{mc}$. For the convenience of writing, we denote that $\mathbf{A}^{p}=[\Phi_{ms},\Phi_{mc}][\mathbf{V}_{sl}^{p};\mathbf{V}_{cl}]$, $\mathbf{B}^{p}=[\Phi_{ms},\Phi_{mc}][\mathbf{V}_{su}^{p};\mathbf{V}_{cu}]$.

The zero gradient condition of $\mathcal{L}\left(\mathbf{V}_{m}^{p}\right)$ with respect to $\mathbf{V}_{sl}^{p},\mathbf{V}_{su}^{p},\mathbf{V}_{cl}$ and $\mathbf{V}_{cu}$, respectively, we have

where $\mathbf{F}=\mathbf{W}(\mathbf{W}^{T}\mathbf{V}_{l}-\mathbf{Y})=[\mathbf{F}_{s}^{1};...;\mathbf{F}_{s}^{p};\mathbf{F}_{c}]$. Following a similar proof to [14], using the Karush-Kuhn-Tucker condition, we can formulate the updating rules for $\mathbf{V}_{m}^{p}$:

where $[\mathbf{H}]^{+}$ and $[\mathbf{H}]^{-}$ denote a matrix that all the negative elements are replaced by $0$ and all the positive elements are replaced by $0$, respectively. That is,

Until now, we have all the update rules done. Updates iterations repeatedly until convergence. The overall optimization process of PSDMF is outlined in Algorithm 1, where the ”Semi-NMF” procedure performs the pre-train as described earlier. Once the optimal partially shared latent representation matrix $\mathbf{V}$ and regression weight matrix $\mathbf{W}$ are obtained, cluster label $y=\arg\max_{c}y_{c,i}$ where $\boldsymbol{y}_{i}=\mathbf{W}^{T}\boldsymbol{v}_{i}$.

To comparatively study the performance of PSDMF, we consider several state-of-the-art baselines as stated bellow:
Multi-view clustering via deep matrix factorization (DMVC) [14], graph-based multi-view clustering (GMC) [21], latent multi-view subspace clustering (lLSMC) [23], large-scale multi-view subspace clustering (LMVSC) [22], multi-view discriminative learning via joint non-negative matrix factorization (DICS) [18], graph-regularized multi-view semantic subspace learning (MvSL) [19], partially shared latent factor learning (PSLF) [10] and semi-supervised multi-view clustering with graph-regularized partially shared non-negative matrix factorization (GPSNMF) [17]. Among them, four methods are unsupervised (i.e., DMVC, GMC, lLSMC, LMVSC) and the others are semi-supervised.

In this paper, we introduced a novel semi-supervised deep matrix factorization model for multi-view learning, called PSDMF, which is able to learn a comprehensive partially shared latent final-layer representation for multi-view data. Through the partially shared multi-layer matrix factorization structure, our method is capable of exploiting view-specific and common features among different views, simultaneously. Benefitting from the label regression term, it can incorporate information from the labeled data. Furthermore, we utilize graph regularization and auto-weighted strategy to preserve the intrinsic geometric structure of the data. An iterative optimization algorithm is developed to deal with the PSDMF. Experimental results show that the proposed model achieves superior performances.

This work was supported by the National Natural Science Foundation of China under Grant Nos. 61722304, 61801133, and 61803096, in part by the Guangdong Science and Technology Foundation under Grant Nos. 2019B010118001, 2019B010121001, and 2019B010154002, National Key Research and Development Project, China under Grant No. 2018YFB1802400.