MetaViewer: Towards A Unified Multi-View Representation

Multi-view representation learning is not a new topic and has been widely used in downstream tasks such as retrieval, classification, and clustering. This work focuses on multi-view representation in unsupervised deep learning scope, and related works can be summarized into two main categories [51]. One is the deep extension of traditional methods, where representative ones include deep canonical correlation analysis (DCCA) [1] and its variants [44, 40, 54]. DCCA intends to discover the nonlinear mapping for two views to a common space in which their canonical correlations are maximally preserved. These methods benefit from a sound theoretical foundation, but also usually have strict restrictions on the number and form of views.

Another alternative is the multi-view deep network. Early deep-based approaches attempted to build different architectures for handling multi-view data, such as CNN-based [12, 53, 38] and GAN-based model [49, 19], etc. Some recent approaches focus on better parameter constraints using mutual information [2, 8], comparative information [28, 52], etc. Most of these existing methods follow a specific-to-uniform pipeline. In contrast, the underlying assumption of our MetaViewer learns from uniform to specific. The most related work is the MFLVC [48], which separates view-private information from latent features at the parameter level. The essential difference is that we model the view-private information in inner-level optimization, allowing outer-level observes the modeling process and future meta learners the optimal fusion scheme.

Optimization-based meta-learning is a classic application of bi-level optimization designed to learn task-level knowledge to quickly handle new tasks [20, 22]. A typical work, MAML [13], learns a set of initialization parameters to solve different tasks with few steps of updates. Similar meta paradigm has been used to learn other manually designed parts, such as the network structure [29], optimizer[45, 59], and even sample weight [36, 32]. Similarly, we try to meta-learn fusion of multi-view feature for a unified representation. There also exist some works that consider both meta-learning paradigms and multi-view data[42, 15, 31]. However, they are dedicated to exploit the rich information contained in multiple views to improve the performance of the meta-learner on few-shot tasks or self-supervised scenario. Instead, we train a meta-learner to derive the high-quality shared representations from multi-view data via the bi-level optimization. To the best of our knowledge this is the first work to learn multi-view representation with meta-learning paradigm.

Embedding module aims to transform heterogeneous view into the latent feature space, where transformed view embeddings have the same dimension as each other. To this end, we conduct a view-specific embedding function $f_{v}$ for each view, where $v=1,2,\dots,V$. Given the $v$-$th$ view data $x^{v}$ of the entity $x$, the corresponding embedding $z^{v}$ can be computed by

where $z^{v}\in\mathbb{R}^{d}$ and $f_{v}$ typically instantiated as a multi-layer neural network with learnable parameters $\phi_{f_{v}}$.

Representation learning module maps the obtained embedding to the view representation, consisting of view-specific base-learners ${\{b_{v}\}_{v=1}^{V}}$ and a view-shared meta-learner $m$ (i.e., MetaViewer). The former learns representation for each view embedding, while the latter takes all embeddings as input and outputs the unified representation $H$ that is ultimately used for downstream tasks. Meanwhile, base-learners are generally required to be initialized from parameters of the meta-learner to learning view-shared meta representation (see 3.2), thus two types learners should be structurally shared rather than individually designed. To meet the above two requirements simultaneously, MetaViewer is implemented as a channel-oriented $1$-$d$ convolutional layer (C-Conv) with a non-linear function (e.g., ReLU[17]), as shown in Fig. 2 (c).

On the one hand, as a meta-learner, we first concatenate embeddings at the channel level, i.e., the number of the channel in the concatenated feature is equal to the number of views, and then train the MetaViewer to learn the fusion of cross-view information $H\in\mathbb{R}^{d_{h}}$ by

where $z^{cat}\in\mathbb{R}^{d\times V}$ indicates the concatenated embedding and $\omega$ is the parameter of the MetaViewer. On the other hand, base-learners could be initialized and trained for learning $v$-$th$ representation $h_{base}^{v}$ via

where $h_{base}^{v}\in\mathbb{R}^{d_{h}}$ and $\theta_{b_{v}}(\omega_{sub})$, or $\theta_{b_{v}}(\omega)$ for short, means the base-learner’s parameter $\theta_{b_{v}}$ is initialized from the channel(view)-related part $\omega_{sub}$ of the MetaViewer’s parameters. Note that this sub-network mechanism also provides a convenient way to handle incomplete views.

Self-supervised module conducts pre-text tasks to provide effectively supervised objects for model training, represented by different heads. Typically, the reconstruction head $r$ achieves the reconstruction object by re-mapping the representation back to the original view space, i.e.,

where $x_{rec}^{v}\in\mathbb{R}^{d_{x}}$ is the reconstruction result and $r_{v}$ is the reconstruction function with learnable parameters $\phi_{r_{v}}$. Expect that, we can also conduct contrastive or correlation head for the meta-learner to mine the associations across views. Similar self-supervised objectives [40, 54, 28, 52] have been extensively studied in multi-view learning, and this is not the focus of this work.

Now, we have conducted the entire structure, which can be end-to-end trained to derive the unified multi-view representation, even for incomplete views, in the specific-to-uniform manner like most existing approaches. However the data-driven fusion and view-private redundant information still cannot be handling well. So we turn to a opposite uniform-to-specific way using a bi-level optimization process, inspired by the meta-learning paradigm. Inner-level focus on the training of view-specific modules for corresponding views, and outer-level updates meta-learner to find the optimal fusion rule through observing the learning over all views. Before the detailed description, we introduce a split way of multi-view data in meta-learning style.

Meta-split of multi-view data. Consider a batch multi-view samples $\{\mathcal{D}_{batch}^{v}\}_{v=1}^{V}$ from $\mathcal{D}$. For bi-level updating, we randomly and proportionally divide it into two disjoint subsets, marked as support set $S$ and query set $Q$, respectively. As shown in 2 (a), support set is used in inner-level for leaning view-specific information, thus the sample attributes in it could be ignored. In contrast, query set retains both view and sample attributes for meta-learner training in outer-level optimization. This meta-split that decoupling view from samples can be naturally transferred to the data with incomplete views $\mathcal{D}_{inc}=\{\mathcal{D}_{c},\{\mathcal{D}_{u}^{v}\}_{v=1}^{V}\}$, where subset with incomplete views $\{\mathcal{D}_{u}^{v}\}_{v=1}^{V}$ is used as support set, and the complete part $\mathcal{D}_{c}$ is left as query set.

Inner-level optimization. Without loss of generality, take the inner-level update after the $o$-$th$ outer-level optimization as an example. Let $\omega^{o}$ be the lasted parameters of the MetaViewer, and $\phi_{v}^{o}=\{\phi_{f_{v}}^{o},\phi_{r_{v}}^{o}\}$ denotes the lasted parameters in embedding and self-supervised modules for brevity. We first initial base-learner from meta-learner, i.e., $\theta_{b_{v}}^{0}=\omega^{o}$, and make a copy of $\phi_{v}^{o}$ for $\tilde{\phi}_{v}$. Note that the copy means gradients with respect to the $\phi_{v}^{o}$ will not be back-propagated to $\tilde{\phi}_{v}$ and vice versa. Thus, $\theta_{b_{v}}^{0}$ and $\tilde{\phi}_{v}$ form the initial state for the inner-level optimization. Suppose the $\mathcal{L}_{inner}^{v}$ is the loss function of the inner-level with respect to the $v$-$th$ view, and then the corresponding update goals are

Consider a gradient descent strategy (e.g., SGD [4]), we can further write the update process of $\theta_{b_{v}}$:

where $\beta$ and $i$ denote the learning rate and iterative step of inner-level optimization, respectively.

Outer-level optimization. After several inner-level updates, we obtain a set of optimal view-specific parameters on the support set. Outer-level then updates the meta-learner, embedding, and head modules by training on the query set. With the loss function $\mathcal{L}_{outer}$, the outer-level optimization goal is

By alternately optimizing Eq. 6 and Eq. 8, we end up with the optimal meta-parameters $\omega^{*}$ and a set of view-specific parameters $\phi_{v}^{\ast}$. For a test sample $x_{test}$, its corresponding representation is derived by sequentially feeding the embedding function and the meta-learner. The overall framework of our MetaViewer is shown in Alg. 1.

We discuss the difference between specific-to-uniform and uniform-to-specific paradigm through the update of fusion parameters $\omega$ with a reconstruction loss $\mathcal{L}_{rec}^{v}$. Using the same structure described in Sec. 3.1, the specific-to-uniform generally optimizes $\omega$ by minimizing reconstruction losses over all views, i.e., $\omega^{\ast}=\mathop{\arg\min}\sum_{v=1}^{V}\mathcal{L}_{rec}^{v}(r_{v}(m(z^{v},\omega),\phi_{r_{v}})),x_{v})$, and the $\omega$ is updated by (with the SGD)

The optimal $\omega^{\ast}$ observes all views and derives the unified representation $H$, that is, from the particular to the general. While the update of $\omega$ in our uniform-to-specific can be written

Note that $\theta^{\ast}_{b_{v}}(\omega)$ contain the optimization process of each view in inner-level as Eq. (6), which means that the optimal $\omega^{\ast}$ update by observing the reconstruction from unified representation to specific views. Fig. 3 intuitively demonstrates the difference between these two manner.

Our uniform-to-specific framework emphasizes learning from reconstruction in inner-level, thus the $\mathcal{L}_{inner}^{v}$ is specified as the reconstruction loss [56, 28]

While parameters updated in outer-level can be constrained by richer self-supervised feedbacks as mentained in Sec. 3.1. Here we provide two instances of outer-level loss function to demonstrate how MetaViewer can be extended with different learning objectives.

MVer-R adopts the same reconstruction loss as the inner-level and $\mathcal{L}_{outer}=\sum_{v}\mathcal{L}_{rec}^{v}(Q^{v},Q_{rec}^{v})$, which is the purest implementation of MetaViewer.

MVer-C additionally utilizes a contrastive objective, where the similarities of views belonged to same entity (i.e., positive pairs) should be maximized and that of different entities (i.e., negative pairs) should be minimized, i.e., $\mathcal{L}_{outer}=\sum_{v}\left(\mathcal{L}_{rec}^{v}+\sum_{v^{\prime},v^{\prime}\neq v}\mathcal{L}_{con}^{v,v^{\prime}}\right)$. Following previous work [48, 19, 52], the contrastive loss $\mathcal{L}_{con}^{v,v^{\prime}}$ is formed as

where $q_{i}^{v}$ is the $v$-$th$ views of the $i$-$th$ query sample, and $d$ is the similarity metric (e.q., cosine similarity [6]). The $N_{Q}$ and $\tau$ denote the number of query set samples and the temperature parameter, respectively. Note that, the derived meta representation can be also used in contrastive learning as a additional noevl view.

Datasets. To comprehensively evaluate the effectiveness of our MetaViewer, we conduct six multi-view benchmarks in experiments. All datasets are scaled to $[0,1]$ and split into training, validation, and test sets in the ratio of $6:2:2$, as shown in Tab. 1. More details of dataset are in Appendix A.

• •

  BDGP is an image and text dataset, corresponding to $2,500$ drosophila embryo images in $5$ categories. Each image is described by a $79$-D textual feature vector and a $1,750$-D visual feature vector [5].

• •

  Handwritten contains $2,000$ handwritten digital images from $0$ to $9$. Two types of descriptors, i.e., $240$-D pixel average in $2\times 3$ windows and $216$-D profile correlations, are selected as two views [56].

• •

  RGB-D dataset contains visual and depth images of $300$ distinct objects across $50$ categories [60, 43]. Two views are obtained by flattening the $64\times 64\times 3$ color images and $64\times 64$ depth images.

• •

  Fashion-MV is an image dataset that contains $10$ categories with a total of $30,000$ fashion products. It has three views and each of which consists of 10,000 gray images sampled from the same category [47].

• •

  MSRA [46] consists of $210$ scene recognition images from seven classes with six views, that is, CENTRIST, CMT, GIST, HOG, LBP, and SIFT.

• •

  Caltech101-20 is a subset of the Caltech101 image set [9], which consists of $2,386$ images of $20$ subjects. Six features are used, including Gabor, Wavelet Moments, CENTRIST, HOG, GIST, and LBP.

Compared methods. We compare the performance of MetaViewer with five representative multi-view learning methods, including two classical methods (DCCA [1] and DCCAE [44]) and three state-of-the-art methods (MIB [8], MFLVC [48] and DCP [28]). Among them, DCCA and DCCAE are the deep extensions of traditional correlation strategies. MIB is a typical generative method with mutual information constraints. DCP learns unified representation both in complete and incomplete views. In particular, MFLVC also notices the view-private redundant information and designs a multi-level features network for clustering tasks.

Implementation details. For a fair comparison, all methods are trained from scratch and share the same backbone listed in Appendix B. We concatenate latent features of all views in compared methods to obtain the unified representation $H$ with the same dimension $d_{h}={256}$, and verify their performance in clustering and classification tasks using K-means and a linear SVM, respectively. For MetaViewer, we train $2,000$ epochs for all benchmarks, and set the batch size is $32$ for RGBD and MSRA and $256$ for others. The learning rates in outer- and inner-level are set to $10^{-3}$ and $10^{-2}$, respectively. All experiments have been verified using the PyTorch library on a single RTX3090.

Clustering results. Tab. 2 lists the results of the clustering task, where the performance is measured by three standard evaluation metrics, i.e., Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI). A higher value of these metrics indicates a better clustering performance. It can be observed that (1) our MVer-C variant significantly outperforms other compared methods on all benchmarks except MSRA; (2) the second-best results appear between MVer-R and MFLVC, both of which explicitly separates the view-private information; (3) A larger number of categories and views is the main reason for the degradation of clustering performance, and our Metaviewer improves most significantly in such scenario (e.g. Fashion-MV and Caltech101-20).

Classification results. Tab. 3 lists the results of the classification task, where three common metrics are used including Accuracy, Precision, and F-score. A higher value indicates a better classification performance. Similar to the clustering results, two variants of MetaViewer significantly outperform the comparison methods. It is worth noting that (1) DCP learns the unified generic representation and therefore achieves the second-best result instead of MFLVC. (2) The number of categories is the main factor affecting the classification performance, and our method obtains the most significant improvement in the RGB-D dataset with $50$ classes. More results including incomplete views are deferred to Appendix C.

As mentioned in 3.3, MetaViewer is essentially learned to learn an optimal fusion function that filters the view-private information. To verify this, we compare it with commonly used fusion strategies [27], including sum, maxima, concatenation, linear layer and C-Conv. The former three are the specified fusion rules without trainable parameters and the remaining two are the trainable fusion layer trained via the specific-to-uniform manner. Tab. 4 lists the clustering results and an additional MSE score on the Handwritten dataset with the same embedding and reconstruction network (see Tab. 1). We can observe that (1) trainable fusion layers outperform the hand-designed rules, and our MetaViewer yields the best performance; (2) the MSE scores listed in the last column indicate that the quality of the unified representation cannot be measured and guaranteed only with the reconstruction constraint, due to the view-private redundant information mixed in view-specific latent features.

Meta-learner structures. We implement the meta-learner as channel-level convolution layers in this work. Albeit simple, this layer can be considered as a universal approximator for almost any continuous function [36], and thus can fit a wide range of conventional fusion functions. To investigate the effect of network depth, width, and convolution kernel size on the performance of the representation, we alternate fix the $32$ kernels and $1\times 3$ kernel size and show the classification results on Handwritten data in Fig. 4. It is clear that (1) the meta-learner works well with just a shallow structure as shown in Fig. 4 (a), instead of gradually overfitting to the training data as the network deepens or widens, (2) our MetaViewer is stable and insensitive to the hyper-parameters within reasonable ranges.

Meta-split ratios. Fig. 5 (a) shows the impact of meta-split mentioned in 3.2 on the classification performance, where the proportion of support set is set from $0.1$ to $0.9$ in steps of $0.1$, and the rest is query set. In addition to the single view, we also compare the sum and concat. fusion as baselines. MetaViewer consistently surpasses all baselines over the experimental proportion. In addition, fusion baselines are more dependent on the better-performing view at lower proportions, instead becoming unstable as the available query sample decreases.

Inner-level update steps. Another hyper-parameter is the number of iteration steps in inner-level optimization. More iterations mean a larger gap from the learned meta representation to the specific view space, i.e., coarser modeling of view-private information. Fig. 5 (b) shows the classification results with various steps, where $n$ step means that the inner-level optimization is updated $n$ times throughout the training. MetaViewer achieves the best results when using $1$ steps, and remains stable within $15$ steps.