Uncertainty-Aware Multi-View Representation Learning

For real-world applications, data usually contains inevitable noise, which is one of the main challenge in representation learning. In order to model the underlying noise, we assume different observations are sampled from different Gaussian distributions, i.e., $\mathbf{x}_{i}^{(v)}\sim\mathcal{N}(\boldsymbol{\mu}_{i}^{(v)},(\sigma_{i}^{(v)})^{2})$. Accordingly, the observations are modeled as

$$\mathbf{x}_{i}^{(v)}=\boldsymbol{\mu}_{i}^{(v)}+\epsilon\sigma_{i}^{(v)},\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}),$$

(1)

where the mean variable $\boldsymbol{\mu}_{i}^{(v)}$ refers to sample identity, and $\sigma_{i}^{(v)}$ reflects the uncertainty of the observation in the $v^{th}$ view.

Based on above assumption, we target on encoding intrinsic information from multiple views into a unified representation. Considering the unified representation as latent variables, from the perspective of Bayesian, the joint distribution of latent variable $\mathbf{h}_{i}$ and multiple observations $\mathbf{x}_{i}^{(v)}$ for $v=1,...,V$ can be decomposed as prior on $\mathbf{h}_{i}$ ($p(\mathbf{h}_{i})$) and likelihood as

$$p(\mathbf{x}_{i}^{(1)},\cdots,\mathbf{x}_{i}^{(V)},\mathbf{h}_{i})=p(\mathbf{x}_{i}^{(1)},\cdots,\mathbf{x}_{i}^{(V)}|\mathbf{h}_{i})p(\mathbf{h}_{i}).$$

(2)

Since there is usually no prior knowledge on latent representation, we simply ignore the prior but focus on the likelihood. The likelihood aims to reconstruct observation of each view from the unified representation $\mathbf{h}_{i}$. The underlying assumption is that observation of each view $\mathbf{x}_{i}^{(v)}$ is conditionally independent given the latent variable $\mathbf{h}_{i}$, for which the likelihood can be factorized as

$$p(\mathbf{x}_{i}^{(1)},\cdots,\mathbf{x}_{i}^{(V)}|\mathbf{h}_{i})=p(\mathbf{x}_{i}^{(1)}|\mathbf{h}_{i})\cdots p(\mathbf{x}_{i}^{(V)}|\mathbf{h}_{i}).$$

(3)

This implies that we can use multiple neural networks to decode the latent variable into different views. Taking one neural network $f^{(v)}(\cdot)$ for example, we use latent variable $\mathbf{h}_{i}$ to reconstruct the Gaussian distribution of observation $\mathbf{x}_{i}^{(v)}$, i.e.,

$$p(\mathbf{x}_{i}^{(v)}|\mathbf{h}_{i})=\mathcal{N}(f^{(v)}(\mathbf{h}_{i}),{(\sigma^{(v)})}^{2}).$$

(4)

The parameters of neural networks are omitted for simplicity. To capture the uncertainty inherent in each observation instead of fixing it for each view, we model the variance to be variables which can vary with different samples. Then we have

$$p(\mathbf{x}_{i}^{(v)}|\mathbf{h}_{i})=\mathcal{N}(f^{(v)}(\mathbf{h}_{i}),{(g^{(v)}(\mathbf{h}_{i}))}^{2}).$$

(5)

Rather than a deterministic output, now we model each observation containing different level of noise. Specifically, the learned variance is a metric that captures the uncertainty caused by noise.

Taking observation $\mathbf{x}_{i}^{(v)}$ as reconstruction target, it leads to the following likelihood

		$\displaystyle p(\mathbf{x}_{i}^{(v)}|\mathbf{h}_{i})=\frac{1}{\sqrt{2\pi{(\sigma_{i}^{(v)})}^{2}}}\exp(-\frac{||\mathbf{x}_{i}^{(v)}-\boldsymbol{\mu}_{i}^{(v)}||^{2}}{{2(\sigma_{i}^{(v)})}^{2}})$		(6)
		$\displaystyle s.t.\quad\boldsymbol{\mu}_{i}^{(v)}=f^{(v)}(\mathbf{h}_{i}),~{}\sigma_{i}^{(v)}=g^{(v)}(\mathbf{h}_{i}).$

In practice, the log likelihood as follows is maximized

$$\ln p(\mathbf{x}_{i}^{(v)}|\mathbf{h}_{i})=-\frac{||\mathbf{x}_{i}^{(v)}-\boldsymbol{\mu}_{i}^{(v)}||^{2}}{{2(\sigma_{i}^{(v)})}^{2}}-\ln{(\sigma_{i}^{(v)})}.$$

(7)

We omit the constant term because it will not affect the optimization. Then, we aim to search a positive scalar $\sigma_{i}^{(v)}$ for the $v^{th}$ view of the $i^{th}$ sample to weigh the reconstruction loss. The magnitude of variance determines the sharpness of Gaussian distribution. The larger the variance, the higher the uncertainty for the observation. Basically, large uncertainty can always reduce the reconstruction loss, but the second term introduced in the objective acts as a regularizer which constrains the uncertainty from increasing too much and avoids a trivial solution.

The reconstruction networks are utilized to enforce $\mathbf{h}_{i}$ to contain intrinsic information of multiple views (Fig. 1(a)), which makes it easier to infer $\mathbf{x}_{i}^{(v)}$. Through this reconstruction process, the latent variable $\mathbf{h}_{i}$ is optimized along with the parameters of networks. In this way, $\mathbf{h}_{i}$ is able to reconstruct each view, and thus information from different views can be well encoded into $\mathbf{h}_{i}$. Note that the flow of information in reconstruction networks is reverse to conventional neural network learning, where the input is observation and the output is latent representation. We term the decoder-like framework (i.e., reconstruction network) as Reversal Network (R-Net).

Accordingly, the final minimization objective of our multi-view model is

		$\displaystyle\mathcal{L}=\sum_{i=1}^{N}\sum_{v=1}^{V}\left(\frac{||\mathbf{x}_{i}^{(v)}-\boldsymbol{\mu}_{i}^{(v)}||^{2}}{{2(\sigma_{i}^{(v)})}^{2}}+\ln{(\sigma_{i}^{(v)})}\right)$		(8)
		$\displaystyle s.t.\quad\boldsymbol{\mu}_{i}^{(v)}=f^{(v)}(\mathbf{h}_{i}),~{}\sigma_{i}^{(v)}=g^{(v)}(\mathbf{h}_{i}).$

The overall proposed model is termed as Dynamic Uncertainty-Aware Networks (DUA-Nets). On the one hand, DUA-Nets estimate uncertainty in multi-view data. Instead of a fixed weight for each view, the model learns input-dependent uncertainty for different samples according to their quality. On the other hand, in DUA-Nets, the latent variable $\mathbf{h}_{i}$ acts as input and aims to reconstruct the original views in a reversal manner. The uncertainty of each view indicates the possible noise inherent in the observation, and thus it can guide the reconstruction process. With the help of uncertainty, the high-quality samples (and views) can be fully exploited while the effect from the noisy samples (and views) will be alleviated. In this way, we model the noise in multi-view data and reduce its impact to obtain robust representations. The learning process is shown in Fig. 1(b).

There may be a natural question: since most existing models estimate uncertainty with the help of class labels, how can we learn the uncertainty inherent in data without supervision? First, if given noise-free data, our model is able to promisingly reconstruct each observation. In this case, the estimated uncertainty is close to zero. While with noise in the data, the reconstruction loss will increase accordingly. The principle behind this is that the real-world data distributions have natural patterns that neural networks can easily capture. The noisy signals are usually high-frequency components that are difficult to model. Thus, it is difficult for neural networks to reconstruct noisy data, which causes large reconstruction loss on these samples. The assumption is consistent with prior study (Ulyanov, Vedaldi, and Lempitsky 2018). Therefore, when the low-quality data is as input, our model tends to output a larger reconstruction loss, and the corresponding uncertainty will be larger to prevent the reconstruction loss from increasing too much. Each R-Net is able to capture the data noise in each view, and further assigns different views with corresponding weights to produce a unified representation. We will show this effect in the experiments section (Fig. 3).

We conduct experiments on six real-world multi-view datasets as follows: UCI-MF (UCI Multiple Features)²²2https://archive.ics.uci.edu/ml/datasets/Multiple+Features: This dataset consists of handwritten numerals (‘0’–‘9’) from a collection of Dutch utility maps. These digits are represented with six types of features. ORL³³3http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html: ORL face dataset contains 10 different images of each of 40 distinct subjects under different conditions. Three types of features: intensity, LBP and Gabor are used as different views. COIL20MV⁴⁴4http://www.cs.columbia.edu/CAVE/software/softlib/: There are 1440 images from 20 object categories. Three types of features that are same to ORL are used. MSRCV1 (Xu, Han, and Nie 2016): This dataset contains 30 different images for each class out of 7 classes in total. Six types of features: CENT, CMT, GIST, HOG, LBP, SIFT are extracted. CUB⁵⁵5http://www.vision.caltech.edu/visipedia/CUB-200.html: Caltech-UCSD Birds dataset contains 200 different bird categories with 11788 images and text descriptions. Features of 10 categories are extracted by GoogLeNet and Doc2Vec in Gensim⁶⁶6https://radimrehurek.com/gensim/models/doc2vec.html. Caltech101⁷⁷7http://www.vision.caltech.edu/Image_Datasets/Caltech101: This dataset contains images of 101 object categories. About 40 to 800 images per category. We use a subset of 1,474 images with 6 views.

We compare the proposed DUA-Nets with multi-view representation learning methods as follows:

• •

  DCCA: Deep Canonically Correlated Analysis (Andrew et al. 2013) extends CCA (Hotelling 1936) by applying deep neural networks to learn nonlinear projection. DCCA maximizes the correlation between learned representations of two views.

• •

  DCCAE: Deep Canonically Correlated AutoEncoders (Wang et al. 2015) employs autoencoders structure to obtain better embedding.

• •

  MDcR: Multi-view Dimensionality co-Reduction (Zhang et al. 2017) applies kernel matching constraint to enhance correlations among multiple views and combine these projected low-dimensional features together.

• •

  DMF-MVC: Deep Semi-Non-negative Matrix Factorization for Multi-View Clustering (Zhao, Ding, and Fu 2017) uses deep neural networks to conduct semi-NMF on multi-view data and seek the consistent representation.

• •

  RMSL: Reciprocal Multi-layer Subspace Learning (Li et al. 2019) uses self representation and reciprocal encoding to explore the consistency and complementary information among multiple views.

There are two parts in the proposed DUA-Nets, view-specific reconstruction network and uncertainty estimation network. We employ similar network architecture for both components. For all datasets, 3-layer fully connected network followed by ReLU activation function is used in the experiments. The latent representation $\mathbf{h}_{i}$ is randomly initialized with Gaussian distribution. Adam optimizer (Kingma and Ba 2014) is employed for optimization of all parameters. The model is implemented by PyTorch on one NVIDIA Geforce GTX TITAN Xp with GPU of 12GB memory.

We apply DUA-Nets and compared methods to learn multi-view representations, then we conduct clustering task to evaluate these learned representations. We employ k-means algorithm because it is simple and intuitive that can well reflect the structure of representations. For quantitative comparison of these methods, we use four common evaluation metrics including accuracy (ACC), normalized mutual information (NMI), rand index (RI), F-score to comprehensively evaluate different properties of clustering results. For each of these metrics, a higher value indicates a better performance. In order to reduce the impact of randomness, we run each method for 30 times. Clustering results are shown in Table 1. We report the performance of our method without uncertainty (R-Nets) as an ablation comparison. It is observed that DUA-Nets achieve better performance on most datasets. Take UCI-MF for example, R-Nets improve 17% over RMSL. DUA-Nets further outperform R-Nets, which validates that our modeling of uncertainty can capture the noise inherent in data and further promote representation learning.

We also conduct experiments on classification task based on the learned representations. KNN algorithm is used for its simplicity. We divide the learned representations into training and testing sets with different proportions, denoted as $G_{ratio}/P_{ratio}$, where $G$ is “gallery set” and $P$ is “probe set”. Accuracy (ACC) is used as evaluation metric. We run 30 times for each partition and report the mean and standard deviation value as well. Classification results are shown in Table 2. DUA-Nets achieve more promising performance than compared methods. RMSL and MDcR perform better on some cases but DUA-Nets show more stable performance on all cases.

According to the comparison between R-Nets and DUA-Nets, it seems the uncertainty is critical for the improvement. Therefore, in this part, we conduct qualitative experiments to provide some insights for the estimated uncertainty.

Ability of capturing uncertainty. We estimate data uncertainty on the modified CUB dataset. There are two views in the dataset, we add noise to half of the data in one view. Specifically, we generate $\frac{N}{2}$ noise vectors that are sampled from Gaussian distribution $\mathcal{N}(\mathbf{0},\mathbf{I})$. Then, we add these noise vectors (denoted as $\boldsymbol{\epsilon}$) multiplied with intensity $\eta$ to pollute half of the original data, i.e., $\mathbf{\tilde{x}}_{i}^{(1)}=\mathbf{x}_{i}^{(1)}+\eta\boldsymbol{\epsilon}_{i}$, for $i=1,...,\frac{N}{2}$. The Gaussian kernel density estimation (Scott 2015) of learned uncertainty is shown in Fig. 2. It can be found that when the noise intensity is small ($\eta=0.1$), the distribution curves of noisy samples and clean samples are highly overlapped. With the increasing of noise intensity, the uncertainty of noisy samples grows correspondingly. This demonstrates that the estimated uncertainty is closely related to sample quality, which validates that the proposed DUA-Nets are aware of the quality of each observation, and guide the integration of views into promising representations.

Capture uncertainty without supervision. In order to investigate the principle behind uncertainty estimation in unsupervised manner, we synthetically add Gaussian noise to different ratio of samples. Typically, we conduct view-specific uncertainty estimation on first two views of UCI-MF dataset individually. Fig. 3 shows the Gaussian kernel density estimation of estimated uncertainty. DUA-Nets are able to capture uncertainty of observations. As the number of noisy samples become larger, the uncertainty distribution of data changes very slightly. This demonstrates that although the noise may significantly pollute data, the neural network is still able to identify the underlying pattern even under large ratio of noisy data. Specifically, the noise inherent in data increases the difficulty to reconstruction thus produces corresponding larger uncertainty. Accordingly, the uncertainty estimated by each R-Net (for each view) reasonably guides the integration of multi-view data.

Uncertainty in improving model performance. In Table 1 and Table 2, we show that the DUA-Nets are superior to R-Nets without uncertainty, which means that the learned uncertainty is beneficial for the representation learning. Here we conduct experiments to further verify the effect of uncertainty. We use UCI-MF dataset (with the first two views) and CUB dataset to conduct clustering task. In the experiment, view 1 of each dataset is selected to add Gaussian noise to half of the samples. As shown in Fig. 4, with the increasing of the noise intensity, clustering performance on noisy view data decreases rapidly. However, the performance of DUA-Nets is quite stable. With the help of uncertainty, DUA-Nets are more robust to noise compared to R-Nets without uncertainty, which demonstrates that uncertainty can alleviate the influence of noisy observations.

There is no explicit hyperparameter in our model, however, the dimension of latent representation needs to be specified in advance. In the experiments, we choose different dimensions of latent representation $\mathbf{h}_{i}$ to investigate its effect. We conduct clustering task on original CUB dataset as well as noisy CUB dataset (half of view 1 polluted by Gaussian noise). The dimensions are selected from [10, 20, 50, 100, 200]. As shown in Fig. 5, the best performance is obtained when the dimension is set to 50. As the dimension decreases too much, the latent representation may not have enough capacity to encode information from all views, which leads to a clear performance decline. Too larger dimensions also produce lower performance, where high-dimensional representation tends to overfit, and may contain possible noise in the final representation. Fig. 6 demonstrates the convergence of proposed method. Typically, the optimization process is basically stable, where the loss decreases quickly and converges within a number of iterations.