Generative Partial Visual-Tactile Fused Object Clustering

Significant progresses have been made on visual-tactile fused sensing (Liu and Sun 2018) in recent years, e.g., object recognition, cross-modal matching and object clustering. For example, Liu et al. (Liu et al. 2016) develop an effective fusion strategy for weakly paired visual-tactile data based on joint sparse coding, which makes great success in household object recognition. Wang et al. (Wang et al. 2018b) predict the shape prior of an object from a single color image and then achieve accurate 3D object shape perception by actively touching the object. Yuan et al. (Yuan et al. 2017) show that there is an intrinsic connection between visual and tactile modalities through the physical properties of materials. Li et al. (Li et al. 2019) uses a conditional generative adversarial network to generate pseudo visual (or tactile) outputs based on tactile (or visual) inputs, then expanding the generated data to classification tasks. Zhang et al. (Zhang et al. 2020) first propose a visual-tactile fusion object clustering framework base on non-negative matrix factorization (NMF). However, all of the methods assume that data are well aligned and complete, which is unrealistic in practical applications. Thus, we design the GPVTF framework to address these problems for object clustering in this paper.

Partial multi-view clustering (Sun et al. 2020a; Li, Jiang, and Zhou 2014; Wang et al. 2020, 2018a), which provides a framework to solve the issue of incomplete (partial) input data, can be divided into two categories. The first category is based on traditional technique, such as NMF and kernel learning. For example, Li et al. (Li, Jiang, and Zhou 2014) propose a incomplete multi-view clustering framework by establishing a latent subspace based on NMF, where incomplete multi-view information is maximized. Shao et al. (Shao, Shi, and Philip 2013) propose a collective kernel learning method to complete missing data and then do clustering tasks. The second category utilizes generative adversarial networks (GANs) to complete the missing data, for the reason that GANs can align heterogeneous data and complete partial data (Dong et al. 2020, 2019; Yang et al. 2020; Jiang et al. 2019). For instance, Xu et al. (Xu et al. 2019) propose an adversarial incomplete multi-view clustering method, which performs missing data inference via GANs and learns the common latent subspace of multi-view data simultaneously. All the methods mentioned above are developed for homogeneous data, they ignore the huge gap between heterogeneous data (i.e., visual and tactile data).

Given the visual-tactile data $V$ and $T$, where $V$ denotes the visual data (i.e., RGB images) and $T$ denotes the tactile data. Noticing that the visual and tactile data collected from different tactile sensors lie in different data spaces. Our proposed GPVTF model consists of two partial feature extraction processes, i.e., visual feature extraction and tactile feature extraction, which learn partial visual features $X_{n}^{(1)}\in\mathbb{R}^{d_{1}\times n}$ from $V$ and tactile features $X_{n}^{(2)}\in\mathbb{R}^{d_{2}\times n}$ from $T$, where $d_{1}$ and $d_{2}$ are the feature dimensions and $n$ is the number of samples; two modality-specific encoders, ${E_{1}(\cdot)}$ and ${E_{2}(\cdot)}$; two generators, $G_{1}(\cdot)$ and $G_{2}(\cdot)$ and their corresponding discriminators, $D_{1}(\cdot)$ and $D_{2}(\cdot)$; two KL-divergence based losses, as illustrated in Figure 2. More details are provided in the following sections. Particularly, since each dataset has different feature extraction processes, the details of these processes are given in the “Experiments” section.

Encoders and Clustering Module: Modality-specific encoders $E_{1}(\cdot)$ and $E_{2}(\cdot)$ are introduced to project both partial visual and tactile features into the modality-specific subspaces, i.e., visual subspace and tactile subspace, respectively. Specifically, in the modality-specific subspaces, the learn latent subspace representations are learned via $Z_{n}^{(m)}=E_{m}(X_{n}^{(m)};{\theta}_{E_{m}})$, where $m=1$ denotes the visual modality, $m=2$ denotes the tactile modality, and ${\theta}_{E_{m}}$ denote the network parameters of the $m$-th encoder. Then the fused representations (i.e., $m=3$ ) can be gained by:

where $\alpha>0$ is the weighting coefficient that balances the ratio of tactile and visual modalities. Next, the K-means method is employed on $Z^{(m)}_{n}$ to get the initial clustering centers $\{{\mu}^{(m)}_{j}\}_{j=1}^{k}$, where $k$ is the number of clusters¹¹1Since we do clustering according to object identity, the $k$ is set to be equal with the number of types of objects in the datasets. Specifically, $k$ is set to be $53$, $119$ and $108$ for PHAC-2, GelFabric, and LMT datasets, respectively.. Inspired by (Xie, Girshick, and Farhadi 2016), we employ Student’s t-distribution to measure the similarity of latent subspace representations $Z^{(m)}_{n}$ and the clustering center ${\mu}^{(m)}_{j}$:

where $\gamma$ is the degrees of freedom of the Student’s t-distribution and set to be $1$ in this paper; $q^{(m)}_{nj}$ are the pseudo-labels, which denote the probability of assigning sample $n$ to cluster $j$ for the $m$-th modality.

To improve cluster compactness, we pay more attention to data points of which are assigned with high confidence, by obtaining the target distribution $p_{nj}^{(m)}$ as follows:

Then the encoders are trained with fused KL-divergence losses, which are defined as follows:

where $m=1$ and $m=2$ correspond to the losses of encoders $E_{1}(\cdot)$ and $E_{2}(\cdot)$, and $\beta$ is a trade-off parameter. The encoders are implemented by a two-layer fully-connected network.

Conditional Cross-Modal Clustering GANs: Noticing that the gap between visual and tactile modalities is very large since their frequency, format and receptive field are quite different. Thus, directly employing GANs in the original space $X_{n}^{(m)}$ might increase the difficulty of training or even lead to non-convergence. To address this challenge, we develop a conditional cross-modal clustering GANs, which generates one latent space conditional on the other latent space. Specifically, the conditional cross-modal cluster GANs including $G_{m}(\cdot)$ and $D_{m}(\cdot)$, where $G_{m}(\cdot)$ competes with $D_{m}(\cdot)$ to generate samples as real as possible, and the loss function is given as:

where $\omega$ is the noise matrix. Noticing that our goal is clustering rather than generation, a prior that consists of normal random variables cascaded with one-hot noise is sampled, which is different from tradition GANs. More specifically, $\omega=(\omega_{n},\omega_{c})$, $\omega_{n}\sim N(0,\sigma^{2}I_{dn})$, $\omega_{c}=e_{k}$, $e_{k}$ is the $k$-th elementary vector in $\mathbb{R}^{k}$ and $k$ is the number of clusters. We choose $\sigma=0.1$ in all our experiments. By this way, a non-smooth geometry latent subspace is created, and $G_{m}(\cdot)$ can generate more distinctive and robust representations which are beneficial for clustering performance, i.e., not only the gap between visual and tactile modalities can be mitigated but also the missing data are completed naturally.

Moreover, since training the GANs in Eq. (5) is not trivial (Wang et al. 2019), a regularizer, which forces the real samples and the generated fake samples to be similar, is introduced to obtain stable generative results, which can be defined as:

Then, the overall loss function of $G_{m}(\cdot)$ is given as follows:

where $\lambda$ is a trade-off parameter which balances the two losses and is set to be 0.1 in this paper. $G_{m}(\cdot)$ is a three-layer network.

The discriminator $D_{m}(\cdot)$ is designed to discriminate the fake representations generated by $G_{m}(\cdot)$ and the real representations in the modality-specific subspaces. The object function for $D_{m}(\cdot)$ can be given as:

The proposed $D_{m}(\cdot)$ is mainly made up of a fully connected layer with ReLU activation, a mini-batch layer (Salimans et al. 2016) that can increase the diversity of fake representations, a sigmoid function which outputs the fake-real possibility of input representations. Then, both the generated fake and real representations are fused. Thus, the fusion representations Eq. (1) can be modified to:

where $\varphi_{m}$ is the weighting coefficients of real and the generated fake representations for the $m$-th modality, i.e., $m=1$ represents visual modality and $m=2$ represents tactile modalities, respectively. The overall loss function of our model is summarized as follows:

where $\mathcal{L}_{E_{m}}$ are the KL-divergence losses, $\mathcal{L}_{G_{m}}$ and $\mathcal{L}_{D_{m}}$ are the conditional cross-modal clustering GANs losses.

The whole process of the proposed GPVTF framework is summarized as below.

Step 1 Initialization: We feed the partial visual and tactile features $X_{n}^{(1)}$ and $X_{n}^{(2)}$ into $E_{1}(\cdot)$ and $E_{2}(\cdot)$ to obtain the initial latent subspace representations $Z_{n}^{(m)}$. Then standard K-means method is applied on $Z_{n}^{(m)}$ to get the initial clustering centers $\{\mu_{j}^{(m)}\}_{j=1}^{k},\forall m=1,2,3$.

Step 2 Training encoders: Eq. (2) is employed to calculate the pseudo-labels $q_{nj}^{(m)}$; $p_{nj}^{(m)}$ and KL-divergence losses $L_{E_{m}}$ are computed by Eq. (3) and Eq. (4), respectively. Then $L_{E_{m}}$ are fed to its corresponding Adam optimizers to train the encoders and the learning rates are set to be 0.0001.

Step 3 Training conditional cross-modal clustering GANs: In this step, we employ the generator losses, i.e., Eq. (5) and Eq. (6) with Adam optimizers to update the parameters of the two generators and the learning rates are set to be 0.000003 and 0.000004 for $G_{1}(\cdot)$ and $G_{2}(\cdot)$, respectively. Next, the two discriminators $D_{1}(\cdot)$ and $D_{2}(\cdot)$ are optimized by Eq. (8) with Adam optimizers and the leaning rates are set to be 0.000001 both for $D_{1}(\cdot)$ and $D_{2}{(\cdot)}$. We update the generators five times while updating the discriminators once.

Step 4 After the framework is optimized, we feed original data to the model and then obtain the completed fusion representations $Z_{n}^{(3)}$ as well as the updated clustering centers $\{\mu_{j}^{(m)}\}_{j=1}^{k}$. Then the predicted clustering labels $q^{(3)}_{nj}$ are calculated by Eq. (2). Finally, we choose the maximum value of $q^{(3)}_{nj}$ as the predicted clustering labels. We implement the model with Tensorflow 1.12.0, and set the batch size to be 64. We summarize the overall training process of the proposed framework in Algorithm 1.

PHAC-2 (Gao et al. 2016) dataset consists of color images and tactile signals of 53 household objects, where each object has 8 color images and 10 tactile signals. We use all the images and the first 8 tactile signals to build the initial paired visual-tactile dataset in this paper. The feature extraction process of the tactile modality is similar with (Gao et al. 2016; Zhang et al. 2020), and the visual features are extracted by AlexNet (Krizhevsky, Sutskever, and Hinton 2012), which is pre-trained on the ImageNet. After feature extraction, 4096-D visual and 2048-D tactile features are obtained. LMT (Zheng et al. 2016; Strese, Schuwerk, and Steinbach 2015) dataset consists of 10 color images and 30 haptic acceleration data of 108 different surface materials. The first 10 haptic acceleration data and all the images are used. We extract 1024-D tactile features similarly with (Liu, Sun, and Fang 2019) and 4096-D visual features by the pre-trained AlexNet. GelFabric (Yuan et al. 2017) dataset includes visual data (i.e., color and depth images) and tactile data of 119 kind of different fabrics. Each fabric has 10 color images and 10 tactile images, which are used in this paper. Since both the visual and tactile data are image formats, we extract 4096-D visual and tactile features with pre-trained AlexNet. Some examples of the used datasets are given in Figure 3.

Partial data generation: The partial visual-tactile datasets are generated in a similar way with partial multi-view clustering settings, e.g., Xu et al (Xu et al. 2019). Supposing that the number of all the visual and tactile samples is $N$ in each dataset, we randomly select $\tilde{N}$ samples as the missing data points. Then, the Missing Rate (i.e., $\mathcal{MR}$) can be defined as $\mathcal{MR}=\frac{\tilde{N}}{N}$.

We compare our GPVTF model with the following baseline methods. We first employ standard spectral clustering methods on the modality-specific features, i.e., visual features $X_{n}^{(1)}$ and tactile features $X_{n}^{(2)}$, which are termed as SC1 and SC2. ConcatPCA concatenates feature vectors of different modalities via PCA and then performs standard spectral clustering. GLMSC (Zhang et al. 2018) proposes a subspace multi-view clustering model under the assumption that each single feature view originates from one comprehensive latent representations. VTFC (Zhang et al. 2020) is a pioneering work to incorporate visual modality with tactile modality in the object clustering tasks based on auto-encoders and NMF. IMG (Zhao, Liu, and Fu 2016) does the incomplete multi-view clustering by transforming the original partial data to complete representations. GRMF (Wen et al. 2018) exploits the complementary and local information among all views and samples based on graph regularized matrix factorization. UEAF (Wen et al. 2019) performs missing data inference with locality-preserved constraint.

Evaluation Metrics: Two widely used clustering evaluation metrics, i.e., Accuracy (ACC) and Normalized Mutual Information (NMI) are employed to assess the effectiveness of the clustering performance. For all the metrics, higher value indicates better performance. More details of these metrics can be found in (Schütze, Manning, and Raghavan 2008).

Experimental results on three public visual-tactile datasets are reported by comparing with the state-of-the-arts in this subsection. Due to the randomness of missing data generation, all experiments are repeated in ten times and reported with the mean value. Generally, the observations are summarized as follows: 1) As shown in Table 1, where the missing rate is set to be 0.1, our GPVTF model consistently outperforms other methods with a clear improvement.

For instance, compared with single-modality methods (i.e., SC1 and SC2), the performance is raised by $12.68\%$ in ACC and $7.42\%$ in NMI on the PHAC-2 dataset, which demonstrates the fact that fusing visual and tactile modalities does improve the clustering performance. The results also show that our model is able to learn complementary information among the heterogeneous data. Compared with partial multi-view clustering method UEAF and visual-tactile fusing clustering method VTFC, the performance is raised by $1.89\%$ and $3.62\%$ in ACC and NMI, respectively. The reason why our GPVTF model achieves considerable achievements is that our model can not only complete the missing data but also well align the heterogeneous data. 2) As shown in Figure 3 and Figure 4, our GPVTF model outperforms other methods under different missing rates ($0.1\sim 0.5$) on all the three datasets. Moreover, our model can also achieve competitive results on the PHAC-2 and LMT datasets even though the missing rate is very large. This observation indicates the effectiveness of the proposed conditional cross-modal clustering GANs. Besides, although the performance of SC2 drops more slowly than ours, its performance is very low in most cases. We also find an interesting phenomenon that some multi-view clustering methods (i.e., GRMF, IMG and GLMSC) even perform worse than single-view methods. The possible reason is that these methods do not take the gap between visual and tactile data into account. Directly fusion the heterogeneous data in a violent way would inevitably lead to performance degradation.

The effect of the proposed cross-modal clustering GANs, fusion KL-divergence losses are analyzed first. Then we report the analyses of most important parameters $\alpha$, $\beta$, $\varphi_{1}$ and $\varphi_{2}$.

Effectiveness of Cross-Modal Clustering GANs, Fusion KL-Divergence Losses: As shown in Figure 6, we first conduct ablation study to illustrate the effect of the proposed conditional cross-modal clustering GANs and fusion KL-Divergence losses when missing rate is set to be 0.1, where “None GANs” means the proposed conditional cross-modal clustering GANs are not employed and “None Fusion KL” means the proposed fusion KL-Divergence losses are not employed, respectively. We can observe that “Ours” outperforms “None GANs” among all the datasets, which proves that the proposed conditional cross-modal clustering GANs promotes to achieve better performance. “Ours” outperforms than “None Fusion KL” proves that the proposed fusion KL-Divergence losses could better discover the information hidden in multi-modality data, and further enhance the performance.

Parameter Analysis: To explore the effect important weight coefficient $\alpha$ that controls the proportion of visual and tactile modalities, the parameter $\alpha$ is tuned from the set {$0.1,0.2,0.3,0.4,0.5$}, and report the clustering performance in Figure 7. Our model achieves the best clustering results, when the value of $\alpha$ is set to be 0.2, 0.2 and 0.1 on the PHAC-2, GelFabric and LMT datasets, respectively. Then, the parameter $\beta$ is tuned from the set {$0.01,0.1,1,10,100$}, and the ACC performance is plotted in Figure 7. In fact, $\beta$ controls the effect of common component, which further helps to update the encoders $E_{1}(\cdot)$ and $E_{2}(\cdot)$ simultaneously. It helps to ease the gap between visual and tactile modalities. It can be seen that when $\beta$ is set to be $1$, we gain the best performance. Thus we empirically choose $\beta=1$ as default in this paper. To the end, we tune the trade-off parameters $\varphi_{1}$ and $\varphi_{2}$ in a similar way with $\beta$. As shown in Figure 8, our proposed GPVTF model performs best when $\varphi_{1}$ and $\varphi_{2}$ are set to be $0.01$. Thus, we empirically choose $\beta=1$, $\varphi_{1}=0.01$ and $\varphi_{2}=0.01$ as default in this paper in order to achieve the best performance.