Clustering-friendly Representation Learning for Enhancing Salient Features

As Fig. 2 shows, we use two embedding functions, $f_{\theta}$ and $g_{\psi}$, which map images to feature vectors distributed on a $d$-dimensional unit sphere. These functions are modeled as deep neural networks with parameters $\theta$ and $\psi$, which are typically a CNN backbone and an L2-normalization layer. $f_{\theta}$ is learned from background dataset $\mathcal{B}$ and assigned the role of extracting background features $W=\{w_{i}\}_{i=1}^{N}$ and $Z=\{z_{i}\}_{i=1}^{M}$ for image samples from $\mathcal{X}$ and $\mathcal{B}$, respectively. The other embedding function $g_{\psi}$ is learned to extract target features $V=\{v_{i}\}_{i=1}^{N}$ from $\mathcal{X}$. To train the background branch $f_{\theta}$, we conduct minimizing instance discrimination and feature decorrelation loss, following the same process as in a previous work [18]. For target branch $g_{\psi}$, parameters are optimized by a newly defined loss function, contrastive instance discrimination loss, and feature decorrelation loss. In the computation of contrastive instance discrimination loss, weighted similarity between negative samples, which includes influences from $W$ and $V$, is used as input to a non-parametric softmax classifier.

We apply the instance discrimination proposed in [21] to learn representations from a background dataset. For given samples $\{b_{i}\}_{i=1}^{M}$, the corresponding representations are $\{z_{i}\}_{i=1}^{M}$ with $z_{i}=f_{\mathbf{\theta}}(b_{i})$, where $z_{i}$ is normalized to $\|z_{i}\|=1$. The probability of representation $z$ being assigned to the $i$th class is given by the non-parametric softmax formulation

where the dot product $z^{T}_{i}z$ is how well $z$ matches the $i$th class and $\tau_{b}$ is a temperature parameter that determines the concentration of distribution. The learning objective is to maximize the joint probability $\prod_{i=1}^{M}P\left(i|f_{\mathbf{\theta}}(b_{i})\right)$ as

We use a constraint for orthogonal features proposed in [18]. The objective is to minimize

where $F=\{f_{l}\}_{l=1}^{d}$ are latent feature vectors defined by the transpose of the latent vectors $Z$, $d$ is the dimensionality of the representations, and $\tau_{2}$ is the temperature parameter.

When normal instance discrimination loss is used, the separation of negative pairs is performed equally for all negative pairs. However, this makes grouping impossible, except in cases where both the target and background features are extremely similar. Fig. 3 shows a conceptual illustration of such a situation. Our motivation is to design a loss function that separates the pairs having the same background but different target features while attracting pairs having the same target features, independent of the background. For example, in Figure 3, the former corresponds to the pairs of zeros with diagonal background lines and ones with the same diagonal background lines, while the latter corresponds to the pairs of ones, independent of background types.

This loss function is realized by introducing weight coefficients depending on pairwise similarity between the background features of target samples $\{w_{i}\}_{i=1}^{N}$. Our new loss function, namely, contrastive instance discrimination loss, is defined as

where $\alpha_{ij}$ is the weight coefficient between the $i$-th and $j$-th samples that determine how strongly target features $v_{i}$ and $v_{j}$ are pulled apart in the learning process. This coefficient is formulated simply as

When the background features of the $i$-th and $j$-th images are similar, $\alpha_{ij}$ becomes large, causing their repulsive force in the target feature space to increase. This effect, shown in Fig. 3, reduces the similarities between the feature vectors of zeros with diagonal lines and ones with diagonal lines. $\alpha_{ij}$ also becomes large for pairs having the same background and target, but there is also the usual contrastive learning effect, and their repulsion becomes weakened. For pairs of small $\alpha_{ij}$, the repulsive force is relatively weak, and thus samples with the same target and different backgrounds are weakly separated. We experimentally demonstrated that our method works according to the perspectives described above; the details are given in Section 4.4.

Temperature parameters $\tau_{x}$ control the distribution of feature vectors, and $\tau_{xb}$ controls the magnitude of the weight coefficient $\alpha_{ij}$. The differences from IDFD are the background module $f_{\theta}$ and $\alpha_{ij}$. In the case of $\lim_{\tau_{xb}\to\infty}\alpha_{ij}=1$, cIDFD is consistent with IDFD because the contribution from the background module $f_{\theta}$ to the target module $g_{\phi}$ disappears.

We consider separately learning the embedding functions $f_{\theta}$ and $g_{\psi}$. In the first step, the $f_{\theta}$ branch learns a background dataset $\mathcal{B}$ by minimizing the objective function

During this step, the input samples are only from $\mathcal{B}$, and the network parameters of $g_{\psi}$ are not updated. After learning, $f_{\theta}$ works as an extractor of features to be discarded. In the second step, we freeze the parameters of $f_{\theta}$ and train $g_{\psi}$ by dataset $\mathcal{X}$. The objective function is a composition of the contrastive instance discriminative loss (4) and feature decorrelation loss for the target features,

where $H=\{h_{l}\}_{l=1}^{d}$ are the vectors defined by the transpose of $V$. The above learning process is summarized as Algorithm1.

We evaluated the performance of cIDFD on three datasets created from commonly used public datasets. Each dataset contains both target and background datasets. Table 1 summarizes the key details. Figure 4 shows sample images from each dataset.

Stripe MNIST We created a synthetic image dataset using handwritten digits from the Modified National Institute of Standards and Technology (MNIST) database [7] and randomly generated artificial stripe patterns, which are shown in Fig. 4. Our goal was to cluster the ten handwritten digits independently of the background patterns. We also prepared a background dataset of stripe-pattern images that are almost the same as those in the target dataset, but not used in its creation (Fig. 4).

CelebA-ROH We assume a clustering task that focuses on certain features of facial images. We made datasets from the popular celebrity facial images dataset CelebA [11]. As a target dataset, we collected images with the target attributes "receding hairlines" or "wearing hats" (Fig. 4). We used the remaining celebrities as the background dataset, which we call CelebA-RNH (Fig. 4). Our goal was to cluster the two target attributes independently of other attributes such as eyeglasses, hair color, or gender.

Birds400-ABC As a target dataset, we collected images from Birds400 [15], which are derived from the Kaggle datasets. Birds400 contains images of 400 types of birds with various backgrounds. From the training split of the original dataset, we utilized 144 bird species with names starting with A, B, or C (Fig. 4). To realize bird-oriented clustering by cIDFD, we used the Landscape Pictures dataset [16], which includes high-quality images of natural landscapes. By randomly cropping and resizing those images, we generated 41,733 samples with $224\times 224$ pixels for the background dataset (Fig. 4).

We compared cIDFD with VAE, cVAE and eight other competitive deep clustering methods: CC, MiCE, SCAN, RUC, NNM, TCL, SPICE, IDFD. Given that VAE and the eight competitive deep clustering methods have no way to handle the background dataset, only the target dataset was used. In the clustering phase, we applied simple $k$-means to representations for VAE, cVAE, IDFD, and cIDFD. The number of clusters $k$ is set equal to the number of classes in each dataset as shown in Table 1. Clustering performance was evaluated by three popular metrics: clustering accuracy (ACC), normalized mutual information (NMI), and the adjusted rand index (ARI). These metrics give values in the range $[0,1]$, with higher scores indicating more accurate clustering assignments.

Table 2 lists performances for each method. These results show that cIDFD clearly outperform the conventional methods. In terms of ACC, the cIDFD scores were improved by approximately 24% for Stripe MNIST, by 7% for CelebA-ROH, and by 25% for Birds400-ABC. In particular, performance under our method is better than performance under cVAE, which is the most similar method in terms of CA setting usage, even for complex datasets such as CelebA-ROH and Birds400-ABC.

Fig. 5 visualizes feature representations of the three datasets, which are learned by IDFD and cIDFD. 128-dimensional representations were embedded into two-dimensional space by UMAP [12]. Point colors indicate ground truth labels. The distribution clearly shows that cIDFD is preferable to IDFD when grouping samples into clusters characterized by ground truth labels, which are the features we focus on. For Stripe MNIST, IDFD created clusters excessively according to both the digits and stripes features, while cIDFD generated almost ten clusters. For CelebA-ROH, IDFD generated one distribution mixing two classes, but cIDFD generated distinct distributions according to the ground truth labels. cIDFD also correctly distinguished an extremely large number of bird species; however, in the IDFD results, samples of many classes were degenerated to several large clusters.

To clearly understand how our method works, we conducted experiments on similarity distribution for both IDFD and cIDFD with the synthetic image dataset Stripe MNIST. Fig. 6 shows the resulting histograms on four types of average similarity, which were calculated for each instance: the first type is similarity to instances of the same background and a different target; the second type is to instances of a different background and the same target; the third type is to instances of the same background and the same target; and the fourth type is to instances of a different target and a different background. In the case of IDFD (Fig. 6), the peaks of the first and second type of similarity are located in almost same lower region. As mentioned in Section 3.3, this situation is problematic. In contrast, cIDFD successfully moved the peak of the second type to the same position of the third type distribution in higher region (Fig. 6). On the other hand, the first and fourth type distributions are located in the same lower region. Consequently, instances with the same target features were clustered independently of the background features.