Instance-Variant Loss with Gaussian RBF Kernel for 3D Cross-modal Retrieval

Assuming dataset $S$ contains $n*N*M$ instances, with $n$ instances of $N$ categories from $M$ modalities, the $i$-th instance $t_{i}$ is a set consisting of the set of modalities $s_{i}$ with a semantic label $y_{i}$ and weight vector $W_{y_{i}}$. Formally:

Generally, the modality instances $\{x_{i}^{1},x_{i}^{2},\cdots,x_{i}^{M}\}$ are in $M$ different representation spaces, and their similarities (distances) cannot be directly measured. The 3D cross-modal retrieval task aims to learn $M$ projection functions $f_{m}$ for each modality $m\in[1,M]$, where $v_{i}^{m}=f_{m}(x_{i}^{m},\theta_{m})$, $V_{y_{i}}^{m}=\{v_{1}^{m},v_{2}^{m},\cdots,v_{n}^{m}\}$, $\theta_{m}$ is a learnable parameter. $v_{i}^{m}$ is a projected feature in the common representation space, and $V_{y_{i}}^{m}$ is the set of all features of the same class. To obtain optimal retrieval performance, we aim to ensure that the distance between same-class instances and the intra-class distance $D(V_{y_{i}}^{m},W_{y_{i}})$ is smaller than the distance between instances from different classes and the inter-class distance $D(V_{y_{i}}^{m},W_{j})$ by a significant margin $\lambda_{0}$.

To better understand the proposed Instance-Variant loss, we will first briefly review the original softmax, A-softmax (Liu et al., 2017), and AM-softmax (Wang et al., 2018). The formulation of the original softmax loss is given by

where $f$ is the input of the last fully connected layer ($f_{i}$ denotes the $i$-th sample), and $W_{j}$ is the $j$-th column of the last fully connected layer. The $W{y_{i}}\cdot\boldsymbol{f}_{i}$ is also called the target logit (Pereyra et al., 2017) of the $i$-th sample. $W_{j}$ is also called the weight vector of the $j$-th category. The relationship between the weight vector and the features’ mean vector (class center) is described in Figure 6 of (Wang et al., 2017). In the A-softmax loss, the authors proposed to normalize the weight vectors (making $\left\|W_{i}\right\|$ to be 1) and generalize the target logit from $\left\|f_{i}\right\|\cos(\theta{y_{i}})$ to $\left\|f_{i}\right\|\psi(\theta_{y_{i}})$, where $\psi(\theta)=\frac{(-1)^{k}\cos(m\theta)-2k+\lambda\cos(\theta)}{1+\lambda}$. In the AM-softmax, the authors proposed to normalize both the weight vectors and instances’ features ($\left\|W_{i}\right\|=\left\|f_{i}\right\|=1$), while the $\psi(\theta_{y_{i}})=\cos(\theta_{y_{i}})-\lambda_{0}$.

The softmax loss function has been extensively applied in uni-modal retrieval tasks, as it not only identifies the optimal plane to separate distinct data classes but also learns the ideal weight vector for each category. In comparison to the class center (feature mean), the weight vector is better suited for metric learning problems involving hard samples. Nevertheless, in contrast to uni-modal retrieval tasks, cross-modal retrieval tasks must also diminish cross-modal discrepancy. We innovatively learn a shared weight vector for data across various modalities ($W{y_{i}}=w_{y_{i}}^{1}=w_{y_{i}}^{M}$). Throughout the weight vector’s iterative process, it will acquire the features of all modal data, effectively addressing the imbalance in modal optimization. This paper assumes that the norm of $W_{i}$ and $f$ are normalized to 1 if not specified. Given the extracted features $\{V_{i}^{m}\}\quad(i\in[1,n*N],m\in[1,M])$, $f_{i}^{m}=V_{i}^{m}$. The new cross-modal softmax loss takes the following form.

To simplify subsequent derivations, we can rewrite Equation 4 as Equation 5.

To ensure the intra-class distance is smaller than the inter-class distance, $1-\cos(\theta_{j}^{m})-(1-\cos(\theta_{y_{i}}^{m}))\geq\lambda_{0},\quad\lambda_{0}>0$, where $\cos(\theta_{y_{i}}^{m})-\lambda_{0}\geq\cos(\theta_{j}^{m})$, where $\cos(\theta_{y_{i}}^{m})>\cos(\theta_{j}^{m})$.

Thus, we can let $\eta(\theta_{j}^{m})=\left\|W_{j}\right\|\left\|\boldsymbol{f}_{i}^{m}\right\|\cos(\theta_{j}^{m})=\cos(\theta_{j}^{m})$,

and $\phi(\theta_{y_{i}}^{m})=\left\|W_{i}\right\|\left\|\boldsymbol{f}_{i}^{m}\right\|\cos(\theta_{y_{i}}^{m})-\lambda_{0}=\cos(\theta_{y_{i}}^{m})-\lambda_{0}$ .

If the network applies the same penalty strength to instances, ambiguous convergence (local optima) severely compromises the feature space’s separability. To address the limitation, we consider enhancing the optimization flexibility by allowing each instance to learn at its own pace, depending on its current optimization status. The Instance-Variant loss takes the following form.

Let $\Gamma=\sum_{j=1,j\neq{y_{i}}}^{N}e^{\eta(\theta_{j}^{m})-\phi(\theta_{y_{i}}^{m})}$,

Under the same condition that factors affecting loss convergence, such as input data and optimizer, $\mathcal{L}_{IV}<\mathcal{L}_{NS^{\prime}}$. Take the partial derivative of $\Gamma$ with respect to $\theta_{y_{i}}^{m}$ ($\theta_{y_{i}}^{m}\in[0,\pi]$) and derivative of $\mathcal{L}_{IV}$ and $\mathcal{L}_{NS^{\prime}}$ with respect to $\Gamma$, $\frac{\partial\Gamma}{\partial\theta_{y_{i}}^{m}}<0$, $\frac{d\mathcal{L}_{IV}}{d\Gamma}>0,\frac{d\mathcal{L}_{NS^{\prime}}}{d\Gamma}>0$. Therefore, we can get that both $\mathcal{L}_{IV}$ and $\mathcal{L}_{NS^{\prime}}$ are monotonically decreasing concerning $\theta_{y_{i}}^{m}$. If we assume the $\mathcal{L}_{IV}$ and $\mathcal{L}_{NS^{\prime}}$ are optimized to the same value and all training features can be perfectly classified (consistent with the L-softmax assumption (Liu et al., 2016)).

When $\mathcal{L}{IV}$ and $\mathcal{L}{NS^{\prime}}$ are optimized to the same value and all training features can be perfectly classified, the $\theta_{y_{i}}^{m}$ of $\mathcal{L}{IV}$ will be smaller than the one of $\mathcal{L}{NS^{\prime}}$, which means $\mathcal{L}{IV}$ will have better performance than $\mathcal{L}{NS^{\prime}}$ (Proved in Appendix). During the training optimization process, an instance will receive a stronger penalty when it is difficult to distinguish (the intra-class distance is large or the inter-class distance is small). Compared to the traditional softmax loss, we introduce the concept of hard negative mining. Moreover, we assess both the intra-class and inter-class distances to define hard instances, effectively enhancing the separability of the feature space. We can also use the hyperparameter $\tau$ to scale $\frac{\Gamma}{1+\Gamma}$, adapting to different data distributions in various datasets. Simultaneously, researchers can improve the retrieval performance of the network by only modifying $\phi(\theta_{y_{i}}^{m})$ and $\eta(\theta_{j}^{m})$. The proposed Instance-Variant loss can take the following form.

where $\lambda_{0}$ and $\tau$ are hyperparameter, $\omega$ is the temperature coefficient (Hinton et al., 2015) for softmax.

Under the constraint of the softmax loss, we can map all modality data onto a shared unit hypersphere by normalizing the weight vector and instance features. We aim for the intra-class metric to be asymptotically correct and empirically reasonable with a finite number of points. Also, the essence of the Gaussian RBF kernel is to measure the similarity between samples. Similar samples can be better clustered together in a space that describes their similarity, and subsequently become linearly separable. We consider the Gaussian Radial Basis Function (RBF) kernel to achieve this.

and define the Intra-Class loss as the negative log-likelihood of the average pairwise Gaussian RBF:

The objective of this loss function is to minimize the distance between any two instances of the same class from different modalities, thereby reducing the intra-class distance among all data within the same class. Compared to existing Intra-Class loss functions (Xiao et al., 2022; Jing et al., 2021), our Intra-Class loss straightforwardly minimizes the distance between two instances without learning class centers, reducing the network’s complexity and the cross-modal discrepancy. We will discuss more mathematic characters in the appendix.

We propose an end-to-end framework for cross-modal retrieval tasks based on proposed Instance-Variant loss, Intra-Class loss, and cross-entropy loss. The overview of the 3D cross-modal retrieval task framework is illustrated in Fig. 2. For 2D image feature extraction, we utilize ResNet18 (He et al., 2016) as the backbone network with four convolution blocks, all with 3 × 3 kernels, where the number of kernels is 64, 128, 256, and 512, respectively. DGCNN is employed as the backbone network to capture point cloud features. DGCNN (Wang et al., 2019b) contains four EdgeConv blocks with kernels set to 64, 64, 64, and 128. MeshNet (Feng et al., 2019) is used to extract mesh features. The shared multimodal feature encoder consists of two fully connected layers with a size of 512,256,C (C is the number of classes). The three proposed loss functions are used to train the network to learn discriminative and modal-invariant features jointly: $\mathcal{L}=\mathcal{L}_{IV}+\mathcal{L}_{IC}+\mathcal{L}_{CE}.$

Training details. We implemented our network using PyTorch (Paszke et al., 2019). For all three datasets, our network is trained with an SGD optimizer and a learning rate of 0.01. The learning rate is reduced by 90$\%$ every 20,000 iterations for ModelNet40 and MI3DOR, and every 4,000 iterations for Pix3D. We set the temperature parameter $\omega=\frac{1}{30}$ and $\lambda_{0}=0.35$ for all three datasets, and $\tau=0.1$, $\tau=0.1$, and $\tau=8$ for ModelNet40, MI3DOR, and Pix3D, respectively. Since our work is the first to employ real-world datasets like MI3DOR and Pix3D for 3D cross-modal retrieval tasks, we lack baseline methods for comparison. As a result, we have chosen Cross-Modal Center Loss (Short as CMCL) (Jing et al., 2021) as our baseline and conducted relevant experiments on the aforementioned datasets. CMCL training details on MI3DOR and Pix3D datasets are the same as those on ModelNet40.

Evaluation Metrics. The evaluation results for all experiments are presented with the Mean Average Precision (mAP) score, a classical performance evaluation criterion for cross-modal retrieval tasks (Jing et al., 2021; Chen et al., 2021; Zeng et al., 2021). The mAP for the retrieval task measures whether the retrieved data belong to the same class as the query (relevant) or not (irrelevant). Given a query and a set of R corresponding retrieved data (R top-ranked data), the Average Precision is defined as:

where $T$ is the number of relevant items in the retrieved set, $P_{r}$ represents the precision of the top $r$ retrieved items, and $\delta(r)$ is an indicator function whose value is one if the $r$-th retrieved item is relevant (here relevant means belonging to the category of the query). The MAP can be calculated by averaging the AP values.

To evaluate the effectiveness of the proposed loss function, we conduct experiments on the ModelNet40, MI3DOR, and Pix3D datasets with three different modalities, including images, point clouds, and meshes. To thoroughly examine the quality of the learned features, we perform two retrieval tasks: uni-modal retrieval and cross-domain retrieval. The performance of our method for 3D uni-modal and cross-modal retrieval tasks is shown in Table 1. Since only the Cross-Modal Center Loss (CMCL) is proposed for 3D cross-modal retrieval, and they only experiment on the ModelNet dataset, we reproduce the CMCL method on MI3DOR and Pix3D datasets to evaluate the retrieval performance from real-image. Our proposed jointly trained method significantly outperforms the state-of-the-art method on all retrieval tasks and datasets. In particular, when the input retrieval data consists of real images, our retrieval method demonstrates a more significant performance compared to the CMCL method. This indicates that our approach can effectively handle real datasets with challenging negative examples. This can be attributed to our Instance-Variant loss assigning different penalty strengths to different instances, improving the space separator. Our method obtained significantly better performance on all retrieval pairs across all datasets, showcasing the strong generalization ability of our proposed method.

The three components of our proposed loss function are as follows: Cross-entropy loss for each modality in the label space, denoted as $\mathcal{L}_{CE}$; Instance-Variant loss in the shared embedding space, denoted as $\mathcal{L}_{IV}$; and Intra-Class loss, denoted as $\mathcal{L}_{IC}$. We also mention the softmax loss without the instance-variant weight, denoted as $\mathcal{L}_{NS^{\prime}}$. To further investigate the impact of each component, we evaluate different combinations for the loss functions, including optimization with $\mathcal{L}_{CE}$, $\mathcal{L}_{IV}$, and $\mathcal{L}_{NS^{\prime}}$ respectively; jointly optimization with $\mathcal{L}_{CE}\&\mathcal{L}_{IV}$, $\mathcal{L}_{CE}\&\mathcal{L}_{IC}$, $\mathcal{L}_{IV}\&\mathcal{L}_{IC}$; jointly optimization with $\mathcal{L}_{CE}$, $\mathcal{L}_{IC}$, and $\mathcal{L}_{IV}$. These five networks are trained with the same setting and hyper-parameters, where the performance is shown in Table 2. As illustrated in Table 2. A) The combination of $\mathcal{L}_{CE}$, $\mathcal{L}_{IC}$, and $\mathcal{L}_{IV}$ achieves the best performance for all cross-modal and uni-modal retrieval tasks. B) As the baseline, cross-entropy loss alone achieves relatively high mAP due to the sharing head of the three modalities forcing the network to learn similar representations in the common space for different modalities of the same class. C) The Instance-Variant loss can be used independently, achieving fairly good retrieval results. When combined with the Intra-Class loss, some tasks’ retrieval performance surpasses the CMCL method’s. D) $\mathcal{L}_{IC}$ can improve both performances of the uni-modal or cross-modal retrieval. E) $\mathcal{L}_{IV}$ is better than the $\mathcal{L}_{NS^{\prime}}$.

Few researchers have discussed the difference between the weight vector and class center (Wang et al., 2017, 2018). The researchers compared weight vector and class center distribution differences before and after full optimization, concluding that their distributions overlap at optimum network optimization (Wang et al., 2017). It has been proved that batch size affects center loss retrieval performance (Jing et al., 2021; Wen et al., 2016) Consequently, we further investigated the impact of batch size on weight vector calculation and Intra-Class loss. To analyze the impact of batch sizes on the performance, we conduct experiments on the ModelNet40 dataset with different batch sizes (32,64,96,128). All networks are trained with the same number of epochs and hyper-parameters. As shown in Table 3, changing the batch size does not significantly impact the retrieval performance, indicating that the weight vector and Intra-Class loss are not affected by the variations in the amount of data within a batch. This also implies that the loss function proposed in this paper can achieve comparable retrieval results with fewer resources. Meanwhile, we noticed that the performance of some retrieval tasks was improved when using smaller batch sizes. We speculate that the primary factors are hyperparameter selection and imbalanced modality optimization issues leading to imperfect feature extraction (Peng et al., 2022).

Considering the real-world application of cross-modal retrieval, we may encounter new data domains or distributions that have not appeared in the training dataset. Therefore, we use real-image of the MI3DOR dataset training the retrieval network, using views of MI3DOR and Pix3D datasets to test. The main reason for choosing the MI3DOR dataset as the benchmark is that it contains both real images corresponding to the models and multi-view images of the models; at the same time, the data distribution in this dataset is more balanced compared to Pix3D, so we do not need to worry about the unbiased nature of the networks used for testing. As illustrated in Table 4, our network has better generalization ability than the CMCL method. However, it fails to address the challenge of unseen domain retrieval, as the network’s performance experiences a significant decline. We will discuss more experimental results about unseen retrieval in the appendix.

T-SNE Feature Embedding Visualization. Fig. 3 demonstrates distinct clusters for each modality, highlighting the proposed approach’s effectiveness in discriminating class samples. Furthermore, the combined features across modalities confirm the learned common space can capture modality-invariant representations.

3D Cross-Modal Retrieval Visualization. Fig. 4 displays the cross-modal retrieval samples for six queries from the ModelNet40 and MI3DOR datasets. Cosine distance measures data similarity across different modalities using normalized features for each query. The figure shows that instances with similar appearances are closer in feature space despite different modalities, proving the network learned modality-invariant features. More experiment results will be illustrated in the appendix.