Riemann-based Multi-scale Attention Reasoning Network for Text-3D Retrieval

Cross-modal retrieval has gained significant attention because of the semantic heterogeneity present in multi-modal data (Liu et al. 2021; Wen, Han, and Liu 2021; Li et al. 2022b; Yu et al. 2020; Li et al. 2023a, 2022a; Wei et al. 2023; Wang et al. 2019; Wehrmann et al. 2019; Tang et al. 2024b; Li et al. 2024c). A key technique in this field is cross-modal alignment, currently divided into two primary approaches: coarse-grained and fine-grained methods. Coarse-grained methods align data by utilizing a shared embedding space and cosine similarity (Fu et al. 2022, 2023), as demonstrated by the self-supervised ranking framework (Fu et al. 2021). Conversely, fine-grained methods (Diao et al. 2021; Lee et al. 2018) emphasize cross-modal interactions at the local feature level, such as the semi-supervised learning approach using Graph Convolutional Networks (GCNs) (Bahdanau, Cho, and Bengio 2014). Although effective, the Transformer model is constrained by its large number of parameters. Inspired by the brain’s recurrently connected neurons, RCTRN has shown promising performance (Li et al. 2023b). MPARN also addresses annotation uncertainty by modeling visual and textual data as probability distributions (Li, Xiong, and Fan 2024). Cross-modal retrieval has broad applications, such as text-to-video, text-to-audio, and image-text retrieval. However, a significant gap exists in current research on 2D-3D retrieval, despite the substantial structural differences and richer semantic information inherent in 3D data.

Indoor scene datasets are currently classified into two main categories: scanned scenes and synthetic scenes. The NYU Depth Dataset V2 (Zhang et al. 2023), one of the earliest indoor scene scanning datasets, contains 1,449 RGB-D images across 464 diverse indoor scenes, offering detailed annotations for understanding key surfaces, objects, and support relationships in indoor environments. Other datasets like SUN RGB-D (Song, Lichtenberg, and Xiao 2015) and ScanNet (Dai et al. 2017) provide extensive RGB-D data, greatly advancing scene understanding research. Synthetic datasets, compared to scanned scenes, offer the advantage of easy scalability to much larger datasets. SceneNet (Handa et al. 2016), a framework for indoor scene understanding, generates high-quality annotated 3D scenes by learning from manually annotated real-world datasets like NYUv2. Using a simulated annealing optimization algorithm, SceneNet samples objects from existing 3D object and texture databases, facilitating the creation of an almost unlimited number of new annotated scenes. SUNG (Song et al. 2016) excels in scalability, while InteriorNet (Li et al. 2018) provides highly realistic data. Synthetic datasets, compared to scanned scenes, offer the advantage of easy scalability to much larger datasets. SceneNet (Handa et al. 2016), a framework for indoor scene understanding, generates high-quality annotated 3D scenes by learning from manually annotated real-world datasets like NYUv2. Using a simulated annealing optimization algorithm, SceneNet samples objects from existing 3D object and texture databases, facilitating the creation of an almost unlimited number of new annotated scenes. SUNG excels in scalability, while InteriorNet (Li et al. 2018) provides highly realistic data. However, none of the aforementioned datasets simultaneously cover both coarse-grained and fine-grained scenes with complex object combinations. To bridge this gap, we propose the T3DR-HIT dataset, a large-scale, high-quality, multi-scale dataset specifically designed for indoor scene understanding.

The textual AFR and point cloud AFR are identical, with each consisting of a stack of six Self-Attention Encoders (Vaswani et al. 2017). These AFR modules fine-tune the features of their respective modalities and map them into a common feature space, enabling the subsequent computation of Riemann Attention. Internally, each AFR layer consists of multi-head self-attention (MSA) sub-layers and feed-forward neural network (FFN) sub-layers. Each of these sub-components (MSA and FFN) is encapsulated within residual connections and layer normalization operations.

The AFR receives text or point cloud inputs, using a scaled dot-product attention mechanism to describe both visual and textual features. The output of the self-attention operator is defined as:

	$\displaystyle Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=softmax\left(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_{e}}}\right)\mathbf{V},$		(1)
	$\displaystyle Att(\mathbf{X})=Attention(\mathbf{X}\mathbf{W}_{Q},\mathbf{X}\mathbf{W}_{K},\mathbf{X}\mathbf{W}_{V}),$

where $\mathbf{W}_{Q}\in\mathbb{R}^{d\times d_{e}}$, $\mathbf{W}_{K}\in\mathbb{R}^{d\times d_{e}}$, and $\mathbf{W}_{V}\in\mathbb{R}^{d\times d_{e}}$ represent the learnable linear transformations for the query, key, and value, respectively. $d_{e}$ indicates the dimensionality of the embedding space. We utilize a compact feed-forward network (FFN) to extract features, which are already integrated into more extensive representations. The FFN is composed of two nonlinear layers:

	$\displaystyle FFN(\mathbf{X}_{i})=\text{GELU}(\mathbf{X}_{i}\mathbf{W}_{1}+\mathbf{b}_{1})\mathbf{W}_{2}+\mathbf{b}_{2},$		(2)
	$\displaystyle GELU(x)=\epsilon x\left(1+\tanh\left[\sqrt{\frac{2}{\pi}}(x+\rho x^{3})\right]\right),$

where $\epsilon$ and $\rho$ are hyperparameters, $\mathbf{X}_{i}$ represents the $i$-th vector of the input set, $\mathbf{W}_{1}$ and $\mathbf{W}_{2}$ are learnable weight matrices, and $\mathbf{b}_{1}$ and $\mathbf{b}_{2}$ are bias terms. The GELU activation function can enhance the model’s generalization capabilities.A complete encoding layer $A_{i}$ ($i=1,\ldots,n$) can be described as follows:

	$\displaystyle\mathbf{S}=Add\&Norm(Att(\mathbf{X})),$		(3)
	$\displaystyle\hat{\mathbf{X}}=Add\&Norm(FFN(\mathbf{S})),$

where Add & Norm includes a residual connection and layer normalization. The multi-layer encoder $A_{i}$ ($i=1,\ldots,n$) is constructed by stacking these encoding layers sequentially, with the input of each layer being derived from the output of the preceding layer. In the AFR, stacking multiple encoder layers enables the automatic adjustment of weights between features, ensuring that crucial ones receive greater attention. This adaptive feature enhancement makes the model more flexible and efficient in handling complex, high-dimensional text and point cloud data, thereby improving the accuracy of subsequent similarity computations.

The text feature $\mathbf{T}\in\mathbb{R}^{s_{T}\times h_{T}}$ and the point cloud feature $\mathbf{P}\in\mathbb{R}^{s_{P}\times h_{P}}$ are both represented as vector sequences, allowing them to be considered as samples originating from two distinct fields distributed across a manifold at specific loci. To facilitate subsequent derivations, we will adopt the Einstein summation convention, using $\mathbf{T}_{\mu}$ and $\mathbf{P}_{\mu}$ as predefined notational constructs to denote the characteristics of text and point clouds pertaining to a specific token.

In Riemannian geometry, directly assessing the similarity between two tensors at different positions is not practically meaningful. To quantify the similarity between tensors at distinct points within two fields, it is essential to first transport them to the same location. This process involves transporting the tensor from the text field at point $P$ to point $Q$, accomplishing this by leveraging the connection $\Gamma$ alongside the displacement $dx$:

	$\displaystyle\mathbf{T}_{\mu}^{(P\rightarrow Q)}$	$\displaystyle=\mathbf{T}_{\mu}^{(P)}+\Gamma_{\mu\gamma}^{\alpha}\mathbf{T}_{\alpha}^{(P)}{dx}^{\gamma}$		(4)
		$\displaystyle=\mathbf{T}_{\alpha}^{(P)}(\delta_{\mu}^{\alpha}+\Gamma_{\mu\gamma}^{\alpha}{dx}^{\gamma}),$

where $\delta_{\mu}^{\alpha}$ is the Kronecker symbol. Calculate the similarity between two tensors at the same position after translation using dot product:

$$sim(\mathbf{T}_{\mu}^{(P\rightarrow Q)},\mathbf{P}_{\mu}^{(Q)})=g^{\mu\gamma}\mathbf{T}_{\mu}^{(P\rightarrow Q)}\mathbf{P}_{\gamma}^{(Q)},$$

(5)

where $g^{\mu\gamma}$ is the metric of the manifold.

By substituting equation 4 into equation 5, we can obtain:

	$\displaystyle sim(\mathbf{T}_{\mu}^{(P\rightarrow Q)},\mathbf{P}_{\mu}^{(Q)})$	$\displaystyle=g^{\mu\gamma}(\delta^{\alpha}_{\mu}+\Gamma_{\mu\gamma}^{\alpha}{dx}^{\gamma})\mathbf{T}_{\alpha}^{(P)}\mathbf{P}_{\gamma}^{(Q)}$		(6)
		$\displaystyle=\Omega^{\alpha\gamma}\mathbf{T}_{\alpha}^{(P)}\mathbf{P}_{\gamma}^{(Q)},$

where $\Omega^{\alpha\gamma}=g^{\mu\gamma}(\delta^{\alpha}_{\mu}+\Gamma_{\mu\gamma}^{\alpha}{dx}^{\gamma})$ is a two-dimensional tensor, and depends on the position of points $P$ and $Q$.

To simplify the parameters, the $\mathbf{T}_{\alpha}\Omega^{\alpha\gamma}\mathbf{P}_{\gamma}$ term related to PQ position is approximately decomposed into a PQ position independent term $t^{\prime}\mathbf{Q}p$ and a position only term $e$, and $\mathbf{Q}$ is matrix decomposed to obtain:

		$\displaystyle\mathbf{T}_{\alpha}\Omega^{\alpha\gamma}\mathbf{P}_{\gamma}=t^{\prime}\mathbf{Q}p+e^{(P,Q)}$		(7)
		$\displaystyle=t^{\prime}A^{\prime}Bp+e^{(P,Q)}=(At)^{\prime}(Bp)+e^{(P,Q)},$

where $t$ and $p$ represent specific token features within the text feature sequence $\mathbf{T}$ and the point cloud feature sequence $\mathbf{P}$, respectively. Using the aforementioned equation, we can compute the local (token-level) similarity between any pair of text and point cloud tokens. This local similarity metric can then be used to enhance attention to intricate details when calculating the overall global similarity.

On any manifold, the local similarity between any pair of text and point cloud tokens can be represented as a local similarity matrix. To enhance the similarity measurement from multiple perspectives, we can compute local similarity matrices on multiple ($k$) manifolds, resulting in a similarity map $\mathbf{M}\in\mathbb{R}^{k\times s_{1}\times s_{2}}$ with $k$ channels. We can use convolutional network $\mathbf{C}$ to construct a mapping that converts the $k$ local similarity matrices to the total similarity $s\in\mathbb{R}$:

$$s=\mathbf{C}\circ\mathbf{M}.$$

(8)

Given the inherent constraints of compressing data within the model, redundant information inevitably persists within both point cloud feature sequences and text feature sequences, hindering the model’s generalization capabilities and exacerbating computational intricacies. Consequently, it becomes imperative to leverage low-rank priors (Hu et al. 2021) as a means of eliminating this redundant information.

When given the original feature map $\mathbf{M}$ containing redundant information, we can use the following equation to extract the low rank component $\mathbf{X}$ from it:

$$\mathbf{X}=arg\min_{x}\left\{{\left\|\mathbf{M}-x\right\|^{2}_{F}+\lambda\left\|\mathbf{D}x\right\|_{1}}\right\},$$

(9)

where $\lambda$ is the regularization coefficient that balances sparse loss and data restoration loss. Assuming $\mathbf{D}$ is orthogonal, then the minimization problem has a closed solution $\mathbf{X}=\mathbf{D}^{-1}soft(\mathbf{DM},\ lambda)$, where $soft$ is the soft interval function:

$$soft(x,\lambda)=\left\{\begin{matrix}x-\lambda,&x>\lambda;\\ x+\lambda,&x<-\lambda;\\ 0,&otherwise.\end{matrix}\right.$$

(10)

This article uses neural networks to approximate the mapping of $\mathbf{D}$. Since the total similarity $s$ is a function of $X$, it is:

$$s=\mathbf{CX}=\mathbf{CD}^{-1}soft(\mathbf{DM},\lambda).$$

(11)

Therefore, a complete neural network can be used to simultaneously approximate $\mathbf{CD}^{-1}$without explicitly approximating $\mathbf{C}$ and $\mathbf{D}^{-1}$separately.

RMARN aims to maximize the similarity between paired point clouds and text samples, while minimizing the similarity between unmatched samples. Consequently, it adopts a similar contrastive loss framework as CLIP, leveraging softmax to derive the retrieval pairing probability, which is grounded on the computed similarity between point cloud text pairs $(p,t)$ sampled from our dataset $D$. The specific formulation is outlined below:

$$\mathcal{L}=-E_{p(t,p|D)}\left[\alpha_{1}log(\frac{e^{\frac{f(t,p)}{\tau_{1}}}}{{\textstyle\sum_{p^{\prime}}{e^{\frac{f(t,p^{\prime})}{\tau_{1}}}}}})+\alpha_{2}log(\frac{e^{\frac{f(t,p)}{\tau_{2}}}}{{\textstyle\sum_{t^{\prime}}{e^{\frac{f(t^{\prime},p)}{\tau_{2}}}}}})\right],$$

(12)

where $f$ is a similarity calculation function implemented by RMARN, $\alpha_{1}$ and $\alpha_{2}$ are served as hyperparameters, used to balance the directional focus when retrieving text and point clouds from each other.

T3DR-HIT is a comprehensive, large-scale Text-3D Retrieval dataset designed to facilitate research and development in cross-modal retrieval, particularly involving textual descriptions and 3D spatial data. The dataset comprises over 3,380 pairs of text and point cloud data, making it one of the most extensive resources available for studying the interaction between these two modalities. T3DR-HIT is divided into two distinct segments to accommodate different levels of granularity in 3D scene representation: coarse-grained Indoor 3D Scenes and fine-grained Chinese Artifact Scenes.

We selected the CLIP text encoder and PointNet as the baseline feature extractors for handling the text and point cloud modalities, respectively. These choices are based on the proven capabilities of CLIP in capturing rich semantic information from textual data and PointNet’s effectiveness in processing irregular 3D point clouds. The encoder employs LayerNorm for normalization, ensuring stable and consistent scaling of the input data. For the activation function, we utilized GELU (Gaussian Error Linear Unit) with the parameters $\epsilon=0.5$ and $\rho=0.044715$ which balances non-linearity and smooth gradient flow. A dropout rate of 0.1 is applied to prevent overfitting by randomly zeroing out a fraction of the neurons during training. Both the Attention layer and the Feed-Forward Network (FFN) in the self-attention encoder are configured with a dimensionality of 512. This dimensional setting ensures that the model can capture complex relationships within the data without excessively increasing computational costs. We trained the model for 100 epochs, utilizing the Adam optimizer, which is well-regarded for its ability to adapt learning rates during training. The learning rate was set to 0.008, providing a balance between making steady progress and avoiding potential overshooting of minima. The $\beta$ for the Adam optimizer were configured as (0.91, 0.9993).

The Table 1 summarizes the performance of various models on the T3DR-HIT dataset, including their respective hyperparameter configurations. In the Fast Model category, the GNN-based models (PointNet and PointNet++) show poor retrieval performance across all metrics (R@1, R@5, R@10), especially with near-zero R@1 scores, highlighting weak retrieval accuracy for the top candidate. The CLIP-based models perform slightly better, but their overall performance remains suboptimal.

In contrast, the Base Model category shows that BERT-based models, especially the BERT + PointNet++ combination, significantly outperform others, achieving an R@10 score of 161, which is much higher than that of other models. Although the CLIP-based models show improvements over the Fast Models, they still lag considerably behind the BERT-based models. The superior performance of the BERT + PointNet++ model can be attributed to its more sophisticated hyperparameter settings, including a higher Low Rank value (256), larger batch size (64), increased number of attention heads (32), and additional SA layers (8), which enhance its feature extraction capabilities and overall retrieval accuracy. This suggests that the BERT + PointNet++ model has substantial potential for 3D object retrieval tasks. The hyperparameter configurations of the models highlight key differences that contribute to their varying performances. These configurations suggest that more complex and well-tuned hyperparameters are crucial for improving the effectiveness of 3D object retrieval tasks.