Long-Range Grouping Transformer for Multi-View 3D Reconstruction

Traditional methods such as SfM [20] and SLAM [21] reconstruct scenes by matching features and estimating camera pose. However, they struggle to handle images in complex situations. Recently, deep learning-based methods have gained significant attention in multi-view 3D reconstruction.

In RNN-based and CNN-based methods, the encoder and fusion modules are separated. 3D-R2N2 [1] and LSM [2] use RNN to fuse multi-view features. However, RNNs are permutation-invariant and time-consuming. 3DensiNet [4] aggregates multi-view features using max-pooling, but this approach cannot establish a learnable view connection. Attsets [9], Pix2Vox [7, 8], and GARNet [10] employ attention layer to capture inter-view correlations. Although these methods have achieved promising results, they are weak for mining the inter-view relationships because the relevant structure is too simple.

In transformer-based methods, EVolT [11] and LegoFormer [12] compress the number of tokens to represent each view using CNN-backbone network, in order to integrate feature extraction and view feature fusion into a single transformer network. However, they lose a certain representation capability for each view. On the other hand, 3D-RETR [13] handles each view employing the transformer network independently and fuses the features using the pooling method. In fact, it is the same structure that separates encoder and fusion as aforementioned, which ignores multi-view associations. UMIFormer [14] adopts decoupled intra- and inter-view feature extraction to strengthen the correlation between views, whereas it introduces additional modules to estimate inter-view relationships.

Different from the above methods that focus on solving multi-view fusion problems, the optimization-based methods, such as BARF [22], NeRS [23], and FvOR [24], jointly optimize 3D shape and camera pose estimation. They have good performance but sacrifice efficiency.

[25] proposes a transformer network and makes success in natural language processing. In order to migrate this method to process images, [26] divides the 2D attention into two groups based on the height and width dimensions. [27] employs both long-range and short-range methods to construct multiple feature groups from a single image. Recently, ViT [15] which processes an image by splitting it into patches has demonstrated superior performance in many vision tasks. STTN [16] introduces ViT into video inpainting and serves as a pioneering attempt to apply the transformer architecture to address a problem with multi-view inputs. It utilizes full-range attention for all tokens in the spatial-temporal dimension. To mitigate the difficulty to process massive information, DSTT [17] employs the short-range grouping attention. It allows the model to capture the local features while paying less attention to the non-local relationships. Therefore, this grouping strategy is suitable for addressing the problem with multi-input with temporal coherence such as video. However, for our task with ambiguous associations between views, it requires a method to establish long-distance dependency relationships.

The encoder is based on the architecture of ViT [15] and is divided into two stages: an independent branch stage and an interleaving branch stage. The first stage consists of $N_{1}$ standard transformer blocks. The second stage including $N_{2}$ basic unit alternately handles intra-view tokens and inter-view tokens separately. The former still employs standard transformer blocks to parallelize all views, while the latter utilizes transformer blocks with the long-range grouping attention (LGA) to establish communication between views and inter-view feature signatures (IFS) to help distinguish the feature of different views as shown in Figure 2(b). At last, we introduce the similar-token merger [14] to compress the tokens from all branches to the specified size.

Long-Range Grouping Attention. This is a special attention layer for establishing the correlations between different views. Given specific $i$-th view feature $X^{i}\in\mathbb{R}^{P\times D_{e}}$, where $P$ is the number of tokens and $D_{e}$ is the token dimension. We perform uniform and regular sampling based on the length and width of $X^{i}$ to obtain $g$ groups. It is noteworthy that tokens in each group are not adjacent to each other in terms of the original view features. Rewrite $X^{i}$ as $X^{i}=\{x_{1}^{i},\ldots,x_{j}^{i}\}$, $i=1,\ldots,N$, where $x_{j}^{i}$ represents the tokens from $j$-th group $i$-th view, and $N$ represents the amount of input view. As shown in Figure 1(d), the tokens sharing the same color belong to a group, which implies that each group not only includes intra-view tokens but also contains the inter-view tokens from their corresponding positions. After grouping, self-attention operations process the tokens in each group independently. Essentially, this method utilizes the principle of divide-and-conquer to reduce the difficulty of model training. The attention operations for all groups are predication about the relationship of views, but there are certain differences between them. Feature diversity is beneficial for building high-quality representations.

Inter-View Feature Signatures. The multi-view reconstruction task is invariant to permutations, hence the positional encoding we set is the same for all views. It may cause the LGA to neglect the relationship whether some tokens are from the different view. Inspire by [28], we introduce a simple structure that only includes a convolutional layer as IFS to enhance differences between tokens from various views. Given series tokens $x_{j}^{i}$ from $j$-th group $i$-th view, we obtain the token features by a trainable linear projection $\phi$. The $\phi$ shares the linear projection which is used to predict ${\bm{V}}_{j}$ in this transformer block, where $\phi(x_{j}^{i})\in\mathbb{R}^{\frac{P}{g}\times D_{e}}$, and $g$, $P$, $D_{e}$ represent the number of groups, the number of tokens and token dimension. The feature signatures $f_{j}^{i}$ from $j$ group $i$ view is obtained as follow:

Furthermore, we define inter-view feature signatures ${\bm{F}}_{j}=\{f_{j}^{1},\ldots,f_{j}^{N}\}$, $j=1,\ldots,g$, where $N$ represent the number of views. Formally, for each view features $x_{j}^{i}$ from $X^{i}$, the LGA with IFS becomes:

Where ${\bm{Q}}_{j},{\bm{K}}_{j},{\bm{V}}_{j}$ represent queries, keys, values from $X_{j}=\{x_{j}^{1},\dots,x_{j}^{N}\}$ respectively, and $d$ is the channel number of the queries. The attention maps obtained from individual groups in the group dimension are concatenated, thereby completing an inter-view association computation.

In our investigation, the previous transformer-based decoders for voxel reconstruction employ transformer blocks to generate a low-resolution (LR) feature representation (e.g., $4^{3}$) and then upsample to the target size in hasty steps. Because the relatively high-resolution (HR) tensor corresponds to a large number of tokens which is unbearable for attention operations. However, it is difficult to reconstruct details through a rapid upsampling approach on the last few layers of the model. To overcome the limitations, we design a progressive upsampling transformer-based decoder.

Our proposed decoder upsamples the feature map of size $4^{3}$ extracted by the encoder to $32^{3}$ gradually through 4 basic units. Each basic unit includes 1 deconvolution layer and 2 transformer blocks. Convolution and multi-head self-attention and processing are complementary [29]. Deconvolution is better at encoding local information and supports upsampling with relatively low consumption, while the transformer has excellent global representation capabilities. Therefore, we hold that combining the advantages of both can build a powerful model.

The structure of a basic unit can easily handle LR feature inputs in the way shown in Figure 2(c). However, as aforementioned, the transformer block cannot normally deal with a large number of tokens brought by HR features. To this end, we modify it to be an HR basic unit as shown in Figure 2(d).

In the HR basic unit, the adjacent voxels are uniformly grouped and sequentially combined to compress the number of tokens. Given voxel features $V=\{v_{1},\ldots,v_{t}\}$, $V\in\mathbb{R}^{T\times D_{r}}$, where $v_{t}$ is $t$-th voxel token, $T,D_{r}$ represent the number of token numbers and token dimension, respectively. In the resolution of $S\times S\times S$, we define $T=S\times\ S\times S$. Grouping every $I\times I\times I$ adjacent voxel token together, we obtain the voxel features $V\in\mathbb{R}^{T’\times{D_{r}}^{\prime}}$, where $T’=\frac{S}{I}\times\frac{S}{I}\times\frac{S}{I}$, ${D_{r}}^{\prime}=I^{3}\times D_{r}$. As a result, voxel features $V$ are grouped into sizes that can be processed by self-attention while preserving the original feature information in a relatively high resolution (e.g., $16^{3}$ and $32^{3}$). In the end, we take voxel features $V$ as input and apply multi-head self-attention to explore voxel location relationships.

It is worth noting that the attention layer under such a method loses certain concern about the relevance of voxels assigned to the same token. We consider that the convolution layers in the network which focus on the local feature can be used to supplement the lack, so we add a skip connection to combine the features represented by the deconvolution and transformer layer.

Following [13], we use Dice loss [30] as the loss function which it is suitable for high unbalanced voxel occupancy. The Dice loss could be formulated as:

where $p_{i}$ and $g_{i}$ represent the confidence of $i$-th voxel grid on the reconstructed volume and ground truth.

The experiments are performed on ShapeNet dataset [18]. Following [1], we utilize a subset of ShapeNet consisting of 13 categories and 43,783 3D objects for a fair comparison. Additionally, we evaluate our model on real-world data from the Pix3D dataset [19], which includes 2,894 untruncated and unoccluded chair images following [7, 8].

Following the same evaluation methods as the current advance works, we measure the reconstruction quality of our models using the mean Intersection-over-Union (IoU) and F-score$@1\%$ [8]. Higher IoU and F-score$@1\%$ values indicate better performance.

IoU: The mean Intersection-over-Union (IoU) is formulated as:

where $\hat{p}(i,j,k)$ and $p(i,j,k)$ represent the predicted occupancy probability and ground truth at $(i,j,k)$. $\text{I}(\cdot)$ is an indicator function and $t$ denotes a voxelization threshold.

F-Score$@1\%$: Following the same setting as [8], we take F-Score [31] as an extra metric to evaluate the performance of 3D reconstruction results, which can be formulated as

where $\text{P}(d)$ and $\text{R}(d)$ denote the precision and recall for a distance threshold between prediction and ground truth. More formally,

where $[\cdot]$ is the iverson bracket. $\mathcal{R}$ and $\mathcal{G}$ denote the reconstructed and ground truth point clouds, respectively. $n_{\mathcal{R}}$ and $n_{\mathcal{G}}$ are the numbers of points in $\mathcal{R}$ and $\mathcal{G}$. For voxel representation, we generate the object surface by the marching cubes algorithm [32]. Then 8,192 points are sampled from the object surface to compute F-Score between prediction and ground truth. F-Score$@1\%$ indicates the F-Score value when $d$ is set to $1\%$.

We utilize the pre-training model of DeiT-B [33] to initialize our encoder and the cls token and distillation token are removed. Our encoder consists of 12 transformer blocks where $N_{1}=6$ and $N_{2}=3$. In LGA Transformer blocks, the $g$ is defined as 49 in LGA. For our decoder, the $I$ is 2 and 4 in the first and second HR basic unit, respectively. To validate the reconstruction performance of the model, we provide two models with the same structure namely LRGT and LRGT+ which use 3 and 8 views as input during training following [10, 14]. During inference, the models can adapt to an arbitrary number of image inputs. In detail, following [8], we use $224\times 224$ RGB images as input and set the voxelized output to $32\times 32\times 32$. LRGT and LRGT+ are trained by an AdamW optimizer [34] with a $\beta_{1}$ of 0.9 and a $\beta_{2}$ of 0.999. The training batch size is 32 for 110 epochs. The learning rate is set to 0.0001 and sequentially decayed by 0.1 after 60 and 90 epochs. We use a threshold of 0.5 for LRGT and 0.4 for LRGT+ to obtain the occupancy voxel grid. Notably, the experiments in Section 4.5 and 4.6 take 3-view-input during training as an instance.

Quantitative results. We compare LRGT and LRGT+ with the SOTA methods [1, 9, 8, 11, 12, 13, 14, 10] on the ShapeNet dataset and the results are shown in Table 1. Obviously, LRGT significantly outperforms the previous methods in all evaluation metrics for both single-view and multi-view reconstruction. LRGT+ degrades part of the performance for single-view reconstruction while further improving the ability to process multi-view inputs.

Qualitative results. Figure 3 shows two sets of reconstruction examples using different methods according to varying numbers of input views on the test set of the ShapeNet dataset. LRGT and LRGT+ outperform the other methods in restoring the overall shape and fine-grained details of the object. Compared with other methods, our model captures more accurate contours of the airplane and restores the chair details better especially the backrest.

Figure 4 shows the reconstruction quantitative results for the methods using different grouping strategies. The baseline method indicates the model that extracts the feature from views independently while without any inter-view communications. The other control methods are obtained by replacing LGA in our proposed LRGT with the specified attention layer. Note that, the full-range attention here is not used on all transformer blocks, so its performance is in a normal state. Actually, we find in extra experiments that the performance of the model which uses full-range attention in every transformer block is terrible and far worse than the models shown in this figure. It is consistent with our previous analysis. In addition, random-range grouping attention is a special experiment that groups the tokens randomly and evenly.

The model with full-range attention exhibits superior overall performance relative to the baseline. It verifies that establishing inter-view associations is indeed conducive to feature representation. However, the performance of the method using full-range attention tends to be saturated or even degraded as the number of input views gradually increases (especially on 16-20 views). Too many tokens from views bring great inference difficulty to the attention operation. The method with token-range grouping attention is even worse than the baseline because the tokens that establish the association between views are weak in representing the views they located.

The remaining three grouping attention methods have achieved relatively good performance, and these attention layers can save about $98\%$ of the computational complexity compared to the full-range attention layer when facing 20-view input. The method using short-range grouping attention focuses on the local information change between different views. It is difficult to construct non-local correlations between intra-view tokens. Therefore, the multi-view input without temporal coherence in our task is not suitably handled by this method. The random-range grouping attention has the potential to represent the macro feature of each view in the groups, however, it is not stable enough due to random. Only long-range grouping attention can provide reliable macro information in the group of attention operation relatively stably. As a result, the method using long-range grouping attention consistently exhibits superior performance compared to other grouping types whatever the number of view inputs. Even for a large number of view inputs (e.g., 18 to 20 views), it still demonstrates a more substantial performance improvement than other grouping types. Therefore, we consider that the long-range grouping attention is more appropriate for our task.

In Figure 6, we visualize several attention weight maps from different groups in an LGA. Each map represents the importance scores between the tokens in a group. In one map, the horizontal axis represents all tokens within this group and the weights refer to the relationship between tokens from the group across views. The part with higher weight means that gains more attention. Due to the grouping mechanism, the corresponding tokens in various groups have proximity positions in the image and similar semantics. However, the weight distribution of the groups with similar semantic arrangements exhibits significant diversity in different maps. It implies that the features get connection following diverse patterns in distinct groups of LGA. Therefore, this structure has a strong representation ability.

It is crucial to evaluate the ability of the proposed methods to handle real-world data. Therefore, we measure our methods on the Pix3D dataset, which provides single-view reconstruction testing with real-world view images. In detail, following [7, 8], we generate the training set using the data from the category of Chair in ShapeNet and synthesize images [35] with random background from the SUN database [36]. Each object has 60 synthesized images.

We provide two models, LRGT and LRGT^†. In LRGT^†, LGA and IFS are removed and replaced by the original self-attention layer while the other structure is the same as in LRGT. Table 5 shows the performance of LRGT, LRGT^†, and other methods [8, 13, 10, 14]. As LGA and IFS are explicitly designed for multi-view input and Pix3D only provides a single view of each object, the performance of LRGT is slightly lower than LRGT^†. Nevertheless, the performance of LRGT is still comparable to previous methods. LRGT^† outperforms other methods, which shows the excellent performance of our decoder to reconstruct real-world data. The qualitative examples as shown in Figure 7 verify that our method outperforms the others in capturing fine details of chairs.