M&M3D: Multi-Dataset Training and Efficient Network for Multi-view 3D Object Detection

The domain gap that exists between different data sets is an indisputable fact. Different cameras can produce different domains in a 3D dataset, mainly because of differences in the way they capture and represent visual information. Different cameras use different sensors that have unique characteristics and technologies, including sensitivity to light, noise levels, and color representation, which may lead to differences in the visual data they capture. A camera also has a variety of shooting settings, and different settings can cause that camera to capture more and different details, such as variations in lighting, resolution, and other specialized parameters. The level of detail affects the way objects and scenes are presented in the image. Even more, the camera may have different color profiles or color calibrations during data production, which can affect the way colors are rendered in the image. This may result in variations in color accuracy and saturation.

This approach has been successful in 2D visual recognition. Then we expect to borrow this form and hope that it is also a practical approach in scenarios where 3D object detection tasks are challenging. As we mentioned above and in the related work section, some of the best 3D models or network designs, such as BEVFormer [5], have shown very strong performance on the NuScenes dataset alone. But it is very challenging to improve upon the existing results with the limited computational resources I have and it is very challenging to improve on the existing results. That’s why we mention the use of ’Multi-dataset Training’ to improve the network’s performance because I want the network to be able to efficiently learn more information about the 3D image from a new perspective while leveraging the properties of these excellent models, and it can transfer to a new target domain. It is important to note that this approach is currently a very promising direction in 3D, and any theoretically feasible way of pre-data is architecture-independent.

Mixing Data from Different Dataset is a key part of our domain adaptation method. In multi-dataset training, the model can train multiple datasets simultaneously, some of which may be relevant to the target task. In this case, the model can improve its performance on the target task by learning on multiple data sets, thereby achieving the effect of transfer learning. Instead of using a single 3D dataset, I aim to take advantage of the adaptation to the unseen scenes or a new type of camera. Here for my NuScenes-mini and Lyft, and also for possible future research, I named the multi-dataset $D={d_{i},i=0,1...n}$, where $d_{i}$ represents one of the pure 3D datasets I used to mix in with various data distriution. With $d_{i}={(B_{i},CLS_{i}),POS_{i}}$, the key data distributions of each class and objects are represented by annotations with a bounding box set $B_{i}$ with $bbox$ with 3D coordinates, and $CLS_{i}$ with related labels $cls$. Also, $POS_{i}$ is used to represent other fixed 3D information such as BEV map size, and camera instincts information that every dataset uses for data format transfer. It is very obvious that NuScenes-mini and Lyft hold different dataset sizes and formats, this imbalance number of the camera images in each dataset needs an aligned simple mixing method. Therefore, as shown in Fig 3, I shuffled and regrouped the images, keeping the 3D position relationship, and a mixed dataset with six positions images set with the diversity of NuScenes-mini and Lyft, will be used for further training and experiments. Various scenes will be included in each epoch that the network would be forced out of overfit and could be used in the data transfer learning.

Labels for Mixed Dataset is a practical challenge I met trying to solve the domain gap of datasets. Each self-driving dataset uses its own set of labels, corresponding to objects with different semantic hierarchies. For example, the NuScenes dataset combines a collection of object classes, including human, vehicle, etc., with potential subclasses represented in the form human.pedestrain.adult. Lyft’s open dataset annotations will contain simple object classifications, but not more refined classifications. After a brief consideration, the label differences shown in Fig LABEL: only contain object labels, while the hidden subclasses continue to follow the structure of the source dataset. As analyzed before, the unreconciled parts of several datasets will more or less destabilize the training because the datasets do not have the same class nomenclature. Additionally, I would like to have a more unified set of statistical methods.

The FPN outputs are 4 channel with size $6*64*h_{i}*w_{i}$ for the $i^{th}$ channel, where $h_{i},w_{i}=\frac{H}{2^{i+1}},\frac{W}{2^{i+1}}i\in(1,2,3,4)$. Then for each channel, the network upscales every other channel with smaller feature sizes and uses a cat function to add them together. Next, as four features with same feature size $\frac{H}{4},\frac{W}{4}$ but different channels, I introduced four fusion Conv2D with different in channels but same 64 out channels. Until now, the network converts the FPN outputs of different sizes of features to a multi-level feature with size $4*64*\frac{H}{4}*\frac{W}{4}$. To fit the following Transformer and the related 3D information, a final reshape is needed to output the 2D multi-level feature in size $6*64*H*W$. Noted that there are 6 camera images fed to the network at the same batch, forming the 3D scenes.

As mentioned before, the Transformer structure will be used for the bbox prediction work. As the integral query in my Transformer network, I chose to customize the 3D anchor box as the query for the task of 3D object detection. The functions and the class I used here to generate the promoted anchors are from the class ’Anchor3DRangeGenerator’ [22] provided by MMDet3D [mmengine2022].

As shown in Fig 11, it generates the center by evenly distributing the predefined size of anchor frames in the minimum and maximum range according to the size of the image. If we give the pre-provided 3D anchor type $B\in(cx,cy,cz,w,h,l,k)$ in camera format, then the function I named $F$ will give information about all possible 3D anchors in a known camera-aware field of view $P\in(X_{min},X_{max},Y_{min},Y_{max},Z_{min},Z_{max})$, which is subsequently passed through the meta info of the datasets (also the camera intrinsic and extrinsic information to know the camera view range $W,H,D$ and twisted angle $K$) into our common visual bbox $(cx,cy,cz,lx,ly,lz)$.

Two points worth noting, however, are that first, we can only obtain 3D coordinates from 2D images. Secondly, the 3D position will only be added to the 2D multi-level feature (Transformer input), not to the anchor as Query, because the 3D anchor already has a 3D position, so there is no need to add the same content again. This means that we need to do some processing on the initial $P_{2D}$ to get the final $P_{3D}$, and my expression for the 3D position needs to end up in the same format as the 2D multi-level feature, but with the same content as the anchor.

As shown in Fig 12, I decided to start with the pixel points of the 2D image, as the picture pixel points directly represent its position in our ’multi-view images’. Firstly, a rough representation of the position of pixel point $i$ in a set of 6 camera images can be written as $(x_{i},y_{i},j)$, where $j$ means the $j$th of the 6 views from ’FRONT LEFT’, ’FRONT’, ’FRONT RIGHT’, ’BACK LEFT’, ’BACK’, ’BACK RIGHT’. Obviously, this representation is not feasible, because there is no way to represent the 3D information in $xy$ here, and it is not consistent with the previous anchor. Then, referring to the previous anchor frame generation, we express any point in a set of 6 images like this: $(w_{i},h_{i},d_{i},j)$, where $w_{i}\in(W_{min},W_{max})h_{i}\in(H_{min},H_{max})d_{i}\in(D_{min},D_{max})$. Then I further normalize the pixel coordinates in order to make sure that such 3D information also works for different $W,H,D$ and to reduce the amount of computation. Then a pixel point in the $j^{th}$ view can be represented as

, where $w_{i}\in(W_{min},W_{max})h_{i}\in(H_{min},H_{max})d_{i}\in(D_{min},D_{max})$

So now, we get 3D coordinates of size $6*D*H*W$. In order to be consistent with the 2D multi-level feature format, we add a fully connected layer to make the final 3D coordinates $6*64*H*W$.

Although the performance of conv2d has been confirmed by a lot of excellent work, it has spatial invariance when processing features at different locations. This means that Conv2D network cannot directly consider the 3D information of a specific location, which has a very large negative impact on our 3D target detection and 3Dbbox prediction. In addition, Conv2D cannot explicitly express the conversion from image features to bbox.

Now, let me explain how some of the data I customized earlier will enter this Transformer network.

Transformer network contains the input and output that will processed by it including 2D image multi-level features $f_{i}\in F_{2D}$, 3D coordinates $pos_{i}\in PE_{3D}$, anchor box query $q_{i}\in Q_{Anchor}$, $v_{i}\in Value$, and the final result $(bbox_{i},cls_{i})\in(Pre\_bbox,Class)=p(Value)$. The self-attention mechanism in Transformer allows the network to automatically learn the relationship between features and notice features at different locations. Specifically, the network combines the input with sequence $f_{i}^{3D}\in F_{3D}=g(F_{2D},PE_{3D})$ with the query to obtain each feature and each anchor attentional correlation between boxes. In other words, the network will automatically learn the relationship between features, notice features at different locations, and calculate the attention score $Attention(F_{3D},Q_{Anchor},Value)$ between them. When predicting the task, we combine the attention score $Attention(F_{3D},Q_{Anchor},Value)$ and the input $f_{j}$ to get the final prediction result $(bbox_{j},cls_{j})$. The grey networks in Fig 13 are introduced separately in the input and output parts below.

Inputs are 3D features combining the 2D multi-level features and 3D coordinates. The 3D features $f_{i}^{3D}\in F_{3D}$ could be extend as $f_{i}^{3D}\in R^{6*H*W}$ where $i=0,1...63$. Also, the 2D multi-level features are $f_{i}\in F_{2D}=R^{6*H*W}$ where $i=0,1...63$ and 3D coordinates $pos_{i}\in PE_{3D}=R^{6*H*W}$ where $i=0,1...63$. However, as in position part 3.2.3, the original $p_{i}\in PE_{3D}=R^{6*H*W}wherei=0,1...D-1$ where $D$ is the depth in known 3D information, which transposed into a FC layer then we have $pos_{i}$. For the combination of features and positional encoder, I cat them of the same size together to be the 3D features.

Outputs from the network will be sent to another network to predict bbox and class results. In the structural sense of Transformer, Output is actually Query at the next moment. But in my simple Transformer, the output is direct, containing the corresponding 3D features. But for a single predicted data $j$, after the Transformer outputs the output, further processing is required to obtain the predicted bbox and class information from the same set of outputs. As shown in Fig 4, here I have designed two identical ‘Perception + Linear’. Because here the 3D output is converted to 3D bbox and category, so only simple regression and classification functions are needed.

This is a part summarizing other pioneer and excellent work that also uses camera data. Comparing their adjustments on the backbone, and image size, their results show more information beyond my limited research.

Through comparison, it is found that existing work confirms that even on a large-scale data set such as NuScenes v1.0, just replacing a larger backbone and inputting a larger image size has a significant impact on the final 3D object detection score. My work is inspired by it. First, these works proved the feasibility of transferring 2D visual recognition to 3D visual recognition, and then some work confirmed the importance of 3D spatial features such as BEV for 3D object recognition, especially models trained using only multi-view camera data. This is also the main reason why my network can process efficiently using only multi-view camera 2D images and BEV maps, combined with the modified Transformer structure.

Table 2 and Table 3 show my network prediction results with multi-dataset training. Although Lyft is not a popular research dataset, I still want to use the predictions on Lyft to show the impacts form my new network design.

In Tabel 4, we can see that for the two test results of the NuScenes-mini part, the changes in NuScenes-mini training and multi-dataset training are very weak. This is because my multi-dataset itself is Contains all NuScenes-mini. This small change is not surprising, because this part is mainly used for the transfer capability of training models, facing data and domains like Lyft that are several times larger than NuScenes-mini. Looking at it this way, the improvement brought by multi-dataset training technology is considerable. The test data on Lyft shows that NDS increased by 0.25 (0.0012$\rightarrow$0.2541), and mAP-6 increased by 0.16 (0.04$\rightarrow$0.1658). mAP increased by 0.10 (0.024$\rightarrow$0.1044). The main increase comes from mAP-6.

For the Transformer structure I designed, the performance improvement it provides is very significant, even NuScenes-mini’s NDS (0.3605$\rightarrow$0.4637), mAP-6 (0.3620$\rightarrow$0.4370) and mAP (0.3284$\ rightarrow$0.3767) exceeds the impact of multi-dataste training. Lyft’s improvement here is also amazing. However, there is no way to further verify whether the changes in Lyft’s NDS (0.2541$\rightarrow$0.3677), mAP-6 (0.2658$\rightarrow$0.2577) and mAP (0.1044$\rightarrow$0.2046) come from changes in the network structure or the training data Increase and diversify.

My lightweight model has demonstrated high efficiency, and I think it has the potential to become a cornerstone choice for handling 3D object detection tasks in the future. The computational efficiency of the model is a key feature. By adopting the network architecture of multi-layer feature and Transformer models and efficient algorithms, my model can be trained and inferred with only one GTX2080Ti (12GB). This not only reduces hardware requirements but also shortens training time, which will not exceed 8 hours at a time. This facilitates rapid iteration and model improvement. Second, we have made significant progress in resource efficiency. The entire training and testing requires no more than 100GB of memory, which means it can run in constrained hardware environments and does not put unnecessary pressure on system resources.