I3DOD: Towards Incremental 3D Object Detection via Prompting

Since the issue of class-incremental learning was raised more than 20 years ago [8], class-incremental learning [9, 10, 11] has received much attention in different fields. Following [12], we classify class-incremental learning into four types: regularisation-based [13, 14], replay-based [15, 16], pseudo-sample generation [17, 18], and architecture-based [19, 20, 21]. To distill knowledge of old tasks, knowledge distillation is introduced into incremental learning by LWF [14]. Recently, DDE [22] demonstrates the importance of causal effect to resist catastrophic forgetting and introduces it to knowledge distillation. In 2D object detection, the early method [23] firstly applies knowledge distillation on the faster-RCNN architecture. Faster ILOD [24] further adopts an adaptive multi-network distillation loss that enhances the accuracy and efficiency of the RPN-based detection method. MVCD [25] extends the application of knowledge distillation to further retain knowledge of old tasks by retaining correlations at the channel, point and instance levels. Class-incremental learning for 3D object detection is still under development. In [6] , SDCoT introduces knowledge distillation into 3D class-incremental object detection to guarantee the performance of model on old classes. However, the above-mentioned methods neglect the object localization information, and assume that all responses produced by old model are reliable for new model.

Prompt-based learning is first proposed in natural language processing to control the model prediction output and save the cost of pre-training. L2P [26] introduces prompt-based learning into class-incremental learning to guide model prediction with less computational cost, as prompt-based learning is able to capture task-specific knowledge with smaller additional parameters. In [27], DualPrompt further improves the performance by adding complementary learning systems to prompt-based learning. However, there still exists difficulties when extends the prompt-based learning into 3D object detection.

According to class-incremental 3D object detection, the model observes a series of class-incremental learning tasks $\{\mathcal{T}^{t}\}_{t=1}^{T}$ and a data stream $\{\mathcal{D}^{t}\}_{t=1}^{T}$, where $T$ represents the total number of tasks. Following SDCoT, the training data $\mathcal{D}^{t}$ and the class set $\mathcal{C}^{t}$ are only available for the $t$-th task, i.e. $\mathcal{D}^{t}\cap(\cup_{i=1}^{t-1}\mathcal{D}^{i})=\emptyset$ and $\mathcal{C}^{t}\cap(\cup_{i=1}^{t-1}\mathcal{C}^{i})=\emptyset$. Given a stream $\{\mathcal{D}^{t},\mathcal{C}^{t}\}_{t=1}^{T}$ in the $t$-th task, we aim to detect 3D objects from all current classes $\mathcal{C}^{t}$ and old classes $\cup_{i=1}^{t-1}\mathcal{C}^{i}$.

We present the overview of our I3DOD in Fig. 2. A core purpose of our method is to tackle catastrophic forgetting of previous knowledge, when the training data $\cup_{i=1}^{t-1}\mathcal{D}^{I}$ is not available for the current task. Specifically, we define the backbone and model in the current task as $\mathcal{F}^{t}$ and $\Phi^{t}$. The old backbone $\mathcal{F}^{t-1}$ and $\Phi^{t-1}$ are learned from the last task. The head of each model is composed of a regression head $\mathcal{H}_{R}$ and a classification head $\mathcal{H}_{C}$. To tackle the forgetting of old classes without affecting the learning of new classes, we propose our solution from two aspects. On the one hand, we propose a prompt guidance block to record the relation between the category semantic information of each class and its the object localization information, which can guide the matching of the two pieces of information to solve catastrophic forgetting. On the other hand, we design a distillation scheme (i.e. $\mathcal{L}_{RFD}$, $\mathcal{L}_{RBD}$ and $\mathcal{L}_{Dis}$) in Fig. 2 to effectively extract knowledge of old model.

In class-incremental 3D object detection, SDCoT [6] introduces a co-teaching distillation to transfer knowledge between two teacher models and a student model. However, this method is very common in 2D vision tasks and neglects the matching relationship between the object localization information and high-level semantic information. As shown in Fig.1, we find that a matching relationship between the two pieces of information of old classes is crucial for the new model to capture the center of object and locate it for object detection. Inspired by L2P [26] and DualPropmt [27], we propose a prompt guidance block to address the above issues from the comprehension of two pieces of information.

As shown in Fig. 2, given an input point cloud $\mathbf{x}_{i}^{t}$, a feature vector $\mathbf{f}_{i}^{t}$ and a subset of the input points $\mathbf{z}_{i}^{t}$ are obtained from the output of $\mathcal{F}^{t}$:

where $\mathbf{x}_{i}^{t}$ and $\mathbf{z}_{i}^{t}\in\mathbb{R}^{3}$, $\mathbf{f}_{i}^{t}\in\mathbb{R}^{M\times D}$.

To build the relationship between the high-level feature $\mathbf{f}_{i}^{t}$ and the center of 3D object, we here introduce our prompt pool $\{\mathbf{p}_{n}\}_{n=1}^{N}$ , where $N$ represents the total prompt quantity and $\mathbf{p}_{i}^{t}\in\mathbb{R}^{D}$. Inspired by a great success of transformer, we introduce a multi-head self-attention module as our prompting function to combine the selected task-share prompts $\{\mathbf{p}_{s}\}_{s=1}^{S}$ and the high-level feature $\mathbf{f}_{i}^{t}$. After obtaining $\{\mathbf{p}_{s}\}_{s=1}^{S}$ and $\mathbf{f}_{i}^{t}$, we feed them into our multi-head prompting self-attention module. Specifically, we obtain $\mathbf{h}_{Q}$, $\mathbf{h}_{K}$, $\mathbf{h}_{V}$ by repeating $\mathbf{f}_{i}^{t}$, where $\mathbf{h}_{Q}$=$\mathbf{h}_{K}$=$\mathbf{h}_{V}$. In addition, we divide each $\mathbf{p}$ to gain $\mathbf{p}_{K}$ and $\mathbf{p}_{V}$, where $\mathbf{p}_{K}$, $\mathbf{p}_{V}\in\mathbb{R}^{S/2\times D}$. We concatenate $\mathbf{p}_{K}$, $\mathbf{p}_{V}$ with $\mathbf{h}_{K}$, $\mathbf{h}_{V}$ and execute parallel self-attention map H times. In the end, we obtain the $h$-th self-attention map $\mathbf{A}_{h}$ as:

where $\mathbf{W}_{k}^{h}$,$\mathbf{W}_{Q}^{h}$,$\mathbf{W}_{V}^{h}$ are the weight matrices of $h$-th self-attention module, and $\sigma$ denotes the softmax function.

We concatenate all $\mathbf{A}_{h}$ and apply the MLP layer and the one-dimensional convolution layer to extract the center information of 3D objects. As mentioned above, we use prompts to learn the relationship between the high-level feature and the object localization information in class-incremental learning, and store them for the next task. The outputs of our prompt guidance block are given as follows:

Overall, we have established the structure of prompt guidance block. By following VoteNet [4], the outputs of prompt guidance block are expected to locate the center of objects. Therefore, for training our prompt guidance block, we compute the center of object as the supervisory label. The loss function is shown as below:

where $\mathbf{c}_{i}^{t}$ represents the ground truth of object center.

In addition, the prompts have learned the matching relationship of categories in the current task. We build a prompt pool to store these prompts and perform the relationship to tackle catastrophic forgetting of old classes in the next task.

In class-incremental 2D or 3D object detection, performing bounding boxes distillation between old and new object detection models is difficult. Given a sample that does not contain old classes, the regression head of old model also generates bounding boxes at low confidence. There is too much noise in the responses of the regression head when applying old model to predict on training data of the current task. Inspired by [7], we can divide all the responses into reliable responses and unreliable responses. Transferring unreliable responses from old model to new model will affect learning new classes, and accelerate catastrophic forgetting of old classes. Therefore, the current method SDCoT [6] cannot adopt the knowledge distillation strategy on bounding boxes.

To tackle the problems above and effectively transfer the 3D knowledge of old model, we propose a reliable dynamic distillation module to dynamically screen out the reliable 3D knowledge from the regression head. As shown in Fig. 2, through the forward calculation of $\Phi^{t}$, we can obtain classification scores $\mathcal{\{}{S}_{i}^{t}\}_{i=1}^{N}$ and bounding boxes $\mathcal{\{}{B}_{i}^{t}\}_{i=1}^{N}$, where $N$ means the total quantity of proposals generated by object proposal generation module. Specifically, we initialize classification scores of $\Phi^{t}$ to our confidence scores for each proposal. Specifically, we select the subset of $\mathcal{\{}{S}_{i}^{t}\}_{i=1}^{N}$ whose top-1 classification score is distributed in old classes. We then compute the average $\mu$ and the standard deviation $\sigma$ of all selected classification scores. The final threshold value $\tau$ is calculated below:

where $\mu=\frac{1}{O}\sum_{i=1}^{O}S_{i}^{t}\cdot\mathbb{I}_{\mathrm{argmax}S_{i}^{t}\in\cup_{i=1}^{t-1}\mathcal{C}^{i}}$, and $\mathbb{I}$ means the indicator function that $\mathbb{I}=1$ when the condition is true; or $\mathbb{I}=0$. The hyperparameter $\zeta$ is designed to control $\tau$.

We obtain bounding boxes $\mathcal{\{}{B}_{i}^{t}\}_{i=1}^{O}$, $\mathcal{\{}{B}_{i}^{t-1}\}_{i=1}^{O}$ of proposals whose confidence score exceeds the threshold $\tau$. These reliable responses can transfer positive 3D knowledge from the teacher model to the student model, which is crucial to prevent catastrophic forgetting. Thus, the proposed reliable dynamic distillation for bounding boxes can be defined as:

Most current CIL distillation methods based on feature [10, 28] neglect the spatial positional relationship in feature space, while further damaging the model plasticity when learning novel 3D classes. They assume that the direct distillation can exert the same effect in the feature space as it does in the output space. However, the distribution of responses in feature space is complex. Directly distilling responses cannot well transfer knowledge form the distribution.

To handle the above problems, we propose the relation feature distillation to capture the responses relation between $\mathbf{f}_{i}^{t}$ and $\mathbf{f}_{i}^{t-1}$. We define our relation feature distillation loss function as:

where $C_{|\mathcal{D}^{t}|}^{2}$ means the total number of paired samples selected from training data $\mathcal{D}^{t}$.

At each training iteration, we compute a supervised loss $\mathcal{L}_{S}$ via following the loss function in VoteNet [4]. The supervised loss of our prompt guidance block ${\mathcal{L}}_{PGB}$ is also included in $\mathcal{L}_{S}$. Overall, the model $\Phi_{i}^{t}$ is optimized as follows:

where $\alpha,\beta,\gamma,\xi$ are hyperparameters in our experiments.

Datasets. By following SDCoT [6], we organize our experiments on SUN RGB-D [29] and ScanNet [30] datasets. We present the details for these two datasets as follows: 1) SUN RGB-D consists of 5,285 training samples for hundreds of object categories and has 5,050 samples to evaluate. In line with the standard evaluation protocol in SDCoT, we use the same 10 categories to report our performance. 2) ScanNet includes 1,201 samples for training and 312 samples for evaluating. Following VoteNet [4], we generate our input point clouds by selecting vertices from the reconstructed meshes and gain the bounding boxes from the point-level labeling. We choose the same 18 categories to start our experiments.

Evaluation metrics. We use mean average precision ([email protected]) and recall in 3D object detection as our evaluation metrics.

For the hyper-parameters in our framework, we define the number of selected prompts $S$ as 10 and $\zeta$ in Eq. 6 as 1.2. The weights of loss functions in Eq. 9 are designed as $\alpha=10$, $\beta=0.8$, $\gamma=1$, $\xi=1$. We adopt the same method in SDCoT to schedule the respective contributions of $\beta$, $\gamma$ and $\xi$. For our prompt guidance block, we randomly initialize prompts as learnable embeddings at the beginning of the first task and adopt $\mathcal{L}_{PGB}$ in Eq. 5 to optimize. After training on the current task, we store these prompts in prompt pool, which are used to initialize prompts in the next task. The adam optimizer with the initial learning rate 0.001 is adopted to train all models. In addition, the adam optimizer is scheduled via the cosine annealing schedule.

Concretely, we design two baselines for class-incremental 3D object detection (i.e. Freeze and add, Fine-tuning in Table. I). The method Freeze and add means that we freeze the model $\Phi^{t-1}$ to initialize the model $\Phi^{t}$. Then, a new classifier based on $\mathcal{C}^{t}$ is added and trained as the only learnable module. Fine-tuning denotes that we fine-tune $\Phi^{t-1}$ (all parameters except the old classifier) with a new classifier on the training data $\mathcal{D}^{t}$. In addition to the two baselines, we compare our I3DOD with SDCoT using the identical class order and task setting.

SUN RGB-D: As shown in Tabs. I–III and Fig. 3, we evaluate the effectiveness of all experiments on SUN RGB-D. In this paper, B means the base task and N denotes the novel task (e.g., 5+5 means that the base task consists of the first five classes and the other five classes belong to the novel task).Our model gains significant improvement over other methods by $0.9\%\sim 22.0\%$ in terms of task average [email protected]. In the setting of 9+1, our I3DOD significantly exceeds the state-of-the-art method SDCoT by $2.9\%$, $8.5\%$ in terms of [email protected] and recall.

ScanNet: We report the performance of all comparison experiments in Tab. II and visualize some evaluation results in Fig. 4. Compared with other methods, our I3DOD achieves solid improvement in all incremental settings. Furthermore, I3DOD outperforms SDCoT by $0.6\%$ in the base task and $4.1\%$ in the novel task. All the experiments show that our I3DOD can tackle catastrophic forgetting from two aspects of category information comprehension and reliable representation distillation.

As presented in Tab. I and Fig. 5, we ablate the effectiveness of each module in our I3DOD. We compare our I3DOD with I3DOD eliminated our proposed modules one by one (i.e., PGB, RDD, RFD). Specifically, we notice that the performance of Ours w/o PGB degrades $0.8\%\sim 2.3\%$ in terms of task average [email protected], which show that our prompt guidance block can perform category information comprehension to gain identical improvement of new and old classes. Furthermore, Ours w/o PGB & RDD is worse by $0.2\%\sim 2.0\%$ in comparison with Ours w/o PGB, and achieves better performance compared with the baseline Ours w/o PGB & RDD & RFD, which reflects our distillation strategy achieves the significant effectiveness.