Revisiting 3D Context Modeling with Supervised Pre-training for Universal Lesion Detection in CT Slices

In this work, we explore to employ 3D convolutions for effective 3D context modeling in the problem of lesion detection from consecutive CT slices (e.g. 9 slices). To advance the time and memory efficiency of normal 3D ResNet, we adopt the Pseudo-3D Residual Network (P3D ResNet) [8] as the prototype of our backbone network. The pseudo-3D convolution simulates $3\times 3\times 3$ convolution with $1\times 3\times 3$ filter on axial-view slices plus $3\times 1\times 1$ filter to build inter-slice connections on adjacent CT slices.

Lesion detection in CT slices aims to predict 2D bounding boxes in a certain slice, thus it requires 2D feature maps corresponding to the target slice for further prediction. Therefore, we need to convert the 3D feature maps to 2D ones for further prediction, meanwhile preserving the precise information of the target CT slice for accurate localization and classification. The designed Modified Pseudo-3D Residual Network (MP3D ResNet) highlights two aspects of modifications to fulfill such demands: 1) Instead of conducting isotropic pooling as in the original P3D ResNet, we neglect pooling operation in the inter-slice dimension. 2) A group transform module is introduced to generate the desired 2D feature maps from the context enhanced 3D features.

Neglecting pooling operation in the inter-slice dimension can help to preserve precise information of the target slice. In the meantime, since the number of input slices (e.g. 9) is rather small, we can get enough receptive field in the inter-slice dimension without downsampling. Regarding 2D feature map conversions, Fang et al. [13] proposed to extract $C$ 2D feature maps ($1\times 1\times H\times W$) corresponding to the center slice and concatenate them to form the converted 2D feature map of size ($C\times H\times W$). However, this method can not fully exploit the 3D context information resided in other adjacent slices.

We, on the other hand, propose a group transform module (GTM) instead to includes all slice’s features to compensate for the information loss. Specifically, we view 3D features ($C\times D\times H\times W$) into 2D ($CD\times H\times W$) and apply a group convolutional layer with the group size of $C$ (every $D$ channel is a group) to fuse all neighboring features to yield the final 2D feature maps ($C\times H\times W$).

Supervised pre-training from natural images has proven to be an effective way for 2D medical image transfer learning[1, 2, 3, 4, 5, 10]. This indicates that using supervised pre-training models from another domain can actually benefit the medical image analysis application. What’s more, compared to self-supervised signals, we believe that supervised labels which carry the semantic information could enable the model to learn semantically invariant and discriminative features more effectively. Therefore, in this section, we aim to develop a method to exploit supervised labels from large-scale 2D natural image object detection dataset (e.g. coco [15]) to pre-train our MP3D FPN.

Previous works [1] have shown that by grouping 3 consecutive CT slices (which is natively 3D data) as a 3-channel RGB image, we can boost the detection performance with Image-Net pre-trained weights, indicating the feasibility of simulating RGB natural image with natively 3D CT slices. This inspires us to reversely decompose the 3 channels of natural RGB images into 3 consecutive CT slices, and train an MP3D FPN with such simulated 3D data. Fig 2 illustrates a comparison of the two correlative strategies. For implementation details, we train the MP3D FPN on COCO dataset for 72 epochs and the final weights are used to initialize MP3D ResNet. To drive the network to learn useful 3D contextual features from inter-slice connections, it is essential to keep the resolution in the inter-slice dimension unchanged for all stages of the backbone.The MP3D detector trained with a slice number of 3 can be used to initialize lesion detectors which takes variable number of slices as network input.

Table 1 presents the comparisons with the previous state-of-the-art (SOTA) methods. Our model surpasses all the SOTA methods on sensitivities at different FPs and [email protected], which includes 3DCE[1], MSB[2], RetinaNet[3], MVP-Net[4] and MULAN[5].

Without using any auxiliary supervision, MP3D FPN outperforms MULAN, the previous SOTA which additionally employs multi task learning and a deeper backbone (DenseNet-121) to improve the detection accuracy, by up to 3.48% on the sensitivity of [email protected]. We re-implement 3DCE with ResNet-50 FPN using the same configuration as our MP3D FPN for fair comparison. Our proposed MP3D achieve a performance gain of 6.05% on [email protected] compared with this 2D convolution based context encoding method, demonstrating the superior 3D context modeling ability of our MP3D backbones. As shown in Table 1, MP3D FPN (248.93 GFLOPS, 45.16 $M$ Params) and MR3D FPN (Modified ResNet 3D, 415.81 GFLOPS, 64.03 $M$ Params) based detector achieve comparable results, but the MP3D based detector consumes much less time and memory. This strongly proves the efficacy and the thrift of our MP3D model.

We perform a number of ablations to probe into our MP3D FPN. The results are shown as follows:

We conducted three groups of experiments to explore effectiveness of the pre-training method.

The pre-trained model takes three slices as input. When training with isotropic pooling, the z-axis degenerates to a single slice after the first two pooling layers, preventing further 3D convolution layers from learning useful 3D contextual information. As shown in Table3, pre-trained weights learned from isotropic pooling gives worse results than the proposed method. This also proves that using decomposed natural image as input can actually helps the 3D model to gain context-encoding ability. Thus the learned weights can potentially be used to boost the performance of other 3D medical image analysis tasks.

As shown in Table4, when both trained for 1x learning schedule (12 epochs), MP3D with pre-trained weights significantly outperforms the one without pre-training, demonstrating faster convergence speed. And it turns out that with 2x learning schedule (24 epochs), model trained with the proposed pre-training weights can achieve comparable results with MP3D model trained from scratch with 6x learning schedule (72 epochs). These results validate the effectiveness of our proposed pre-training scheme.