The Importance of Anti-Aliasing in Tiny Object Detection

Previous approaches to improving the performance of tiny object detectors can be roughly categorized into seven types (Tong and Wu, 2022). In the subsequent paragraphs, representative examples from each category will be reviewed.

Super-resolution techniques play a crucial role in enhancing the fine details and characteristics of tiny targets, enabling standard object detection techniques designed for larger objects to be applied effectively to images that contain tiny targets. For instance, a generative neural network approach can be employed to enhance the resolution of the entire image (Bai et al., 2018). Alternatively, the super-resolution technique can be selectively applied to small regions of interest within the image (Yang et al., 2019b).

Contextual information, obtained from regions of the image surrounding an object, can greatly assist in the detection of that object. Many techniques incorporate contextual cues into their networks. Hu and Ramanan (2017) defined templates that utilized very large receptive fields to capture extensive contextual information to aid in the detection of small faces. Another example is the scale selection pyramid network (SSPNet; Hong et al., 2022). SSPNet encompasses a context attention module (CAM) to generate hierarchical attention heatmaps, allowing the network to prioritize areas that are more likely to contain small objects of interest.

Data augmentation can be used to enhance performance by expanding the training dataset through image transformations. PyramidBox (Tang et al., 2018) exploited the data-anchor-sampling (DAS) technique that reshaped a randomly selected object within the image to match a smaller anchor size. Yu et al. (2020) proposed an efficient augmentation technique named ”scale match” to align the object size distribution differences between general datasets and tiny object detection datasets. The scale match technique involves sampling a size-bin from the histogram of object sizes in the external dataset. To ensure that the range of sampled size-bins does not deviate significantly, the authors also introduced a monotone scale match (MSM) strategy for the sampling in a monotonically increasing or decreasing order. Jiang et al. (2021) introduced an enhanced version of the scale match technique called SM+, which extended it from the image level to the instance level.

Multi-scale representation learning is a crucial and effective strategy employed in the detection of small or tiny objects. This approach is commonly applied in general object detection to handle objects of varying sizes, such as through the feature pyramid network (FPN) proposed by Lin et al. (2017a). Gong et al. (2021) argued that while FPN brought positive impacts, it also introduced negative effects caused by the top-down connections to tiny object detection. To address this, they introduced a statistic-based fusion factor that dynamically adjusted the weight of different layers during feature fusion. Tang et al. (2018) introduced a low-level feature pyramid network (LFPN) that initiated from a middle layer instead of starting the top-down structure from a high-level layer because of the observation that not all high-level semantic features are equally beneficial for smaller targets. Liu et al. (2021b) proposed a feature rescaling and fusion (SFRF) network that incorporated a nonparametric adaptive dense perceiving algorithm (NADPA), which had the capability to automatically select and generate a resized feature map that focused on the high-density distribution of tiny objects.

The anchor mechanism predetermines locations which can efficiently scan the image and match potential object regions where objects are anticipated to be detected. Zhang et al. (2017) maintained consistent anchor density across different scales and proposed a scale compensation anchor matching strategy to ensure that all scales had an adequate number of matched anchors. Deng et al. (2020) ensured proper matching for objects of all sizes by incorporating anchors of varying sizes across multiple layers. Zhang et al. (2018) developed a specialized single-level face detection framework that utilized different dilation rates for anchors with different sizes. Xu et al. (2022) introduced a novel label assignment method to address the challenges related to insufficient positive samples and the disparity between uniformly distributed prior anchors and the Gaussian-distributed receptive field. To handle the severe misalignment between anchor boxes and axis-aligned convolutional features, S²ANet (Han et al., 2021) included a feature alignment module. In addition to anchor-based methods, some point-based anchor-free methods are also designed for tiny object detection. For example, Oriented RepPoints (Wentong et al., 2022) represent objects in a set of sample points that effectively bound the spatial extent of the objects and identify semantically significant local areas.

Training detectors for large objects is relatively straightforward, a significant challenge lies in effectively training detectors for small objects. Luo et al. (2019) enhanced the robustness and overall performance by using multi-scale training images. Similarly, Hu and Ramanan (2017) proposed an efficient approach where separate face detectors are trained with features extracted from multiple layers of a single feature hierarchy.

Optimizing the loss function is a valuable strategy that helps enhance the overall performance of detecting tiny objects. Liu et al. (2021a) introduced a novel feedback-driven loss function leveraging the information from the loss distribution as a feedback signal, enabling the model to be trained in a more balanced manner. SSPNet (Hong et al., 2022) included an attention-based loss function to supervise the different layers for extracting information from different ranges of objects.

Our proposed approach considers a new aspect of tiny object detection: anti-aliasing. It is, therefore, independent of the previous work mentioned. Hence, there is a potential that all these different existing techniques could be enhanced by integrating them with our proposed method.

The low-pass filter is a textbook-style solution to aliasing. Following this idea, several CNN anti-aliasing methods have been proposed (Zhang, 2019; Zou et al., 2020; Li et al., 2020). Zhang (2019) proposed BlurPool which incorporated a Gaussian blur layer before each down-sampling module. Vasconcelos et al. (2021) indicated the order of low-pass filters and convolutions had an impact on the effectiveness of anti-aliasing and updated the residual networks architecture accordingly (Vasconcelos et al., 2020). Zou et al. (2020) introduced an adaptive content-aware low-pass filtering layer (AdaBlurPool) to generate distinct filter weights for each spatial location and channel group. Although these methods have achieved impressive results on other tasks such as image classification, domain generalization, instance segmentation, and semantic segmentation, they do not seem promising for tiny object detection, and the experimental results in Tab. 1 confirm this. Gaussian blur (Zhang, 2019; Vasconcelos et al., 2021) is likely to result in a loss of information about tiny features and make tiny objects less recognizable. Whereas AdaBlurPool (Zou et al., 2020) is not theoretically justified, and the adaptive low-pass filter is generated by learnable group convolutional layers that are not guaranteed to learn an appropriate low-pass filter (even after many iterations and even if initialized to Gaussian filters).

In contrast to the above methods, Li et al. (2020) ensures that low-frequency features are not corrupted by high-frequency artifacts by replacing standard down-sampling layers with a discrete wavelet transform (DWT) and an inverse discrete wavelet transform (IDWT). They implemented these transforms using various orthogonal and biorthogonal discrete wavelets including Haar, Daubechies, and Cohen. Such WaveletPool operations were used to replace standard down-sampling operations in ResNets. The resulting WaveCNet (Li et al., 2020) demonstrated the robustness of WaveletPool layers for general object detection in adversarially-attacked scenes. Here we apply WaveCNet, with slight modification, to tiny object detection.

Wavelet Pooling (Li et al., 2020) depends on the wavelet transform theorem (Mallat, 1989), and is implemented using the DWT and the IDWT.

WaveCNet (Li et al., 2020) replaces all down-samplings in a ResNet with WaveletPool, as described in the preceding section. Max-pooling is directly replaced by WaveletPool. Strided convolution is replaced by a convolution with the stride of 1 followed by a wavelet-pooling, ie.,

where ”Pool” and ”Conv” denote pooling and convolution respectively and $\circ$ denotes ”followed by”.

As discussed in (Vasconcelos et al., 2021), whether to apply a low-pass filter before or after a convolutional layer matters for anti-aliasing. Their experiments show better performance is achieved if the low-pass filter comes after the convolution. Despite this, previous anti-aliasing networks (Vasconcelos et al., 2021; Zou et al., 2020; Li et al., 2020), have not consistently done so, and the order in which convolution and low-pass filtering is performed differs across different branches in ResNet blocks, as shown in Fig. 2. As Fig. 2(a) shows, a standard down-sampling residual block could be simplified as a main path and a skip connection. Fig. 2(b) shows the regular order used in most other anti-aliasing CNNs. The low-pass filter follows convolution in the main path but, these operations occur in the reverse order in the skip connection. Experimental results (shown in Tab. 1) confirm that using a consistent order in both pathways is beneficial for tiny object detection, and hence, we use the configuration shown in Fig. 2(c).

Our previous work showed that using a bottom-heavy version of ResNet50 produced better performance on tiny object detection (Ning et al., 2022). This bottom-heavy backbone, BHResNet50, basically shifts calculations from the top layers to the early layers without introducing additional computational costs. This is done by delaying the down-sampling layers, resulting in a decrease in the number of convolutional layers applied to low-resolution features in deeper layers, and an increase in the number of convolutional layers used in earlier, higher-resolution, layers. Here, we implement a bottom-heavy anti-aliased architecture by replacing all down-sampling processes with WaveletPool in a BHResNet50.

The backbone of an object detector is typically pre-trained on an image classification task as this leads to more rapid and stable training on the detection task (Girshick et al., 2014; Kornblith et al., 2019). Here, two pre-training datasets were employed.

Performance on tiny object detection tasks was evaluated using three standard datasets.

As described in Sec. 2.2, a number of different anti-aliasing methods have been proposed. Tab. 1 compares the performance of these different methods on tiny object detection. The results demonstrate that Wavelet Pooling has superior performance, compared to the alternatives, in this application.

As described in Sec. 3.2, we reverse the order in which convolution and anti-aliased down-sampling are performed in the skip connections in the proposed backbones. The experimental results, shown in Tab. 1, show that this change produces a small increase in performance on tiny object detection for Wavelet Pooling and other, alternative, anti-aliased down-sampling methods. We therefore use the order shown in Fig 2(c) in all subsequent experiments.

To select an appropriate wavelet to use in the anti-aliasing module, the performance produced with different wavelets was tested on TinyPerson. These results were compared to those produced using a standard CNN backbone. These backbones were pre-trained on CIFAR100 to reduce pre-training time. As shown in Tab. 2, Cohen(3,3) wavelet produced the best results for most metrics, both for the ResNet50 and BHResNet50 backbones. Cohen(3,3) was, therefore, selected as the wavelet used in subsequent experiments.

By replacing the standard ResNet50 backbone in the current state-of-the-art method SSPNet with our proposed anti-aliasing backbones, we get better $mAP$ among all intervals of TinyPerson. As shown in Tab. 3, using WaveletPool increases the $mAP_{tiny}$ by 1.31% compared to plain SSPNet while integrating WaveletPool with the bottom-heavy backbone increases performance by an additional 0.1%. Hence, overall the proposed method improves on the previous state-of-the-art result by 1.41% in terms of $mAP_{tiny}$. All networks were trained for 12 epochs with stochastic gradient descent (SGD) as the optimizer. The learning rate was initialized to 0.002/0.003 for ResNet50+WaveletPool and BHResNet50+WaveletPool respectively and decreased by a factor of 0.1 after 8 and 11 epochs.

Table 4 shows that integrating WaveletPool into the ResNet50 backbone, and its bottom-heavy variant, used by TinaFace (Zhu et al., 2020b), the current state-of-the-art method for WiderFace, achieves new state-of-the-art results on all three sub-tasks. Both backbones are trained using SGD optimizer with batch size 16. Training was performed for a total of 630 epochs and the learning rate varied between 3.75$e$-3 and 3.75$e$-5 every 30 epochs following the cosine schedule (Loshchilov and Hutter, 2016).

For the DOTA detection task, we evaluated the proposed backbones incorporating WaveletPool by integrating them with the current state-of-the-art method: Oriented RepPoints (Wentong et al., 2022). The proposed method achieved the best $mAP$, and better $AP$ for the majority of object categories. Both backbones were trained for 12 epochs with SGD as the optimizer. The learning rate was initialized to 0.008 and decreased by a factor of 0.1 after 8 and 11 epochs. The results were produced using the DOTA evaluation server.

Knowledge distillation is a method used in machine learning to transfer knowledge from an accurate model, known as the teacher model, to a simpler model, called the student model (Hinton et al., 2015). The goal is to distill the knowledge learned by the teacher model into the student model, enabling the student model to achieve comparable performance to the teacher model but with reduced computational requirements. Inspired by (Suzuki et al., 2022), we use the original ResNet50 as the teacher and an architecture incorporating WaveletPool (either ResNet50 or BHResNet50) as the student. The student network was trained using a short training schedule which had only ten epochs and the learning rate was reduced by 1/10 at epochs 3, 6 and 9. As can be seen in Tab. 6, using a short schedule alone clearly has a negative impact on the performance of the model. However, when combined with the knowledge-distillation approach, the achieved $mAP$ was similar to that achieved using the original, full, training schedule. This demonstrates that it is possible to train the proposed backbones efficiently which could be helpful for exploring different architectures based on WaveletPool in future research.