A Guide to Image and Video based Small Object Detection using Deep Learning : Case Study of Maritime Surveillance

Data Augmentation. In computer vision, data augmentation is commonly used to address the problem of limited labelled data samples. Its goal is to generate a large, high-quality, and diverse set of training datasets that will enable deep learning models to be more robust and generalizable. The traditional methods of data augmentation can be broadly categorized into: (i) geometric transformations-based, including rotation, scaling, flipping, cropping, padding, translation, affine transformation, etc. (ii) Photometric transformations-based, i.e., changing the color components, which include brightness, contrast, hue, saturation, etc. In addition to these pixel-level adjustments based data augmentation methods, there are several patch-level manipulation methods, such as random erase [zhong2020random], CutOut [devries2017improved], CutMix [yun2019cutmix] and grid mask [chen2020gridmask]. Recent advances in Generative Adversarial Networks (GANs) provide a new avenue for data augmentation [antoniou2017data] by synthesizing realistic training samples of different styles [CycleGAN2017] or even novel unseen classes [wang2018low]. Moreover, Cubuk et al. [cubuk2019autoaugment] proposed a reinforcement learning-based data augmentation method, “AutoAugment”, to automatically search for the optimal augmentation strategy to train a classification model. Various data augmentation techniques have been used with the existing object detection methods, such as the horizontal flipping used with Fast R-CNN [girshick2015fast] and Cascade R-CNN [cai2019cascade], saturation and exposure shifts used in YOLO [redmon2016you] and YOLO9000 [redmon2017yolo9000], and the “Mosaic” strategy proposed with YOLOv4 [bochkovskiy2020yolov4]. Zoph et al. [zoph2020learning] extended AutoAugment [cubuk2019autoaugment] to the object detection task by performing the augmentation operations on the bounding boxes. However, existing object detection methods generally perform worse on small objects, compared to medium or large objects. There are two main reasons: (i) there are much less images containing small objects in the training dataset, leading to a model that is biased towards medium or large objects; (ii) in those images containing small objects, the small object regions are too small, leading to a limited number of matched anchors. This namely decreases the probability of small objects to be detected. To address these problems, Kisantal et al. [kisantal2019augmentation] proposed two data augmentation methods accordingly. (i) An oversampling method was used to increase the number of training samples of small objects. (ii) To increase the number of small objects appearing in a single image, multi-copy-pasting of small objects was used to increase the likelihood of matching anchors with small target objects. Based on the copy-paste augmentation strategy [kisantal2019augmentation], Chen et al. [chen2019rrnet] proposed an adaptive resampling augmentation method, which uses a pre-trained semantic segmentation model to determine suitable image regions for the augmented object pastes. This method effectively addresses the problems of background and scale mismatches when performing random pastes. In order to exploit additional datasets of different object scale distributions to pre-train the network for small object detection, Yu et al. [yu2020scale] proposed a scale match approach to align the scale distributions of the pre-training dataset with that of the target small object dataset. Similarly to the Mosaic strategy [bochkovskiy2020yolov4], Chen et al. [chen2020stitcher] proposed to balance the scale distribution of a training dataset by stitching multiple images of medium- or large-size objects to form a down-scaled collage image. Moreover, a feedback-driven decision paradigm based on the loss statistics of the minority small-scale objects was proposed to guide the image stitching process.

Super Resolution. The limited region of interest (RoI) for small objects results in insufficient feature information for an accurate detection prediction. To address this problem, a straightforward method is to perform super-resolution, namely recovering high-resolution images from their low-resolution counterparts [wang2020deep]. There are typically two types of super-resolution strategies for small object detection: (i) image super-resolution and (ii) feature super-resolution. Haris et al. [haris2021task] proposed to concatenate a super-resolution network prior to a detection network for an end-to-end training. The super-resolution process was also driven by the detection objectives, thus leading to better detection-oriented super-resolved images. Bai et al. [bai2018sod] proposed a multi-task generative adversarial network for small object detection (SOD-MTGAN). More specifically, SOD-MTGAN is composed of: (i) a generator which reconstructs super-resolved RoI images from the small blurred ones, and (ii) a multi-task discriminator to perform detection on the super-resolved RoI images and differentiates real high-resolution RoI images from the fake generated ones. Image super-resolution can help recover details of small objects in an image, thereby resulting in a moderate improvement in detection performance. However, image super-resolution based methods for small object detection suffer from several limitations. Firstly, super-resolving whole images can inevitably enlarge other irrelevant regions, which adversely impact detection performance. Secondly, if super-resolution is only performed on RoI images, object detection on the super-resolved RoI images will largely limit the detection performance due to the lack of context information. This second limitation can be alleviated by performing super-resolution on deep feature maps, which are generated by convolving context. Li et al. [li2017perceptual] proposed a Perceptual GAN to improve small object detection by generating the super-resolved features of small objects that cannot be discriminated from the features of large objects. Similarly, Noh et al. [noh2019better] used GAN to generate super-resolved features for small objects. This was shown to significantly improve the detection performance by providing a direct supervision to learning the super-resolved features of small objects using high-resolution features with appropriate receptive fields. In their article [pang2019jcs], Pang et al. introduced a unified network, called JCS-Net, to integrate the classification and super resolution tasks and to exploit the relationship between large and small scale objects (pedestrians) for recovering the detailed information.
Finally, several other methods perform semi-preprocessing steps to improve detection performance. For example, in [ozge2019power] the authors used the overlapped tiling technique to increase the likelihood of small objects being present in the training stage.

2D-CNN. The majority of deep learning-based methods for detecting small objects rely on CNNs. These object detection methods can typically be categorized into anchor-based or anchor-free methods. Anchor-based methods primarily consists of two types of methods, namely, two-stage methods and one-stage methods (see Section 3). One-stage methods generally have a faster detection speed, while two-stage methods tend to have higher detection performance.

Image Transformer. Several studies have suggested the use of transformers [vaswani2017attention] for detecting objects following Dosovitskiy et. al.’s pioneering work [dosovitskiy2020image]. The Vision Transformer (ViT) was used for the first time in ViT-FRCNN [beal2020toward] to examine the feasibility of transformers for complex object detection tasks. However, the SOD results revealed that the proposed method was not suitable and modifications were necessary to improve the detection performance. Moreover, the proposed method combines transformers and CNNs (i.e., does not merely use transformers). As a way to mitigate the reliance on CNNs and to propose a purely transformer-based object detection technique, You Only Look at One Sequence (YOLOS) was proposed in [fang2021you] to test the transferability of pre-trained transformers from image recognition to object detection. But YOLOS was unable to benefit from multi-scale features and achieved limited performance. With these limitations in mind, [song2021vidt] proposed a method that integrates Vision and Detection Transformers (ViDT), and introduced three major contributions: (i) a new attention mechanism called Reconfigured Attention Module (RAM); (ii) a lightweight encoder-free neck architecture; and (iii) a token matching for knowledge distillation.

Mixed Architecture. The use of both CNNs and transformer architectures has been proposed in various studies. The Most common approach is to first use CNN networks as the backbone and extract several appropriate feature maps. Then these feature maps should be fed into a transformers for decision making. In the early work of transformer-based object detection (OD), Carion et al. [carion2020end] proposed DEtection TRansformer (DETR) using transformers (with both encoder and decoder) on top of CNNs. DETR outperformed CNN-only based SOTA methods, while alleviating the need for complex post-processing steps such as Non-Maximum Suppression (NMS). Considering the computational cost of DETR, [zhen2022deeply] proposed another compact end-to-end variant which represents the large weight matrix in one layer by low order matrices. Additionally, a decoder-only detector ($\text{D}^{2}$ETR) was proposed in [lin2022d] to address complexity. Furthermore, two additional modifications of DETR were introduced in [jiang2021guiding] in order to enhance learning and SOD performance. First, in order to to update the positional information of the queries, a module called Guided Query Position (GQPos) was added to the decoder. Second, the authors proposed Similar Attention (SiA), a new fusion scheme that interpolates the low-resolution attention weight map to generate a high-resolution attention map, since multi-scale feature learning is computationally expensive. This idea was motivated from the fact that the relative positions of the objects is unique across different scales. A CNN-transformer based on deformable attention (following the idea of deformable convolution [dai2017deformable]) and attending to just a small set of sampling locations has been proposed by Zhu et al. [zhu2020deformable], which has the advantage of being trained much faster than DETR (with 10 times fewer training epochs). SOD performance was also improved by adding a multi-scale deformable attention module. Their method was referred to as “Deformable DETR”. Despite the fact that DETR and Deformable DETR only account for spatial information, they are still fast enough for Video SOD. A new method of extracting small-size features, SOF-DETR, has been proposed in [dubey2021improving], together with a normalized inductive bias. In a nutshell, SOF-DETR uses a multi-scale feature representation of the input image. Consequently, the input of the transformer captures richer information (both semantic and geometrical information) that is more suitable for SOD. Pre-training is performed only on the CNN block in DETR and Deformable DETR, but not on the transformer module. This was addressed by [dai2021up], who proposed UP-DETR, which utilizes unsupervised pre-training for a pre-trained CNN backbone. However, since the pre-training of the transformer and CNN is done separately, they are unlikely to perform as well together. In FP-DETR [wang2021fp], the pre-training was thus performed on the encoder module (not the decoder) using ImageNet before fine-tuning the object detection task with a task adaptor. In [wang2022resc], a transformer-based object detection framework was proposed (RESC), which minimizes post-processing steps and the number of hyperparameters. RESC converges faster than DETR. In addition to being lighter, it enables the use of the FPN structure [lin2017feature] to detect small objects.

Multi-Scale Learning. Multi-scale feature learning is one of the most common approaches for SOD, and several architectures have been developed to support it. Amudhan et al. [amudhan2021rfsod] introduced RFSOD, a lightweight single-stage detector that can be used in embedded systems for real time applications. RFSOD’s architecture is similar to that of the YOLO detector, and uses $3\times 3$ and $1\times 1$ convolutions for lightweight detection. By transferring and concatenating information from the earlier layers to the deeper layers, RFSOD increases the spatial resolution of the information in the last layers. This is critical for SOD and the concatenation is performed until that the receptive field reaches the size of $50\times 50$, so that objects of size 32×32 and smaller can be detected. Chalavadi  et al. [chalavadi2022msodanet] proposed mSODANET which consists of three main components: backbone network, Hierarchical Dilated Network (HDN), and Bi-directional Feature Aggregation Module (BFAM). EfficientNet [tan2019efficientnet] was used to fully exploit the visual information contained in input images of varying sizes. Furthermore, the HDN was used to learn the contextual information of objects while the BFAM aims to resolve the network’s limitation of top-down information flow (parallel connections from the last layers to the first layers) with cross-scale connections in order to improve the model efficacy. Fu et al. in [fu2021small] extended the ResNet structure to ResNeXt-RC and proposed IIHNet. IIHNet is a convolution-based network based on three key concepts: (i) information fusion; (ii) information exchange between different resolutions and modules; and (iii) a multi-scale network. Furthermore, [he2021small] proposed a lightweight network known as YOLO-MXANet which uses a powerful backbone based on the MobileNext [zhou2020rethinking] named SA-MobileNeXt, as a mean to incorporate both spatial and channel attention. Along with the addition of another scale from the shallower layers to improve the performance of SOD, the number of parameters was markedly reduced from 61.5 M to 13.8 M. The authors in [qi2022small] proposed a single stage SODNet composed of an adaptively spatial parallel convolution module (ASPConv) and a fast multi-scale fusion module (FMF) to optimize the spatial information extraction and to fuse the spatial and semantic information. By design, FMF preserves both spatial and semantic information. Following the SSD idea, Cui et al. [cui2018mdssd] proposed a Multi-scale Deconvolutional Single Shot Detector (MDSSD), where multiple feature maps at different scales are upsampled to increase the spatial resolution. For better localization of small objects, concatenation is used in [liu2020small], instead of summation in the fusion block to preserve more information across layers.

Context Learning. Objects are not isolated and they usually co-vary with other objects or particular backgrounds, which provides a rich source of contextual associations. For context learning, there are typically two types of approaches: (i) deep CNNs provide an implicit way to model the spatial context for each pixel through the convolution and pooling operations. In order to incorporate the local context information, existing methods generally manually select the surrounding regions and aggregate their features to enhance the target regional feature [li2016attentive, chen2018context]. In order to model the global context information, enlarging the receptive field to cover the whole image and performing global pooling is commonly employed. Besides, Bell et al. [bell2016inside] regarded feature maps as four sequences of feature maps arranged in the four cardinal directions, i.e., right, left, up and down, and proposed to model the global context information by using four recurrent neural networks (RNNs) to process each sequence and concatenating the outputs. To enhance the context learning of deep CNNs, a number of strategies have been developed to capture the multi-scale context [cui2020context, lim2021small] (See Multi-scale Learning in Section 4.1.3). Moreover, an attention mechanism has been used to effectively extract contextual information for object detection [li2016attentive, shen2019indoor]. (ii) Another line of methods involves explicitly modeling the contextual information, such as scene-to-object and object-to-object relationships at the semantic level or in terms of the spatial layout. Fu et al. [fu2020intrinsic] proposed a context reasoning method for small object detection, which models the object-to-object relationships using the semantic features and the spatial geometric information (i.e., location, size, and aspect ratio) of object regions with a graph convolutional network (GCN). Using the learned contextual relations, the regional features were then updated for both classification and regression, resulting in improved performance for detecting small objects. Leng et al. [leng2021realize] proposed to model object-to-object relations and use the reliable object proposals with their pairwise relations to help classify and localize ambiguous object proposals.

Region Proposal. SOD performance of deep networks can be greatly enhanced by higher input image resolution. Using high-resolution data, however, requires considerably more computational power. To mitigate this bottleneck, one approach is to select the most promising regions and discard the rest of the input image. QueryDet was developed by Yang et al. [yang2022querydet] which first localizes small objects roughly, then refers to high resolution feature maps for better adjustment of bounding box coordinates. Bosquet   et al. [bosquet2020stdnet] proposed STDnet which relies on two components: Region Context Network (RCN) and Region Of Interest (ROI) Collection Layer (RCL). As a result of processing only specific areas, high-resolution feature maps are kept in deeper layers, thereby increasing SOD performance. Additionally, in order to improve adaptation, both the number and the size of anchor boxes were learned by k-means in [bosquet2020stdnet]. In [liu2021modified], MdrlEcf was proposed as a way to exploit deep reinforcement learning (DRL) with a new reward function and an efficient attention network added to a CNN for the task of SOD with very high resolution remote sensing images. Based on FastMask [hu2017fastmask], Wilms et al. [wilms2018attentionmask] proposed AttentionMask, a class-agnostic object proposal generation algorithm that is well suited for SOD. AttentionMask is biologically-inspired and includes scale-specific attention maps.

3D-CNN. While 3D-CNN is the easiest tool for integrating temporal and spatial information of video frames, it is rarely used for the task of object detection. In contrast, 3D-CNN has been deeply investigated for 3D object detection [maturana2015voxnet], action recognition [ji20123d], anomaly detection [shin20203d], etc. In contrast, of those limited studies, 3D-CNN was used in [lin2019smoke] as a feature extractor in combination with Faster R-CNN in order to detect and localize smoke.

RNN. The recurrent neural network (RNN) is a type of neural network that processes temporal or time-series data. It has been widely used for video-based visual tasks following the pipeline shown in Fig. 6 (a). Tripathi et al. [tripathi2016context] proposed to use a recurrent neural network to extract the temporal context information, which is subsequently used to compute a regularization loss to better optimize the training of an object detector. Lu et al. [lu2017online] proposed Association LSTM, which is composed of an SSD and an LSTM networks. More specifically, SSD performs object detection on each frame. The features of the detected objects by SSD are stacked and then forwarded to the LSTM. An additional association error loss is applied to the LSTM outputs of two adjacent frames, to enforce the consistency of two neighboring frames in the temporal space. Compared to Association LSTM, which only uses limited motion information between two frames, Xiao et al. [xiao2018video] proposed a spatio-temporal memory network (STMN) to leverage the motion information across multiple frames. STMN is a bi-directional RNN, which is used to process the convolutional features of a sequence of multiple neighboring frames and also transfer the outputs to each frame. Therefore, the spatial and motion information of multiple neighbouring frames is all incorporated to compute the detection prediction for a target frame, thus effectively improving the detection performance. Moreover, to refine feature maps across frames, Liu et al. [liu2018mobile] proposed an inter-weaved recurrent-convolutional network, coined as Bottleneck-LSTM. By using depthwise separable convolutions and bottleneck design principles, Bottleneck-LSTM achieves a real time inference as well as a high detection performance.

Video Transformer. Due to their superior ability to detect long-range correlations, transformers have recently become very popular in object detection. Transformers have been applied to video based SOD to capture long term spatio-temporal dependencies. As described in [he2021end] and [zhou2022transvod], TransVOD is the first end-to-end system for video object detection using spatio-temporal information. TransVOD uses multiple frames of the video as inputs to its spatial transformers, and uses another temporal transformer on top of it. These two transformers can link each object query and memory encoding outputs simultaneously. Two other extensions of TransVOD have been developed, called TransVOD++ and TransVOT Lite. TransVOD++ uses hard query mining (HQM) strategy to mitigate the redundancy of the number of objects and targets. Experiments show that the TransVOD framework can improve the performance of SOD. TransVOD++ is the first to achieve $90\%$ mAP on ImageNet VID dataset. The second extension, TransVOT was designed for real time object detection.

Attention based learning. Multi-scale feature learning poses a challenge to real time object detection due to its increased complexity. This is because all areas in the input data (image/video) are exploited to localize objects. An alternative to reduce time and computational load is to use attention (whether spatially, temporally, or channel-wise) to eliminate irrelevant information and focus on that which is relevant to the object of interest.
For small object detection in maritime environments, Chen et al. [chen2021improved] proposed a single stage method, called ImYOLOv3 which integrates both spatial and channel attention modules (DAM) into a YOLOv3 network in order to better distinguish between ships and backgrounds. Their proposed end-to-end framework was successfully applied to optical remote sensing images. By adjusting receptive fields on three network branches, ImYOLOv3 achieved promising results for large, medium, and small sized objects. Nie et al. [nie2020attention] used both the channel attention modules and the spatial attention modules in a Mask-RCNN model to enhance the information propagation from the lower layers to the top layers. The use of the attention mechanism was shown to significantly improve the detection accuracy of small ship detection. Liu et al. [liu2021attention] used the Convolutional Block Attention Module (CBAM) [woo2018cbam], which sequentially applies channel and spatial attention modules, to refine intermediate features of the object detection network. A similar attention mechanism was also used in [hu2021pag, fu2021improved, dong2021ship, li2021enhanced]. Wang et al. [wang2021ship] used the Squeeze-and-Excitation (SE) attention module [hu2018squeeze] to dynamically perform channel-wise feature re-calibration, leading to an enhanced representational capacity of their detection network and an improved overall detection performance. A similar attention mechanism was also used in [hu2021ship]. Chen et al. [cheng2021robust] proposed a global attention module to adaptively fuse multi-modal features extracted from image and radar data for small floating waste detection [cheng2021flow].

Semantic Segmentation. Smart modifications of the loss functions can result in a better feature representation for maritime small object detection. It was demonstrated in [moosbauer2019benchmark] that multitask (joint) learning, such as segmentation and object detection, can improve the performance of each task. A possible explanation is that due to joint learning, feature representation is no longer task-specific nor over-fitted to the training dataset. Cane et al. [cane2018evaluating] proposed the use of state-of-the-art deep semantic segmentation networks such as ENet [paszke2016enet], ESPNet [mehta2018espnet] and SegNet [badrinarayanan2017segnet] for maritime object detection. As a result of this, the segmentation stream improved greatly while the network needed fewer annotated, labelled images to train. Park et al. [park2022lightweight] proposed a lightweight Mask-RCNN by using an efficient backbone, i.e., MobileNetV2, to jointly perform warship detection and segmentation. To reduce the cost of dense pixel-level annotation, Zust et al. [vzust2022learning] proposed a weakly supervised method to train a semantic segmentation network for maritime obstacle detection.

Tsinghua-Tencent 100K. Table IV reports the detection performance of the state-of-the-art methods on images with small objects, whose number of pixels are in the range of (0,32], in terms of recall, accuracy and F1-score. As shown in Table IV, Liang et al.[liang2018small] achieved the best Recall of 93.0% and a moderate accuracy of 84.0%. In contrast, YOLOv3-Final [wan2021efficient] attained the best accuracy of 91.0% with a recall of 91.0%, leading to the best F1-score of 91.0%.

MS COCO. Table V shows the detection results of deep learning-based methods on MS COCO dataset. For comparison, we report $\text{mAP}^{@0.5}$ and $\text{mAP}^{@[0.5,0.95]}$. Since the comparison was made using different setups, we denote the results of object detection with sizes smaller than $32\times 32$ with normal values, the results of objects with sizes smaller than $16\times 16$ with “+” and the results on a subset of MS COCO including the three classes of stop signs, mice, and fire hydrants with values marked with “*”. As shown, Full Deformable DETR (arXiv20)[zhu2020deformable] achieved the best $\text{mAP}^{@[0.5,0.95]}=34.4$. In general, smaller objects produce poorer results. FPN (CVPR17)[lin2017feature] achieves the best values for both $\text{mAP}^{@0.5}$ and $\text{mAP}^{@[0.5,0.95]}$ for smaller objects, with values of 11.8 and 4.8, respectively. Finally DETR-GQPos-SiA (arXiv21)[jiang2021guiding] achieves the best $\text{mAP}^{@0.5}$ of 24.4 for normal small objects on MS COCO. Table V also shows the results for the MS COCO subset separately. As can be observed, transformer-based deep learning methods currently have the best SOTA results.

USC-GRAD-STDdb. For the evaluation on video sequences, we selected the recently released dataset, USC-GRAD-STDdb to compare existing SOTA methods. Table VI shows the results obtained on this dataset for various metrics of $\text{mAP}^{@0.5}$, $\text{mAP}^{@[0.5,0.95]}$, FPPI, and FPS. By default, the results for this particular dataset are reported for sizes smaller than $16\times 16$. The team who collected the USC-GRAD-STDdb dataset proposed STDnet-ST++ (PR21)[bosquet2021stdnet], which remains the leading technique in terms of average precision. In terms of FPPI, STDnet-bST (EAAI20)[bosquet2020stdnet], another framework proposed by the same team, performs best. Finally, RetinaNet (ICCV17)[lin2017focal] achieves the best results in terms of runtime speed.

UAVDT. As for this video dataset, Table VII shows the results for $\text{mAP}^{@0.5}$, and $\text{mAP}^{@[0.5,0.95]}$. The values are by default for objects smaller than $32\times 32$. However, smaller sizes than $16\times 16$ are indicated by “+”. As for USC-GRAD-STDdb dataset, STDnet-ST++ (PR21)[bosquet2021stdnet] is seen again to be the leading method for generic small object detection task.

TinyPerson. Table VIII shows the detection results obtained (i.e., MR and AP with IoU thresholds set to be 0.25, 0.5, 0.75) for the state-of-the-art methods on the images of tiny and small objects, whose number of pixels are in the range of [2,20] and [20,32], respectively. Recent methods are generally based on two commonly used object detection architectures, i.e., Faster RCNN-FPN and RetinaNet. Among these methods, MSM+ [jiang2021sm+] achieved the best performance for almost all AP results. S-$\alpha$ [gong2021effective] achieved the best results among the methods based on RetinaNet with respect to all MR evaluations. In contrast, MSM [yu2020scale] achieved relatively better results compared to other methods based on Faster RCNN-FPN in terms of all MR scores. Overall, the two-stage detection methods are seen to outperform the one-stage methods on TinyPerson.

Other Maritime Image and Video Datasets. Table IX presents detection results for other maritime datasets and the best results are marked in bold. The Table provides more information and identifies the leading methods for each metric. Figure 8 shows some of the predicted bounding boxes for different datasets and techniques. Generally, it is observed that using general object detection frameworks to detect small objects is challenging, whereas small object specific methods can better locate those objects.