Flying Bird Object Detection Algorithm in Surveillance Video Based on Motion Information

These methods first use an image object detector to detect video frames as independent images. Then, the unique Spatio-temporal information of video data is used to improve the accuracy of detection results [11, 12, 13, 14]. For example, Seq-NMS [11] uses Intersection over Union (IOU) to find the same object in adjacent frames and then re-scores the object with low confidence to improve the detection score of the object in the video. Literature [12] proposed a complete framework for video object detection based on static image object detection and general object tracking. Firstly, object detection and object tracking techniques are used to generate tubelets object proposal frames. Then, the strong detection boxes in the tubelets are used to enhance the weak detection boxes. Finally, a temporal convolutional network is used to re-score the tubelets. [13] extends the popular still image detection framework to solve the generic object detection problem in videos by fusing temporal and contextual information from tubelets. this method effectively incorporates temporal information into the proposed detection framework by locally propagating detection results between adjacent frames and globally modifying detection confidence along the tubelets generated by the tracking algorithm. Literature [14] generates tubelets by matching Bboxes across time and then takes a similar idea as Literature [13] to correct false and missed detections in the tubelets.

These methods generally require that the detector has better detection performance in some frames or that the object to be detected has rich features in some frames. When the features of the object to be detected are not obvious in any frame, the algorithm will not achieve satisfactory results.

When the features of the object to be detected are not obvious in some frames, in the feature extraction stage, the feature information of other adjacent frames can be aggregated to enhance the features of the object in these frames. Literature [15] proposed a method of using optical flow propagation features, then used the features for video object detection, which improved the speed of video object detection. Literature [16, 17, 18] use optical flow to propagate features and then aggregates the features of the current frame and the propagated features to enhance the expression ability of the current frame to improve the detection accuracy of video objects. Literature [19, 20] use the characteristics of LSTM to realize the aggregation or association of feature information of different frames to realize the detection of video objects. Literature [21] models the semantic and spatio-temporal relationships between candidates within the same frame and between adjacent frames to aggregate and enhance the features of each candidate box. In Literature [22], the Sequence-level Semantic Aggregation (SELSA) method was proposed, which randomly sampled some frames from the whole video and aggregated the object candidates of the current frame with the weight of the distance between the object candidates to enhance the characteristics of the object candidates of the current frame. Literature [23] aggregated the ROI features of the current frame and the most similar ROI features of other frames to obtain the Temporal ROI feature so that the Temporal ROI feature contained the temporal domain information of the object to be detected in the whole video. Literature [24] combined class-aware pixel-level feature fusion and class-aware instance-level feature fusion to improve the detection accuracy of video objects. Literature [25] introduces a multi-level Spatio-temporal feature aggregation framework, which fully uses frame-level, proposition-level, and instance-level information under a unified framework, to improve the accuracy of video object detection. Literature [26, 27] designs a unique external memory to store the features of other frames and then fuses the features of the current frame to complete feature aggregation.

However, the feature aggregation operation of this kind of method is carried out in the intermediate feature layer. First, the feature extraction of each frame image is performed separately, and then the extracted features are aggregated, which has a good effect on the ImageNet dataset [34]. However, in our scene, most of the flying birds, the appearance features on any single frame are not particularly rich, the bird size is small, and the bird features are easily lost when extracting the features of a certain frame image alone. At the same time, looking at consecutive frames makes it more obvious where there are birds, which indicates that the flying bird has complementary feature information on multiple consecutive frames. Therefore, we use ConvLSTM to aggregate the features of the flying bird object on consecutive frames before the input of the model (instead of aggregating at the intermediate feature layer) and use the aggregated features to locate the flying bird object.

Due to the small proportion of pixels and low SNR of flying birds in the monitoring image, there are still many false and missed detections only using the previous detection method. Therefore, based on the above detection, we use the method of object tracking to obtain the MR of the flying bird object and generate the St-Cubes of the flying bird object to improve the SNR. This St-Cube is then used to classify and localize flying bird objects.

After processing the above methods, the detection effect still does not substantially improve. We further analyzed and found that there was a subset of flying birds with slow movements in our data. At this time, if the St-Cube of the flying bird object is still obtained by the above method, the slow moving flying bird object will lack the surrounding environment information, which will lead to the degradation of the flying bird object detection performance in this part. Therefore, the flying bird object’s motion information (motion speed) is considered in this paper. When the motion amount of the flying bird object is lower than a certain threshold, its motion range is expanded, and the environmental information of the flying bird object is increased to improve the identification of the flying bird object.

It has been introduced before that the characteristics of the most flying bird objects are not obvious in a single frame, but they have complementary characteristics in multiple consecutive frames. Therefore, we need to aggregate the features of flying bird objects over multiple consecutive frames.

The recurrent neural network ConvLSTM(structure shown in Fig. 3) contains three gates, namely the input gate, output gate, and forget gate, which is used to control the input and output and what information needs to be forgotten and discarded. At the same time, the input gate and output gate can also be understood as controlling the writing and reading of the memory cell. So ConvLSTM can retain useful information and discard redundant or unimportant information.

The coarse-detection phase captures the suspicious flying bird object, and the input is the whole video image, which has the characteristics of many background interference and redundant information (different frames have many identical backgrounds). According to the characteristics of ConvLSTM structure, it can remove redundant or unimportant information while aggregating the features of the flying bird object on consecutive frames. So, in the coarse-detection stage, we use ConvLSTM to extract and aggregate the features of the flying bird object on consecutive frames. Specifically, given the input $n$ consecutive frames of images ${{{{\bf{X}}_{t}}\in{{\bf{R}}^{\left({\text{H}\times\text{W}\times 3}\right)}}|t=\left({1,2,\cdots,n}\right)}}$ (Where H and W are the height and width of the input image, and $n$ is an odd number), using ConvLSTM network $\mathcal{N}_{\text{ConvLSTM}}$ to extract and aggregate the features of flying bird object on the $n$ consecutive frames, to get spatial-temporal aggregation features ${\bf{H}}_{n}\in{{\bf{R}}^{\left({\text{H}\times\text{W}\times\text{C}}\right)}}$ (Where C is the number of channels) of the flying bird object,

where, when $t=1$, ${\bf{H}}_{0}=\bf{0}$. ${\bf{\Theta}}_{\text{ConvLSTM}}$ is the learnable parameter of the ConvLSTM network. The Spatio-temporal aggregation features ${\bf{H}}_{n}$ of $n$ consecutive frames of images are input into the subsequent classification and positioning module to determine the category and spatial location information of the suspicious flying bird object.

In convolutional neural networks, deeper layers generally have smaller sizes, better global semantic information, and can predict larger objects. The shallower depth layers generally have a larger size, have more delicate spatial information, and can predict smaller objects. However, the large feature layer often does not have a relatively high degree of semantic information, and the small feature layer does not have sufficient spatial positioning information. Therefore, relevant researchers have proposed the structure of FPN [35] to combine the strong semantic information of the small feature layer and the strong spatial positioning information of the large feature layer. However, the researchers of PANet(Path Aggregation Network) [36] found that when FPN transmitted information, there was information loss due to the transfer distance when the information was transmitted to the low-level feature layer. Therefore, path-enhanced FPN, namely PANet structure (see Fig. 4), was proposed. The PANet structure opens up a green channel for low-level information transmission and avoids low-level information loss to a certain extent. In this paper, the PANet structure is used to classify and locate flying bird objects.

The Spatio-temporal aggregation features ${\bf{H}}_{n}$ of $n$ consecutive frames are input into the PANet structure to extract high-level abstract features of flying bird objects ${\bf{F}}_{{\text{FBO}}_{n}}$,

where, ${\bf{\Theta}}_{\text{PAN}}$ is the learnable parameter of the PANet network. The high-level abstract features ${\bf{F}}_{{\text{FBO}}_{n}}$ of flying bird objects can be directly used for classification and localization of flying bird objects.

When the distance between the flying bird object and the surveillance camera is different, the size of the flying bird object is also different, so the flying bird object to be detected has a multi-scale property. According to the multi-scale property of the flying bird object, this paper uses the MultiScale Detection Head (MS-D Head) to detect the suspicious flying bird object.

The objects in the middle frame of $n$ consecutive frames have symmetric contextual information, which can get more accurate results in prediction. Therefore, this paper predicts the suspicious flying bird object in the middle frame of $n$ consecutive frames as the detection result of the Coarse-detection stage. Specifically, the high-level abstract feature ${\bf{F}}_{{\text{FBO}}_{n}}$ of the flying bird object is input into the MS-D Head to obtain the output of the model,

where, ${\bf{\Theta}}_{\text{MS-D}}$ is the learnable parameter of the MS-D Head. Then post-processing operations such as Boxes Decoding and non-maximum suppression were performed on the output of the model to obtain the location of the suspicious flying bird object in the middle frame of $n$ consecutive frames,

where, ${\left\{\cdot\right\}}_{\text{frame}^{\left(\frac{n+1}{2}\right)}}$ means the locations of the flying bird objects in $\left(\frac{n+1}{2}\right)^{\text{th}}$ frames. ${\text{P}_{\text{IDk}}}$ indicates the predicted position of the object with IDk (the object with IDk is taken as an example unless otherwise specified). $\mathbb{F}_{\text{P}}\left(\cdot\right)$ denotes the post-processing method.

From the $\left(\frac{n+1}{2}\right)^{\text{th}}$ frame, there are detection results of the suspicious flying bird object, and we start to track the suspicious flying bird object from the $\left(\frac{n+1}{2}+1\right)^{\text{th}}$ frame. In some cases, the appearance characteristics of flying bird objects are not obvious, so we only use their motion information when tracking them and use a relatively simple SORT [37] object tracking algorithm to track suspicious flying bird objects,

where, $\left\{{\left\{{\text{P}_{\text{IDk}}}\right\}}_{\text{frame}^{\left(i\right)}},{\left\{{\text{P}_{\text{IDk}}}\right\}}_{\text{frame}^{\left(i+1\right)}},\cdots\right\}$ represents the position on consecutive image frames of a suspicious flying bird object with ID number k, and $\mathbb{F}_{\text{track}}\left(\cdot\right)$ represents the SORT object tracking method. After obtaining the position of the suspicious flying bird object on consecutive image frames, we can find the Motion Range (MR) of the suspicious flying bird object on $n$ consecutive frames. Specifically, the minimum circumscribed rectangle ${\text{Rect}}_{\text{IDk}}$ at $n$ positions is calculated according to the position of the same object on $n$ consecutive frames of images,

where, $\mathbb{F}_{\text{MinRect}}\left(\cdot\right)$ denotes the function to find the minimum circumscribed rectangle of $n$ rectangular boxes. For example, to find the minimum circumscribed rectangle $\left[\left(x_{\text{min}},y_{\text{min}}\right),\left(x_{\text{max}},y_{\text{max}}\right)\right]$ (Using the horizontal and vertical coordinates of the top left and bottom right vertices of the rectangle) of $\left\{{box}_{1},\cdots,{box}_{n}\right\}$, the specific calculation method is as follows,

where, $\left[\left(x_{1_{{box}_{n}}},y_{1_{{box}_{n}}}\right),\left(x_{2_{{box}_{n}}},y_{2_{{box}_{n}}}\right)\right]$ denotes the horizontal and vertical coordinates of the upper left and lower right vertices of ${box}_{n}$ in the image. The obtained minimum circumscribed rectangle ${\text{Rect}}_{\text{IDk}}$ is the MR of the flying bird object in $n$ consecutive frames. The diagram of the motion range of the flying bird object on five consecutive frames of images is shown in Fig. 5, which, after cropping, can be used as a St-Cubes for flying bird objects.

We crop the MR of the suspicious flying bird object in $n$ consecutive frames to generate the St-Cubes of the flying bird object. The St-Cube eliminates the interference of other background and negative samples, which can improve the SNR of the flying bird object. However, suppose the flying bird object moves too slowly. In that case, the resulting St-Cubes will lack the necessary environmental information (see Raw St-Cube in Fig. 6), which is not conducive to the detection of flying bird objects. To balance the contradiction between SNR and environmental information, this paper proposes an ASt-Cubes extraction method based on the amount of motion of the flying bird object, which adaptively adjusts the size of the MR of the flying bird object according to the speed of the object’s motion. There are two steps.

Firstly, the amount of motion of the flying bird object over $n$ consecutive frames is calculated. If an object of the same size moves fast on $n$ consecutive frames, its MR is large; otherwise, its MR is small. Therefore, we use the ratio of the area of the MR of the flying bird object on $n$ consecutive frames to the area of the single frame image occupied by the flying bird object to define its motion amount on $n$ consecutive frames,

where, ${\sigma}_{mov}$ is the motion amount, $S\left(\cdot\right)$ represents the function to calculate the area, and ${\text{Obj}}_{\text{IDk}}$ represents the object with ID number k. The area of MR is then the area of the minimum circumscribed rectangle ${\text{Rect}}_{\text{IDk}}$. Since the area occupied by a flying bird object in a single image frame may vary due to its shape changes, and it is not easy to calculate accurately, we use the rectangular area of its bounding box to approximately represent its area in this paper.

Then, according to the amount of motion of the flying bird object, the MR of the flying bird object is adaptively adjusted as the Adaptive MR ${\text{ARect}}_{\text{IDk}}$ of the flying bird object. Specifically, a motion hyperparameter $\gamma$ is introduced. When the amount of motion of the flying bird object is less than $\gamma$, the MR of the flying bird object is expanded to make the amount of motion of the flying bird object reach $\gamma$. Therefore, the Adaptive MR of the flying bird object can be expressed as follows,

The ${\text{ARect}}_{\text{IDk}}$ is used to crop the corresponding $n$ consecutive frames of video image $\left\{{\text{frame}^{\left(1\right)}},\cdots,{\text{frame}^{\left(n\right)}}\right\}$ respectively, and the $n$ frame screenshots obtained are the Adaptive St-Cubes (ASt-Cubes) (${\text{ASC}}_{\text{IDk}}$) of the flying bird object,

The input of the fine-detection model is the ASt-Cubes of the flying bird object extracted earlier. The coarse-detection model may detect multiple suspicious flying bird objects at one time, so there will be multiple ASt-Cubes, and the fine-detection model will detect each ASt-Cubes separately. So it is possible to run a coarse-detection model once and a fine-detection model many times. Therefore, to balance the accuracy and speed, the way of aggregating the Spatio-temporal features of the flying bird object in the fine-inspection stage is used to merge the input of multiple consecutive frames. At the same time, to reduce data redundancy, except for the middle frame, the rest of the frames are input in the form of a grayscale image single channel. Specifically, firstly, grayscale the screenshots of the ASt-Cubes of the flying bird object except for the middle screenshot,

where, $\mathbb{F}_{\text{Gray}}\left(\cdot\right)$ is a function that finds the grayscale of a color image. Then, the processed screenshots of the ASt-Cubes are Concatenate in the channel dimension as the input of the fine-detection stage,

where, the second argument of the $\mathbb{F}_{\text{concat}}$ function indicates that the concatenation operation is performed in the third input dimension (height, width, channel). The length and width of ${\bf{X}}_{{\text{S}}_{\text{IDK}}}$ are equal to the length and width of rectangle ${{\text{ARect}}_{\text{IDk}}}$, and the number of channels is $n+2$, which contains the flying bird object’s features in the ASt-Cubes. It is input into the fine-detection model to classify and locate the flying bird object accurately.

In order to further improve the speed of the whole flying bird object detection process, this paper uses a lightweight U-Shaped Network (USN) (in the experiment, we use MobilenetV2 [38]as the backbone network of the USN) as the feature extraction network of the flying bird object in the fine-detection stage, as shown in Fig. 7.

We feed ${\bf{X}}_{{\text{S}}_{\text{IDK}}}$, which aggregates the ASt-Cubes of the flying bird object, into the LW-USN feature extraction network to obtain the aggregated ASt-Cubes feature ${\bf{F}}_{\text{IDK}}$ of the flying bird Object,

where, ${\bf{\Theta}}_{\text{LW-USN}}$ is the learnable parameter of the LW-USN.

The ASt-Cube of a flying bird object may contain more than one object. Moreover, due to the interference of background and negative samples, the detection accuracy of the coarse-detection model is not satisfactory; there will be false and missed detection. So, the ASt-Cube may contain no object, one object, or multiple objects. Therefore, the detection model in the fine-detection stage should still have the ability of multi-object detection. However, since the ASt-Cubes of flying bird objects are only a small area (relative to the input image) and cannot contain many flying bird objects, the output of the fine-detection model need not be designed with a complex structure. In summary, the paper uses a relatively simple Single Scale Detection Head (SS-D Head) structure as the output structure of the fine-detection model (see Fig. 7). Specifically, ${\bf{F}}_{\text{IDK}}$ is fed into the SS-D Head to obtain the output of the fine-detection model,

where, ${\bf{\Theta}}_{\text{SS-D}}$ is the learnable parameter of the SS-D Head. Then, post-processing operations such as Boxes Decoding and non-maximum suppression are performed on the output to obtain the final detection result of the flying bird object,

where, ${Classes}_{\text{IDK}}$ represents the category of the object in the ASt-Cube ${\text{ASC}}_{\text{IDk}}$ of the flying bird object, and ${Boxes}_{\text{IDK}}$ (in this paper, the position of the object in the middle frame of $n$ consecutive frames is taken as the detection result) is the bounding box of the corresponding object in this region.

Finally, the bounding box of the flying bird object in the ASt-Cubes is mapped to the original video image. That is, the final detection result of the flying bird object is obtained.

To prove that, for the characteristics of flying birds in our task (most flying birds do not have obvious features on a single frame image), it is necessary to perform feature aggregation on continuous video images before input to the model, and we set up a set of comparison experiments. Specifically, we set 3 modes for aggregating features of flying birds. In the first (aggregation mode I), we directly place the ConvLSTM module before the input of the CSPDarkNet53, the backbone network of the YOLOV4 model (i.e., the input of CSPDarkNet53 was originally an image, but now the input is the output of ConvLSTM aggregations of the features of five consecutive frames of images. As shown in Fig. 9b); In the second (aggregation mode II), the ConvLSTM module is embedded between the first CrossStage module and the second CrossStage module of CSPDarkNet53, the backbone network of YOLOV4 model (that is, for each image frame, Need to run to the first CrossStage module of CSPDarkNet53 to perform feature aggregation, as shown in Fig. 9c); In the third (aggregation mode III), the ConvLSTM module is embedded between the second CrossStage module and the third CrossStage module of CSPDarkNet53, the backbone network of YOLOV4 model (that is, for each image frame, Need to run to the second CrossStage module of CSPDarkNet53 to aggregate the features, as shown in Fig. 9d).

Because the bird’s features are not obvious and its size is small in our detection task, the feature is easy to disappear in the process of feature extraction of a single frame image, and the deeper the layer is, the more obvious the feature disappears. Experimental results are shown in TABLE I. Without Fine-Detection, the $\text{AP}_{50}$ of the aggregation mode I is 2.71% higher than aggregation mode II and 28.38% higher than aggregation mode III. The experimental results prove the above point of view, that is, for the flying bird object with unobvious features, the feature aggregation should be performed before the feature extraction as much as possible. At the same time, we can see that without Fine-Detection, the $\text{AP}_{50}$ of aggregation mode III is even 1.83% lower than that of the still image based detection method YOLOV4. This indicates that feature aggregation, when the features of the flying bird object gradually disappear, will affect the feature expression of the flying bird object instead. Therefore, our algorithm adopts the aggregation mode I to aggregate the features of flying bird objects. In addition, for the same aggregation mode, the detection accuracy will be greatly improved after adding the Fine-Detection stage.

We design test experiments with different numbers of consecutive frame inputs to evaluate the impact on the detection accuracy and efficiency of the proposed method. Specifically, there are three consecutive frames of input, five consecutive frames of input, seven consecutive frames of input, etc. Theoretically, with the increase in the number of consecutive frames, the motion information of the flying bird object will be gradually enriched. However, the motion range of the flying bird object will also increase, and the difficulty of feature aggregation will also increase. Therefore, as the number of consecutive frames increases, the detection accuracy of the algorithm should be a process of first rising and then falling. Meanwhile, as the number of consecutive frames increases, the algorithm’s running time will increase accordingly. The algorithm’s detection performance test results are shown in TABLE II (the motion amount parameter ${\sigma}_{mov}$ is set to 4.0). The experimental results show that the algorithm has the fastest running speed but the lowest detection accuracy when input is input for three consecutive frames. The detection accuracy and running time are better when the input is for five consecutive frames. Therefore, five consecutive frames of input are used in our algorithm.

We obtain the ASt-Cubes of different sizes of the flying bird object by setting different motion amount parameter ${\sigma}_{mov}$. If the MR is small, the context background information is less; if the MR is large, the SNR is large. Therefore, different sizes of MRs of the same flying bird object have different effects on the algorithm’s performance. Fig. 10 shows the influence of different motion amount parameter ${\sigma}_{mov}$ on the algorithm’s accuracy.

We can see from Fig. 10 that with the increase of parameter ${\sigma}_{mov}$, the detection precision of the proposed method has a first rise after the fall of a trend. As previously analyzed, when the MR is too small, it lacks contextual information, and when the MR is too large, it is easy to introduce more noise (in an extreme case, when the MR is already consistent with the original input image, the previous processing will be meaningless, because the input of the fine-detection stage is directly the original image. At the same time, because the fine-detection model is relatively simple and the input size is small (160 $\times$ 160), the detection effect is bound to be poor), so whether the MR is too small or too large will affect the accuracy of the algorithm. Through experiments, we find that when the motion amount parameter ${\sigma}_{mov}$ is 4.0, the detection performance of the algorithm is the best.

Through parameter analysis experiments, we conclude that when the number of consecutive input frames is 5, the algorithm can get a good balance between accuracy and speed. When the motion parameter is 4.0, the algorithm’s accuracy reaches the highest. So we suggest the following parameter setting scheme. The number of consecutive input frames is set to 5, and the motion amount parameter ${\sigma}_{mov}$ is set to 4.0.