P2P-Loc: Point to Point Tiny Person Localization

Fully Supervised Methods. General object detection models object as a bounding box. As instance-level annotation, a bounding box is created to render the center and scale of the object. Under this setting, more and more fully supervised detectors[14, 15, 16, 17, 18, 19, 20, 21, 22] are expected to represent the object accurately. However, such supervision form would consume huge workforce physical resources, and it takes much time and heavy workload to annotate the object at the instance level.

Weakly Supervised Methods. To lower the annotation cost and make full use of massive network data, the weakly supervised object detection (WSOD) [23, 24, 25, 26, 27] trains a detector when only image-level labels are available, which will cause WSOD to focus only on local areas and lack the ability to distinguish instances due to the lack of instance-level constraints. Different from WSOD, weakly supervised object localization utilizes the activation map from the last convolution layer to generate semantic-aware localization maps for object bounding-box estimation [28].

Human Keypoint Detection. Human keypoint detection, also known as human posture estimation, aims to locate the position of joint human points accurately. The COCO dataset [30] contains over 200, 000 images and 250, 000 person instances labeled with 17 key points. During inference, the pose detector predicts the position of every visible keypoint. Similar to COCO, the Human3.6M [31] is one of the most extensive 3D human pose benchmarks. With this benchmark, the detector needs to predict the position of each keypoint in the 3D coordinate system. Three-point annotation form datasets used in [32] are intended to locate the object accurately without considering other redundant information or reduce annotation burden by using the form of the center point [33].

Object Localization. A new task [32] has been proposed to estimate object locations without annotated bounding boxes. In object localization, a true positive is counted when the distance between the estimated location and a ground truth point is less than d, and vice versa. Then the evaluation results of precision and recall can be calculated. In addition, several evaluation metrics are adopted to indicate if the number of location points is incorrect, such as Mean Absolute Error (MAE), Root Mean Squared Error(RMSE), and Mean Absolute Percent Error (MAPE) [34].

Formulation of Semantic Parts. The image set for training is $I=\{I_{1},I_{2},...I_{N}\}$, coarse points sets are defined as $A=\{A_{1},A_{2},...A_{N}\}$, in which $N$ is the number of images. $A_{i}=\{a_{i1},a_{i2},...a_{iM_{i}}\}$. $a_{ij}=(x,y)$ is the 2d coordinate of the annotated coarse point of the $j$-th object, and $M_{i}$ is the number of objects in the image $I_{i}$. The set of all points in $I$ is $\omega$ that is defined as $\Omega=\mathop{\cup}\limits_{i=1,2..N}\{a|a\ is\ the\ point\ in\ I_{i}\}=\{\omega_{1},\omega_{2},..\omega_{h},..\}$. Whether $\omega_{h}$ is annotated or not is denoted by $l_{h}$; $l_{h}=1$ means $\omega_{h}$ is annotated, whereas $l_{h}=0$ means it is not. Accordingly, the annotation set can be denoted as $L=\{l_{1},l_{2},..l_{h},..\}$.

Objective Function. The set of annotated points in $S_{t}$ is represented as $S_{t}^{+}$, and the annotated frequency of the semantic part $S_{t}$ is defined as $Q(S_{t})$.

Statistic Semantic Points. For a point $\omega_{h}\in S_{t}$, the learning objective is to minimize the distance between the estimator $E(\omega_{h};\theta)$ and the probability $Q(S_{t})$ (instead of hard supervision, 1 or 0). To achieve a high score predicted by $E(a;\theta^{*})$, there are two constraints: (1) High $Q(S_{t})$; (2) $S_{t}$ is semantically distinguishable from points of other categories and background ($S_{0}$). Such as SSPs.

TinyPerson. TinyPerson is a tiny object detection dataset collected from high-quality videos and pictures. It contains 72,651 annotated low-resolution human objects in 1,610 images, and most annotated objects in TinyPerson are smaller than $32\times 32$ pixels. In the process of training and inference, the sub-image cut from the original image is used as input. In order to avoid incomplete objects caused by cutting, there is a constant overlap between adjacent sub-images. Then, the NMS strategy is used to fuse the same results of an image.

Visdrone-Person. As a large-scale dataset captured by UAV, Visdrone contains four tracks: (1) Image object detection, (2) Video object detection, (3) Single object tracking, and (4) Multi-object tracking. The experiments in this paper are conducted on the image object detection track of Visdrone. Considering the application scenarios of localization, we mainly focus on the object with a relatively small size. Therefore, the images containing human beings are used to construct a new human detection dataset named Visdrone-Person, and only the labels of pedestrians and persons are applied in this dataset. Visdrone-Person contains 10,209 images, with 6,471 images used for training, 548 for validation, and 3,190 for testing. The same cutting strategy used by TinyPerson is employed to get the sub-images with the appropriate size.

Uniform Distribution. Supposing manual annotating workers do not have any preference when labeling objects, the position of an annotated point follows the uniform distribution.

Rectified Gaussian Distribution (RG). Mostly, manual annotation does have a preference(e.g., mainly on the head or body). According to the large number theorem, it is reasonable to assume that the annotation labeled by a man is like a Gaussian distribution. However, Gaussian distribution has two problems when modelling the position distribution of labeled points. One is how to determine the variance of a Gaussian distribution. The other is that the range of the labeled points is bounded ($[-0.5,0.5]$), while the Gaussian is unbounded. Therefore, the RG distribution ($RG(x^{\prime},y^{\prime};\mu,\sigma)$) is adopted to handle these problems.

Implementation. As a baseline model, RepPoints [36] is suitable for coarse point supervised object localization. This comprehensive framework consists of one estimator and one locator. To ensure that the proportions of positive and negative samples in training are appropriate, we regard some points around the labeled points as positive samples during label assigning in training. The network outputs the classification results and the position offset of each point. In practice, simply treat the input point as a pseudo box with a fixed length and width centered on the input point. Then, for self-refinement, the pseudo box is regarded as the supervision to train an estimator, after which the refined pseudo bounding box is fed to the locator. Finally, predicted bounding boxes produced by the locator are converted to the localization points in the evaluation.

Setting. ResNet-50 is used as a backbone network. The training epoch is set to 12. The initial learning rate is 0.01, and it decreases with the factor 0.1 in the 8^th and 11^th epoch. We choose Faster RCNN with FPN and Sparse RCNN[37] for comparison.

Metric. We adopt the Average Precision (AP) as the metric. According to the bounding-box size, the scale interval is divided into tiny[2,20], small[20,32], normal[32,+$\infty$], and all[2,+$\infty$]. In point evaluation, the point-box distance, instead of IOU, is used as the matching criterion. Specifically, the threshold of point-box distance $\tau$ is set as 1. It means that as long as the predicted point falls within an unmatched ground truth, the match between the predicted point and the ground truth is successful.

Decline of Model Confusion. As mentioned in Sec. III, the uncertainty of annotation semantics limits the localization performance during the self-refinement. To quantitatively analyze this uncertainty, we collect the refined points in each iteration and calculate their relative positions $(x^{\prime},y^{\prime})$ within the object’s bounding box. The distribution is defined as Eq (10):

in which $|x^{\prime}|\leq 0.5$ and $|y^{\prime}|\leq 0.5$.

Reduction of False Positives. Fig. 7 shows that the self-refinement significantly reduces the number of false positives, leading to the improvement of the network’s discriminative ability.

Analysis of Upper Bound. As shown in Tab. VII, we perform experiments based on VisDrone-Person with RepPoints as both the estimator and the locator. The real box means that precision bounding box annotations are used as the supervision information during training, and that the center point of the box was adopted for point evaluation. Box center represents using ($x_{c}$, $y_{c}$) as coarse point annotation during training, and it is the center of the original bounding box annotations. For box head, box foot, and box corner, we adopt ($x_{c}$, $y_{c}-h/4$), ($x_{c}$, $y_{c}+h/4$), and ($x_{c}-w/4$, $y_{c}-h/4$) respectively as point annotations.

Locators. In Tab. IV, self-refinement improves $AP^{all}_{1.0}$ and $AP^{normal}_{1.0}$ by 8.82 and 42.9 points respectively under RP+RP conditions. Meanwhile, $AP^{all}_{1.0}$ and $AP^{normal}_{1.0}$ are increased by 13.2 and 24.2 points respectively over RP+SR.

Estimators. It is shown in Tab. V that even with different estimators, self-refinement improves $AP^{all}_{1.0}$ by 20.27 points and $AP^{small}_{1.0}$ by 32.28 points with FR+FR. Our self-refinement algorithm is compatible with different frameworks. When different detectors are used as estimators or locators, the performance under various scales has almost been improved.

Dataset. As shown in Tab. IV and Tab. V, the performance of the self-refinement showed 8.2 and 20.3 points increase on VisDrone-Person and TinyPerson, respectively.

Number of Iteration. The results in Tab. VI reveal that the performance significantly improves as the number of iterations increases. Then, $AP_{1.0}^{all}$ gains 30.6, 16.87 and 7.33 points in the initial pseudo box size setting with 8*8, 16*16 and 32*32 pixels, respectively. Alternatively stated, the greater the number of iterations is, the slower the performance grows.

Pseudo Box Size. Tab. VI also depicts the effect of different initial pseudo box sizes. Increasing the pseudo box size from 8*8 pixels to 32*32 pixels shows that a larger pseudo box size achieves a better performance. However, self-refinement with more iterations drives a performance boost.

Annotation Initialization. In Tab. I, the results show that the initial distribution of generated points affects performance. Manual annotation is more consistent with the RG distribution. The improvement brought by self-refinement is greater when encountering coarser annotation. In addition, the performance becomes relatively stable and insensitive to the initial annotation distribution through self-refinement.