PIoU Loss: Towards Accurate Oriented
Object Detection in Complex Environments

Object detection is a fundamental task in computer vision and many detectors [34, 25, 21, 17] using convolutional neural networks have been proposed in recent years. In spite of their state-of-the-art performance, those detectors have inherent limitations on rotated and densely crowded objects. For example, bounding boxes (BB) of a rotated or perspective-transformed objects usually contain a significant amount of background that could mislead the classifiers. When bounding boxes have high overlapping areas, it is difficult to separate the densely crowded objects. Because of these limitations, researchers have extended existing detectors with oriented bounding boxes (OBB). In particular, as opposed to the BB which is denoted by $(c_{x},c_{y},w,h)$, an OBB is composed by $(c_{x},c_{y},w,h,\theta)$ where $(c_{x},c_{y})$, $(w,h)$ and $\theta$ are the center point, size and rotation of an OBB, respectively. As a result, OBBs can compactly enclose the target object so that rotated and densely crowded objects can be better detected and classified.

Existing OBB-based approaches are mostly built on anchor-based frameworks by introducing an additional angle dimension optimized by a distance loss [24, 18, 19, 6, 41, 43] on the parameter tuple $(c_{x},c_{y},w,h,\theta)$. While OBB has been primarily used for simple rotated target detection in aerial images [18, 50, 31, 26, 23, 1, 39], the detection performance in more complex and close-up environments is limited. One of the reasons is that the distance loss in those approaches, e.g. SmoothL1 Loss [34], mainly focus on minimizing the angle error rather than global IoU. As a result, it is insensitive to targets with high aspect ratios. An intuitive explanation is that object parts far from the center $(c_{x},c_{y})$ are not properly enclosed even though the angle distance may be small. For example, [19, 6] employ a regression branch to extract rotation-sensitive features and thereby the angle error of the OBB can be modelled in using a transformer. However, as shown in Figure 1(a), the IoU of predicted boxes (green) and that of the ground truth (red) are very different while their losses are the same.

To solve the problem above, we introduce a novel loss function, named Pixels-IoU (PIoU) Loss, to increase both the angle and IoU accuracy for OBB regression. In particular, as shown in Figure 1(b), the PIoU loss directly reflects the IoU and its local optimum compared to standard distance loss. The rationale behind this is that the IoU loss normally achieves better performance than the distance loss [45, 35]. However, the IoU calculation between OBBs is more complex than BBs since the shape of intersecting OBBs could be any polygon of less than eight sides. For this reason, the PIoU, a continuous and derivable function, is proposed to jointly correlate the five parameters of OBB for checking the position (inside or outside IoU) and the contribution of each pixel. The PIoU loss can be easily calculated by accumulating the contribution of interior overlapping pixels. To demonstrate its effectiveness, the PIoU loss is evaluated on both anchor-based and anchor-free frameworks in the experiments.

To overcome the limitations of existing OBB-based approaches, we encourage the community to adopt more robust OBB detectors in a shift from conventional aerial imagery to more complex domains. We collected a new benchmark dataset, Retail50K, to reflect the challenges of detecting oriented targets with high aspect ratios, heavy occlusions, and complex backgrounds. Experiments show that the proposed frameworks with PIoU loss not only have promising performances on aerial images, but they can also effectively handle new challenges in Retail50K.

The contributions of this work are summarized as follows: (1) We propose a novel loss function, PIoU loss, to improve the performance of oriented object detection in highly challenging conditions such as high aspect ratios and complex backgrounds. (2) We introduce a new dataset, Retail50K, to spur the computer vision community towards innovating and adapting existing OBB detectors to cope with more complex environments. (3) Our experiments demonstrate that the proposed PIoU loss can effectively improve the performances for both anchor-based and anchor-free OBB detectors in different datasets.

In this section, we present in detail the PIoU Loss. For a given OBB $\bm{b}$ encoded by $(c_{x},c_{y},w,h,\theta)$, an ideal loss function should effectively guide the network to maximize the IoU and thereby the error of $\bm{b}$ can be minimized. Towards this goal, we first explain the IoU method. Generally speaking, an IoU function should accurately compute the area of an OBB as well as its intersection with another box. Since OBB and the intersection area are constructed by pixels in image space, their areas are approximated by the number of interior pixels. Specifically, as shown in Figure 3, $\bm{t}_{i,j}$ (the purple point) is the intersection point between the mid-vertical line and its perpendicular line to pixel $\bm{p}_{i,j}$ (the green point). As a result, a triangle is constructed by OBB center $\bm{c}$ (the red point), $\bm{p}_{i,j}$ and $\bm{t}_{i,j}$. The length of each triangle side is denoted by $d_{i,j}^{w}$, $d_{i,j}^{h}$ and $d_{i,j}$. To judge the relative location (inside or outside) between $\bm{p}_{i,j}$ and $\bm{b}$, we define the binary constraints as follows:

where $d_{ij}$ denotes the L2-norm distance between pixel $(i,j)$ and OBB center $(c_{x},c_{y})$, $d_{w}$ and $d_{h}$ denotes the distance $d$ along horizontal and vertical direction respectively:

Let $B_{\bm{b},\bm{b}^{\prime}}$ denotes the smallest horizontal bounding box that covers both $\bm{b}$ and $\bm{b}^{\prime}$. We can then compute the intersection area $S_{\bm{b}\cap\bm{b}^{\prime}}$ and union area $S_{\bm{b}\cup\bm{b}^{\prime}}$ between two OBBs $\bm{b}$ and $\bm{b}^{\prime}$ using the statistics of all pixels in $B_{\bm{b},\bm{b}^{\prime}}$:

The final IoU of $\bm{b}$ and $\bm{b}^{\prime}$ can be calculated by dividing $S_{\bm{b}\cap\bm{b}^{\prime}}$ and $S_{\bm{b}\cup\bm{b}^{\prime}}$. However, we observe that Eq. 1 is not a continuous and differentiable function. As a result, back propagation (BP) cannot utilize an IoU-based loss for training. To solve this problem, we approximate Eq. 1 as $F(\bm{p}_{i,j}|\bm{b})$ taking on the product of two kernels:

Particularly, the kernel function $K(d,s)$ is calculated by:

where $k$ is an adjustable factor to control the sensitivity of the target pixel $\bm{p}_{i,j}$. The key idea of Eq. 8 is to obtain the contribution of pixel $\bm{p}_{i,j}$ using the kernel function in Eq. 9. Since the employed kernel is calculated by the relative position (distance and angle of the triangle in Figure 3) between $\bm{p}_{i,j}$ and $\bm{b}$, the intersection area $S_{\bm{b}\cap\bm{b}^{\prime}}$ and union area $S_{\bm{b}\cup\bm{b}^{\prime}}$ are inherently sensitive to both OBB rotation and size. In Figure 3, we find that $F(\bm{p}_{i,j}|\bm{b})$ is continuous and differentiable. More importantly, it functions similarly to the characteristics of Eq. 1 such that $F(\bm{p}_{i,j}|\bm{b})$ is close to 1.0 when the pixel $\bm{p}_{i,j}$ is inside and otherwise when $F(\bm{p}_{i,j}|\bm{b})\sim 0$. Following Eq. 8, the intersection area $S_{\bm{b}\cap\bm{b}^{\prime}}$ and union area $S_{\bm{b}\cup\bm{b}^{\prime}}$ between $\bm{b}$ and $\bm{b}^{\prime}$ are approximated by:

In practice, to reduce the computational complexity of Eq. 11, $S_{\bm{b}\cup\bm{b}^{\prime}}$ can be approximated by a simpler form:

where $(w,h)$ and $(w^{\prime},h^{\prime})$ are the size of OBBs $\bm{b}$ and $\bm{b}^{\prime}$, respectively. Our experiment in Section 5.2 shows that Eq. 12 can effectively reduce the complexity of Eq. 10 while preserving the overall detection performance. With these terms, our proposed Pixels-IoU $(PIoU)$ is computed as:

Let $\bm{b}$ denotes the predicted box and $\bm{b}^{\prime}$ denotes the ground-truth box. A pair $(\bm{b},\bm{b}^{\prime})$ is regarded as positive if the predicted box $\bm{b}$ is based on a positive anchor and $\bm{b}^{\prime}$ is the matched ground-truth box (an anchor is matched with a ground-truth box if the IoU between them is larger them 0.5). We use $M$ to denote the set of all positive pairs. With the goal to maximize the PIoU between $\bm{b}$ and $\bm{b}^{\prime}$, the proposed PIoU Loss is calculated by:

Theoretically, Eq. 14 still works if there is no intersection between $\bm{b}$ and $\bm{b}^{\prime}$. This is because $PIoU(\bm{b},\bm{b}^{\prime})>0$ based on Eq. 9 and the gradients still exist in this case. Moreover, the proposed PIoU also works for horizontal bounding box regression. Specifically, we can simply set $\theta=0$ in Eq. 5 for this purpose. In Section 5, we experimentally validate the usability of PIoU for horizontal bounding box regression.

OBB detectors have been actively studied for many years and several datasets with such annotations have been proposed [39, 1, 23, 26, 31, 50, 18, 12]. As shown in Table 1, most of them only focused on aerial images (Figure 4 (a),(b)) while a few are annotated based on existing datasets such as MSCOCO [22], PASCAL VOC [7] and ImageNet [5]. These datasets are important to evaluate the detection performance with simple backgrounds and low aspect ratios. For example, aerial images are typically gray and texture-less. The statistics in [39] shows that most datasets of aerial images have a wide range of aspect ratios, but around 90% of these ratios are distributed between 1:1 and 1:4, and very few images contain OBBs with aspect ratios larger than 1:5. Moreover, aspect ratios of OBBs on PASCAL VOC are mostly close to square (1:1). As a result, it is hard to assess the capability of detectors on objects with high aspect ratios and complex backgrounds using existing datasets. Motivated by this, we introduce a new dataset, namely Retail50K, to advance the research of detection of rotated objects in complex environments. We intend to make this publicly available to the community (https://github.com/clobotics/piou).

Figure 4 (c) illustrates a sample image from Retail50K dataset. Retail50K is a collection of 47,000 images from different supermarkets. Annotations on those images are the layer edges of shelves, fridges and displays. We focus on such retail environments for three reasons: (1) Complex background. Shelves and fridges are tightly filled with many different items with a wide variety of colours and textures. Moreover, layer edges are normally occluded by price tags and sale tags. Based on our statistics, the mean occlusion is around 37.5%. It is even more challenging that the appearance of price tags are different in different supermarkets. (2) High aspect ratio. Aspect ratio is one of the essential factors for anchor-based models [33]. Bounding boxes in Retail50K dataset not only have large variety in degrees of orientation, but also a wide range of aspect ratios. In particular, the majority of annotations in Retail50K are with high aspect ratios. Therefore, this dataset represents a good combination of challenges that is precisely the type we find in complex retail environments.(3) Useful in practice. The trained model based on Retail50K can be used for many applications in retail scenarios such as shelf retail tag detection, automatic shelf demarcation, shelf layer and image yaw angle estimation, etc. It is worth to note that although SKU-110K dataset [9] is also assembled from retail environment such as supermarket shelves, the annotations in this dataset are horizontal bounding boxes (HBB) of shelf products since it mainly focuses on object detection in densely packed scenes. The aspect ratios of its HBB are distributed between 1:1-1:3 and hence, it does not cater to the problem that we want to solve.

Since images were collected with personal cellphone cameras, the original images have different resolutions; hence they were uniformly resized into $600\times 800$ before annotation took place. Figure 5 shows some statistics of Retail50K. We see that the dataset contains a wide range of aspect ratios and orientations (Figure 5 (a) and (b)). In particular, Retail50K is more challenging as compared to existing datasets [23, 39, 18] since it contains rich annotations with extremely high aspect ratios (higher than 1:10). Similar to natural-image datasets such as ImageNet (average 2) and MSCOCO (average 7.7), most images in our dataset contain around 2-6 instances with complex backgrounds (Figure 5 (c)). For experiments, we selected half of the original images as the training set, 1/6 as validation set, and 1/3 as the testing set.

We introduce a simple but effective loss function, PIoU, to exploit both the angle and IoU for accurate OBB regression. The PIoU loss is derived from IoU metric with a pixel-wise form, which is simple and suitable for both horizontal and oriented bounding box. To demonstrate its effectiveness, we evaluate the PIoU loss on both anchor-based and anchor-free frameworks. The experimental results show that PIoU loss can significantly improve the accuracy of OBB detectors, particularly on objects with high-aspect ratios. We also introduce a new challenging dataset, Retail50K, to explore the limitations of existing OBB detectors as well as to validate their performance after using the PIoU loss. In the future, we will extend PIoU to 3D rotated object detection. Our preliminary results show that PIoU can improve PointPillars [16] on KITTI val dataset [8] by 0.65, 0.64 and 2.0 AP for car, pedestrian and cyclist in moderate level, respectively.

The paper is supported in part by the following grants: China Major Project for New Generation of AI Grant (No.2018AAA0100400), National Natural Science Foundation of China (No. 61971277). The work is also supported by funding from Clobotics under the Joint Research Program of Smart Retail.