CRRS: Concentric Rectangles Regression Strategy for Multi-point Representation on Fisheye Images

Rectangular boundary box with regular shape and simple computation is convenient for position regression. Currently, it is one of the most popular representations of object detector[11, 12, 13]. However, rectangular boxes always contain much background information. This feature exerts a negative influence on the learning of semantic information and object detection model. Unfortunately, this defect is more serious when the images are distorted or the objects are dense and severely hidden. Therefore, more suitable representation methods are proposed accord with object shape and position. It is common to use oriented boxes to conduct people detection in overhead fish eye images[14, 15]. Compared with rectangular box, circle box[16, 17] only requires radius prediction with one Degree of Freedom(DoF) and is rotation invariant. Ellipse box[4] can include less background area, so it is more adaptive for severely overlapped objects. In addition, lots of researches aim to represent distorted objects more exactly with reduced background information. Thus, multi-point representation, like irregular quadrilateral box[5] and 24-side box[7, 9] is put forward.These representation methods are able to extract feature to locate objects more precisely.

IoU matters in label allocation, loss function, evaluation index and non-maximum suppression(NMS). Various representations make irregular IoU calculation more difficult. It is relatively simple to compute IoU of two circles[17], but the condition of two polygon is more complicated. Li et.al[5]took advantage of cross product of vector. Based on statistic approach, they proposed a method to estimate the positional relationship of sampling point and irregular quadrangle. Xie et.al[8]put forward Polar IoU, an algorithm to calculate mask IoU on the basis of polar vector. In this way, the complex IoU computation of polygon prediction box and truth box can be avoided. Previous work[9] adopted the mean of 24 concentric circles IoU in the stage of label allocation as approximate value. But on account of the lack of accurate calculation of 24 polygon IoU, there is no way of describing prediction performance quantitatively by mAP. For the distorted objects in 360°images, Cao et.al[18] proposed a new detection method using Field-of-View Bounding Boxes to replace rectangular boxes. Also, they proposed an approximate method called FoV-IoU which is more accurate than Sph-IoU[19]. This method helps object detectors to achieve better prediction results during training, inference, and evaluation stages. For multi-poins representation, this paper proposes a pixel-level method to accurately calculate polygon IoU and obtain accurate mAP evaluation results.

MTL[20] is widely used in the field of computer vision, like classification, depth estimation and semantic dense prediction. Since different tasks have different objective functions, it is necessary to consider how to allocate task loss weights when training a multi-task model. Therefore, many researchers have proposed various multi-task loss weight adjustment strategies. Different tasks can extract features by hard parameter sharing or soft parameter sharing for the best learning performance. Because diverse tasks may be unbalanced during training process, setting fixed weight for every task is no longer available. It is necessary to set adaptive weights in the loss function through unique strategies. The aim that all the tasks are synchronous[21] or auxiliary task accelerates main task[22, 23] is achieved thanks to these weighting methods.

Bayesian uncertainty can be used to design weight parameter[24, 25]. This method weights the losses of different tasks based on the uncertainty of the task loss. Specifically, the uncertainty weighting strategy weights the loss function of different tasks to reflect the difficulty of the task. Later some researches utilized learning rate to assign weighting for different losses[10, 26, 27]. Also there are certain classical algorithms update weight by gradient[28, 29].The gradient sharing strategy is a method that combines the loss functions of multiple tasks into a total loss function. This strategy allows different tasks to share model parameters to increase training speed and reduce the risk of over-fitting. Specifically, the gradient sharing strategy can balance more difficult tasks by using lower loss weights. GradNorm[28] balances learning among tasks by normalizing the gradient norm to a common scale or considering the relationship between the gradient directions of each task’s loss function. HydaLearn[30] introduces a dynamic weighted method for multi-task learning loss with auxiliary tasks. However, the above methods are mainly used for weight adjustment among multiple tasks. In addition, the information entropy-based strategy is a method that assigns task weights based on the mutual information between tasks. Specifically, this strategy calculates the mutual information between tasks to assign task weights. If two tasks have high mutual information, their weights will be adjusted to higher weights, and vice versa.If the special problem of single-task multi-loss is also considered as multi-task learning, the weight adjustment strategy of certain methods is no longer applicable. [31] has proven that for single-task multi-loss problems, using the coefficient of variation to design weight adjustment strategies can achieve better results. This paper synthesizes DWA[10] and Welford algorithm[32] to dynamically adjust the weights of single-task multi-loss.

Polygon bounding box: For the ground truth box, we first determine $C^{gt}(x_{c}^{gt},y_{c}^{gt})$ as the centroid of each object based on the instance mask. The formula is as follows:

$$x_{c}^{gt}=\frac{M_{10}}{M_{00}},\quad y_{c}^{gt}=\frac{M_{01}}{M_{00}}$$

(1)

where the zero-order moment of the object contour $M_{00}=\sum_{i}\sum_{j}V(i,j)$, the first moment is respectively: $M_{10}=\sum_{i}\sum_{j}i\cdot V(i,j)$ and $M_{01}=\sum_{i}\sum_{j}j\cdot V(i,j)$, $i$ and $j$ respectively represent the horizontal and vertical coordinates of each pixel point composing the object contour, $V(i,j)$represents the gray-scale value of $(i,j)$ pixel. For distorted objects, centroid outweighs center of rectangular bounding box. There may even be probable that the center falls outside the object mask, which affects the confirmation of vertices. We obtained $N$ boundary points using equal-angular sampling: $P_{0}^{gt}(x_{0}^{gt},y_{0}^{gt}),P_{1}^{gt}(x_{1}^{gt},y_{1}^{gt}),...,P_{N-1}^{gt}(x_{N-1}^{gt},y_{N-1}^{gt})$. In these points, $P_{0}^{gt}$ is the intersection point between the ray emitted from the centroid $C^{gt}$ in the direction of the positive half-axis of the pixel coordinate system and the object contour. Then every $360^{\circ}/N$, other points are sampled clockwise to form an approximate polygonal ground truth box for the object contour.

For the prediction box, we calculate $N$ boundary points $P_{0}^{pd}(x_{0}^{pd},y_{0}^{pd}),P_{1}^{pd}(x_{1}^{pd},y_{1}^{pd}),...,P_{N-1}^{pd}(x_{N-1}^{pd},y_{N-1}^{pd})$ based on the outputs of the network$(x_{c}^{pd},y_{c}^{pd},r_{0},r_{1},...,r_{N-1})$ and Formula (2):

$$\left\{{\begin{array}[]{*{20}{c}}{x_{k}^{pd}=x_{c}^{pd}+{r_{k}}\cdot\cos(k\cdot\frac{{360^{\circ}}}{N})}\\ {y_{k}^{pd}=y_{c}^{pd}+{r_{k}}\cdot\sin(k\cdot\frac{{360^{\circ}}}{N})}\end{array}}\right.$$

$(2)$

where $k=0,1,...,N-1$, $(x_{c}^{pd},y_{c}^{pd})$ is the centroid coordinates predicted by the network. $r_{0},r_{1},...,r_{N-1}$ are the Euclidean distance between the centroid predicted and $N$ predicted boundary points.

The calculation of irregular IoU: The complexity of overlapping shape between two polygons is much greater than that of two rectangles. Therefore, we propose a pixel-level confirmation method that can accurately calculate the area of any polygon and the IoU. The polygon is constructed by the above method can be easily divided into $N$ triangles. Since it is relatively simple to determine whether a point is inside a triangle, the problem of calculating the polygon area can be simplified. It is more easy to confirm whether the pixels contained in the circumscribed rectangle of the polygon are inside any of the triangles. As shown in Fig.3, the triangle composed of $P_{0}$, $P_{N-1}$, $C$ and the point $J$ can illustrate the judging method. Construct three sets of vectors $\overrightarrow{{P_{0}}{P_{N-1}}}$ and $\overrightarrow{{P_{0}}J}$, $\overrightarrow{{P_{N-1}}C}$ and $\overrightarrow{{P_{N-1}}J}$, $\overrightarrow{C{P_{0}}}$ and $\overrightarrow{CJ}$. Perform a cross product calculation on each set of vectors:

$$\begin{array}[]{l}{cross1=\overrightarrow{{P_{0}}{P_{N-1}}}\times\overrightarrow{{P_{0}}J}}\\ {cross2=\overrightarrow{{P_{N-1}}C}\times\overrightarrow{{P_{N-1}}J}}\\ {cross3=\overrightarrow{C{P_{0}}}\times\overrightarrow{CJ}}\end{array}$$

$(3)$

If $cross1$, $cross2$ and $cross3$ have same sign, point $J$ is in the triangle. In order to compute area, we set the gray-scale value of pixels inside the polygon to 1, and set the rest to 0. Summing up the gray-scale values of all pixels will give the area of the polygon. The key of calculating IoU is to compute the intersection area. We first determine which pixels are included in the ground truth polygon and the predicted polygon respectively. After converting the masks of the two polygons to Boolean types, the ”and” operation is performed. The number of intersection pixels is counted to calculate the polygon IoU(Poly IoU). Our method has been validated to be applicable to both convex and concave polygons.

We designed a Concentric Rectangles Regression Strategy with EIoU loss[33] inspired by the previous concentric circles(the design of aspect ratio in EIoU loss and CIoU loss[34] cannot help with circles regression). This new strategy not only accelerates convergence but also efficiently and accurately completes the polygonal position regression by fully utilizing Eiou Loss.

The construction of concentric multi-rectangle: As shown in Fig.4, four bounding points $P_{0}^{gt}$,$P_{6}^{gt}$,$P_{12}^{gt}$,$P_{18}^{gt}$ are aligned with the centroid axis. So we combine them to form $Rect_{0}^{gt}$ and $Rect_{1}^{gt}$. The remaining boundary points form rectangles with the centroid respectively and the total number of rectangles is $N-1$. Each rectangle has its center point aligned with the centroid. The Euclidean distance from the centroid to the boundary points is taken as half of the diagonal of the rectangle, so we call it the concentric multi-rectangle. Similarly, the multi-rectangle construction method for the polygon prediction box is the same.

Loss function: The regression of the rectangular box needs to consider the degree of overlap, distance, and aspect ratio. Among the current Iou-based loss function[34, 33, 35, 36], efficient IoU loss(EIoU) has best performance. This paper uses EIoU Loss to calculate the loss of the ground truth-predicted pairs of the above-mentioned concentric multi-rectangle. By this means, the position regression of each boundary point of the polygon can be achieved. As shown in Fig.4, the procedure to calculate ground truth-predicted pairs for $Rect_{i}^{gt}$-$Rect_{i}^{pd}$ is as follows:

$$\begin{array}[]{l}EIo{U_{i}}=Io{U_{i}}-\displaystyle{\frac{{{\rho^{2}}({C^{gt}},{C^{pd}})}}{{{{({c_{i}}\_diag)}^{2}}}}}\\ {\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}-\displaystyle{\frac{{{{(Rect_{i}^{gt}\_w-Rect_{i}^{pd}\_w)}^{2}}}}{{{{({c_{i}}\_w)}^{2}}}}}\\ {\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}-\displaystyle{\frac{{{{(Rect_{i}^{gt}\_h-Rect_{i}^{pd}\_h)}^{2}}}}{{{{({c_{i}}\_h)}^{2}}}}}\end{array}$$

$(4)$

where $i=0,1,...,N-3$, ${\rho^{2}}({C^{gt}},{C^{pd}})$ represents the Euclidean distance between the centroid of the ground truth box and the predicted box (as well as the center of $Rect_{i}^{gt}$-$Rect_{i}^{pd}$), ${c_{i}}\_w$, ${c_{i}}\_h$ and ${c_{i}}\_diag$ are the width, height, and diagonal length of the circumscribe rectangle respectively of $Rect_{i}^{gt}$, $Rect_{i}^{pd}$. $Rect_{i}^{gt}\_w$ and $Rect_{i}^{gt}\_h$ are the width and height of the ground truth rectangular box. $Rect_{i}^{pd}\_w$ and $Rect_{i}^{pd}\_h$ are the width and height of the prediction rectangular box. Then EIoU loss can be computed:

$$\begin{array}[]{l}EIoU\_Los{s_{i}}=1-EIo{U_{i}}\end{array}$$

$(5)$

Finally, we obtain $N-2$ loss terms that jointly participate in the position regression calculation.

The overall loss includes multi-point regression loss, confidence loss, and classification loss. Among these losses, multi-point regression loss requires accumulation of single-point loss, significantly increasing the number of loss terms. In order to balance the different loss components, this experiment draws on the weight adjustment methods of multi-task learning. In this paper, we utilizes the DWA[9] adjustment strategy. The weight of the $j$-th loss term is determined as follows:

$$\begin{array}[]{l}\displaystyle{\frac{{n\times\exp({r^{j}}/T)}}{{\sum\limits_{i=1}^{n}{\exp({r^{i}}/T)}}}}\end{array}$$

$(6)$

where $n$ is the number of loss items, $r^{i}$ is loss ratio of epoch-$i$, $T$ is temperature. There are two ways to calculate loss ratio:

(1)the ratio of epoch $i$ loss to epoch $i-1$ loss

$$\begin{array}[]{l}\displaystyle{r^{i}=\frac{L_{i}}{L_{i-1}}}\end{array}$$

$(7)$

(2)the ratio of epoch-$i$ loss to average loss in $i-1$ epochs:

$$\begin{array}[]{l}\displaystyle{r^{i}=\frac{L_{i}}{\sum_{j=1}^{i-1}L_{j}/(i-1)}}\end{array}$$

$(8)$

Comparing the calculation methods of two loss rates, the mean loss ratio is more stable. Mean loss can effectively reduce the impact of loss fluctuations during the training process on the loss ratio. Welford algorithm[32] is used for calculating the mean loss , and the formula is:

$$\begin{array}[]{l}\displaystyle{\bar{x}_{n}=\bar{x}_{n-1}+\frac{x_{n}-\bar{x}_{n-1}}{n}}\end{array}$$

$(9)$

where $x_{n}$ is epoch-$n$ loss in our experiment. The loss rate is calculated in the form of loss ratio, which can transform losses of different scales into the same scale. According to the definition of the loss ratio, the better the training effect of a certain regression task, the faster the loss decreases. Similarly, the smaller the loss rate, the smaller the weight assigned to it. This means the model focuses more on learning the part with a poorer training effect. This strategy meets the requirement of balancing the losses of each part. However, the temperature coefficient can also smooth the weights of each part. The higher the temperature coefficient, the more evenly the weights are distributed. When the temperature coefficient tends to infinity, the loss weight of each part is the same. Setting a temperature coefficient higher than 1 can moderately increase the weight of the part with a lower loss rate. Compared with GradNorm[28], there is no extra backward process required, so DWA can simplify computation.

At the beginning, we designed several different forms of multi-rectangle construction and selected a small sample size of data as the training set.

$\bullet$ 22 vertex-shared rectangles: Our boundary points aligned with the centroid axis are grouped into two pairs. Each pair forms a rectangle with the centroid as one of the vertices (the diagonal is the line connecting the two boundary points). The remaining 20 boundary points are used to construct 20 rectangles, each with the centroid as one of the vertices (the diagonal is the line connecting the centroid and the boundary point). Each rectangle must have one vertex at the centroid.

$\bullet$ 22 concentric rectangles: Section III explains this method in detail.

Loss functions both use EIoU loss. As shown in Table I, Scheme 2 performs significantly better than another. We believe that the accuracy of the centroid has a greater impact on the final detection results compared to the multiple points.The shared centroid design in Scheme 2 precisely aggregates the power of multiple scattered rectangle regressions. Such effect enables the multiple rectangles to converge faster to the same center point, that is, the centroid. Although Scheme 1 also has the feature of shared vertices, the vertices are jointly determined by the width and height, making regression more difficult. Scheme 1 does not have any commonality between multiple rectangles, and therefore no collective force is formed.

Based on above analysis, three types of position regression loss functions (poly IoU, 24 concentric circles loss[9] and 22 concentric rectangles EIoU) were used either individually or in combination for contrast experiment. The mAP results are shown in Table II. Using the Poly IoU alone for position regression resulted in poor detection performance. After analysis, there are three main reasons explaining this phenomenon. Firstly, Rectangular bounding boxes can use IoU loss better than Euclidean distance loss because of the strong correlation between the four points. However, the boundary points of a polygonal box are relatively independent. Hence area loss alone cannot accurately predict multiple points.The direct analogy of Poly IoU to rectangular IoU without adaptive improvement loses effect. Secondly, Poly IoU loss struggles to achieve synchronous regression of multiple points. By splitting Poly IoU loss into 22 rectangular IoU losses, each point can be regressed separately and aggregated to form the overall regression. Thirdly, Poly IoU does not restrict the shape and orientation of the predicted bounding boxes. The same IoU value corresponds to different shapes and orientations, decreasing prediction accuracy.

Compared to Poly IoU loss, the detection performance is significantly improved when using 24 concentric circles GIoU loss or the 22 concentric rectangles EIoU loss. If Poly IoU loss is combined with them, it can further improve the detection performance. Overall, the combination of the proposed concentric rectangles EIou loss and the Poly-IoU loss is a good choice for multi-point representation regression strategy. The detection results and ground thruth of each method are shown in Fig. 5.

Exploration about temperature: At the initial stage of the experiment, the temperature coefficient is set to 1. However, it is found that the losses of each part fluctuat greatly, and the final prediction performance is poor. Therefore, the temperature coefficient is adjusted, resulting in a change in the weight distribution. Through a series of comparisons of temperature coefficients, the experiment find that setting the temperature coefficient to a value greater than 1 can reduce the instability of the loss and improve the prediction performance. Since the temperature coefficient has a smoothing effect on the weights, increasing the temperature coefficient can moderately reduce the difference between the weights. This can avoid sharp changes in weights between different batches. Meanwhile, increasing the temperature can moderately raise the weight of the loss with good training performance. This trend can reduce the risk of falling into a local minimum to some extent. But too high temperature may cause the weights to lose significant discrimination. Thus the model cannot focus on learning the poorer performing regressions. Therefore, it is necessary to find the optimal value through experimental comparison. As shown in TableIII, based on the mAP under different temperature coefficients, the final temperature is set to 20.

Exploration about loss ratio: The line chart of losses is shown in Fig.6. It can be observed that the single-epoch loss fluctuates greatly, but the average loss value is relatively smooth. If the loss ratio is defined by single-epoch loss, the value strongly depends on the loss in the previous epoch. Then the irregular fluctuations in the single-epoch loss will affect the accuracy of the loss ratio and weight allocation. Therefore, the experiment improved the above loss ratio by changing the single loss to a more stable average loss, making the calculation of the loss rate more accurate.