BPJDet: Extended Object Representation for Generic Body-Part Joint Detection

We here only discuss emerging CNN-based detectors that achieve promising results over traditional approaches using hand-crafted features. For human body detection, it belongs to either general object detection containing the person category [23, 24, 25, 26, 65, 66, 61, 64] trained on common datasets like COCO [27], or pedestrian detection [2, 3, 5, 74, 75] trained on specific benchmarks CityPersons [28] and CrowdHuman [29]. A key challenge in human body detection is occlusion. Therefore, many researches propose customized loss functions [2, 3], improved NMS [74, 5] or tailored distribution model [75] for alleviating problems of crowded people detection. PedHunter [4] designs a mask-guided module to leverage the head information to enhance the pedestrian feature. Currently, with the proposed burdensome yet powerful Detection Transformer (DETR) [76], some query-based crowd pedestrian detection methods [77, 78, 79, 80] are dominant. On the other hand, detection of body part has also been intensively and extensively studied. Detection of face [13, 14], head [15, 16, 81] and hand [58, 18] are all vigorous fields. Face detectors are usually based on well-designed networks and trained on dedicated datasets. While, head and hand detectors are rarely studied alone, and often used to facilitate downstream tasks. Precise design of body or part detector is beyond the scope of this paper. We focus on the joint detection of them.

We mainly pay attention to the joint detection of human body and one or more body parts like face, head and hands. First of all, body-head joint detection is the most popular couple, because the human head is a salient structural body part and plays a vital role in recognizing people, especially when occlusion occurs. For example, DA-RCNN [54] proposes to handle the crowd occlusion problem in human detection by capturing and cross-optimizing body and head parts in pairs with double anchor RPN. JointDet [55] presents a head-body relationship discriminating module to perform relational learning between heads and human bodies. Recently, BFJDet [56] firstly investigates the performance of body-face joint detection, and proposes a bottom-up scheme that outputs body-face pairs for each pedestrian. It also adopts independent detection followed by pairwise association. For body-hand joint detection, [17] devises heuristic strategies to match hand-raising gestures with body skeletons [67] in classroom scenes. BodyHands [18] proposes a novel association network based on Mask R-CNN [11] to jointly detect hands and the body location, and introduces a new corresponding hand-body association dataset. Besides, body-parts joint detection is also important for instance-level human-parts analysis. DID-Net [57] adopts Faster R-CNN [23] with two carefully designed detectors for the human body and two body parts (hand and face) separately. Hier R-CNN [58] extends body parts up to six, and modifies Mask R-CNN [11] with a new designed hierarchy branch inspired by FCOS [66] for subordinate relationship learning of body and parts. Unlike all of them, our approach BPJDet is not restricted to one or more body parts, and can handle detection and association in an end-to-end way.

Superior body-part association is meaningful for boosting many downstream tasks. According to their dependence on body part associations, we can roughly divide them into two categories: (1) low-level application: such as person surveillance [31, 32], robust person ReID [33, 34], human parsing [35, 36, 37] and human pose estimation [50, 51, 52, 53]. (2) high-level understanding: such as robot teleoperation [30], hand-related object contact [38, 39, 40, 41, 42] and human-object interaction (HOI) [43, 44, 45, 46]. Actually, there are essential differences between the two categories. The low-level tasks usually apply body-part associations to their methods for performance enhancing like robust person search by adding head [32], occluded person ReID relying on body part-based features [34], keypoints regressors using body parts dependence [50] and part affinity fields for learning to associate body parts [67, 52, 53]. Body-part association is an alternative for them. While, the high-level tasks are often inseparable from body-part associations. For example, human-machine systems should recognize the physical contact state of a body-part like hand [38, 30]. Human contact estimation [39] and understanding [40] may also need to analyze physical states of hands including their visual appearance and surrounding local context. In the well-defined field of HOI, body-part association is even more essential for understanding human activities [43, 45] and interactions [44, 46]. To confirm the benefits of BPJDet, we choose to strengthen two basic representative tasks: low-level accurate crowd head detection and high-level hand contact estimation by establishing our proposed stronger body-part associations.

Most modern object detectors include three categories: box-based [23, 26, 24, 25, 61] using predefined anchors, point-based [82, 65, 66, 83, 84, 63, 64] free of anchor boxes and query-based [76, 78, 79, 80, 85] representing objects with a set of queries. Those box-based detectors heavily rely on handcrafted anchors that include parameters like number, size and aspect ratio. The object representation of them is rare and difficult to expand. Those point-based detectors directly predict object using keypoint or center point regression without needing of anchor boxes. This mechanism can eliminate the hyperparameters finetuning of anchors, achieve similar performance to anchor-based detectors with less computational cost, and keep better generalization ability. The last query-based DETRs abandon the traditional object representation and streamline the detection training pipeline by viewing it as a set prediction problem. However, their implicit encoder-decoder design does not align with our concept of extending representation. Instead, we propose a novel extended object representation by exploring both box-based (anchor-based) and point-based (anchor-free) detectors.

Prior to us, some literatures have attempted to expand the point-based object representation, especially for human pose estimation that has an essential overlap with the object detection task. For example, CenterNet [65] models objects using heatmap-based center points and subtly represents human poses as a 2K-dimensional property of the center point. Similarly, FCPose [69] adapts single-stage anchor-free FCOS [66] with proposed dynamic filters to process person detections by predicting keypoint heatmaps and regressing offsets. Other single-stage pose estimators like SPM [68], AdaptivePose [70] and YOLO-Pose [86] also learn to regress keypoints as series of joint offsets. Recently, ED-Pose [87] presents an explicit box detection framework which unifies the contextual learning between global human-level and local keypoint-level information based on DN-DETR [85]. It re-considers human pose estimation task as two explicit box detection processes with a unified representation and regression supervision, which much resembles our joint body (global) and part (local) detection design.

Currently, most body-part joint detection methods [57, 58, 55, 56, 18] are based on two separate stages including detection and post-matching. The inherent drawbacks of post-matching for body-part association are often unavoidable. For example, once the true main body of one detected body-part is undetected or improperly boxed, it will be a high probability event by mistakenly allocating this body-part to one person bounding box it just falls within, even if the error is quite obvious or unreasonable. In Section 5.3, we give some ordinary failure cases of body-hand in Fig. 8 and body-parts in Fig. 9 to explicitly reflect their defects. To alleviate this problem, we now have at least three strategies originally designed for human pose estimation to complete a single-stage body-part association. They are (1) center-offset regression as in CenterNet [65] and FCOS [66], (2) part affinity fields as in OpenPose [67, 53] and OpenPifPaf [52], and (3) hierarchical offset regression as in SPM [68]. We illustrate their discrepancies in Fig. 2. These correlation methods have greatly inspired our work. We apply the first intuitive yet still sufficiently effective center-offset regression strategy to validate the desirability to extend this single-stage association to traditional object representation.

In our proposed BPJDet, we train a dense single-stage anchor-based detector to directly predict a set of objects $\{\widehat{\mathcal{O}}\!\in\!\widehat{\mathbf{O}}\!\parallel\!\widehat{\mathcal{O}}\!=\!cat(\widehat{\mathcal{O}}^{box},\widehat{\mathcal{O}}^{dis}),\widehat{\mathbf{O}}\!=\!\widehat{\mathbf{O}}^{b}\cup\widehat{\mathbf{O}}^{p}\}$, which contains human body set $\widehat{\mathbf{O}}^{b}$ and body-part set $\widehat{\mathbf{O}}^{p}$ concurrently. A typical extended object prediction $\widehat{\mathcal{O}}$ is concatenated of the bounding box $\widehat{\mathcal{O}}^{box}$ and corresponding center point displacement $\widehat{\mathcal{O}}^{dis}$.

It locates an object with a tight bounding box $\mathbf{\hat{b}}\!=\!(\hat{b}_{x},\hat{b}_{y},\hat{b}_{w},\hat{b}_{h})$ where coordinates $(\hat{b}_{x},\hat{b}_{y})$ are the center position, $\hat{b}_{w}$ and $\hat{b}_{h}$ are the width and height of $\mathbf{\hat{b}}$, respectively. It also records the relative displacement $\mathbf{\hat{d}}\!=\!(\hat{d}_{x_{i}},\hat{d}_{y_{i}})|^{k}_{i=1}$ of center point of body-part ($\widehat{\mathcal{O}}\!\in\!\widehat{\mathbf{O}}^{b}$) or affiliated body ($\widehat{\mathcal{O}}\!\in\!\widehat{\mathbf{O}}^{p}$) to each $\mathbf{\hat{b}}$. Here, $k$ is the body-part number. Considering that body-part may be one or more components (e.g., two both hands with $k\!=\!2$, and the COCOHumanParts with $k\!=\!6$), we allow $\mathbf{\hat{d}}$ to be compatible with dynamic 2D offsets. The $\hat{o}$ and $\mathbf{\hat{c}}$ are predicted objectness and classification scores, respectively. For these regressed offsets, we will explain how to decode them for the body-part association with fusing detected body-part set $\widehat{\mathbf{O}}^{p}$ in Section 4.4.

Intuitively, we can benefit a lot from this extended representation. On one hand, an appropriate large body bounding box $\widehat{\mathcal{O}}^{box}$ possesses both strong local characteristics and weak global features (such as surrounding background and anatomical position) for its body part $\widehat{\mathcal{O}}^{dis}$ offset regression. This enables the network to learn their intrinsic relationships. On the other hand, compared to methods [58, 55, 56, 18] training multiple subnetworks or stages, mixing $\widehat{\mathcal{O}}^{box}$ and $\widehat{\mathcal{O}}^{dis}$ up for synchronous learning can be leveraged directly in BPJDet without the need of complicated post-processing. By designing a one-stage network that uses shared heads to jointly predict $\widehat{\mathcal{O}}^{box}$ and $\widehat{\mathcal{O}}^{dis}$, our approach can achieve high accuracy with much less computational burden during training and inference.

Our network structure is shown in Fig. 3. We choose the recent cost-effective one-stage YOLOv5 [61] as the basic backbone $\mathcal{N}$. Specifically, following YOLOv5, we feed one RGB image $\mathbf{I}\!\in\!\mathbb{R}^{h\times w\times 3}$ as the input to $\mathcal{N}$, keep its beneficial data augmentation strategies (e.g., Mosaic and MixUp), and output four grids $\widehat{\mathit{G}}\!=\!\{\widehat{\mathcal{G}}^{s}\!\parallel\!s\!\in\!\{8,16,32,64\}\}$ from four multi-scale heads. Each grid $\widehat{\mathcal{G}}\!\in\!\mathbb{R}^{A_{a}\times A_{o}\times\frac{h}{s}\times\frac{w}{s}}$ contains dense object outputs $\widehat{\mathbf{O}}$ produced from $A_{a}$ anchor channels and $A_{o}$ output channels. Supposing that one target object $\mathcal{O}$ is centered at $(b_{x},b_{y})$ in the feature map $\mathbf{F}^{s}$, the corresponding grid $\widehat{\mathcal{G}}^{s}$ at cell $({b_{x}}/{s},{b_{y}}/{s})$ should be highly confident. When having defined $A_{a}$ anchor boxes $\mathcal{B}^{s}\!=\!\{(B^{w}_{i},B^{h}_{i})|^{A_{a}}_{i}\}$ for the grid $\widehat{\mathcal{G}}^{s}$, we will generate $A_{a}$ anchor channels at each cell $({b_{x}}/{s},{b_{y}}/{s})$. Furthermore, to obtain robust capability, YOLOv5 allows detection redundancy of multiple objects and four surrounding cells matching for each cell. This redundancy makes sense to the detection of body or part position $\mathcal{O}^{box}$, but it is not completely facilitative to the regression of offsets $\mathcal{O}^{dis}$. We interpret this in Section 4.3.

Then, we explain the arrangement of one prediction $\widehat{\mathcal{O}}$ from $A_{o}$ output channels of $\widehat{\mathcal{G}}^{s}_{i,x,y}$ which is related to $i$-th anchor box at grid cell $(x,y)$. As shown in the example of grid cells in Fig. 4, one typical predicted $\widehat{\mathcal{O}}$ embedding consists of four parts: the objectness or probability $\hat{o}$ that an object exists, the bounding box candidate $\mathbf{\hat{b}}^{\prime}=({\hat{b}_{x}}^{\prime},{\hat{b}_{y}}^{\prime},{\hat{b}_{w}}^{\prime},{\hat{b}_{h}}^{\prime})$, the object classification scores $\mathbf{\hat{c}^{\prime}}\!=\!(\hat{c^{\prime}}_{1},..,\hat{c^{\prime}}_{k+1})$, and the body-part offsets candidate $\mathbf{\hat{d}}^{\prime}\!=\!(\hat{d}^{\prime}_{x_{i}},\hat{d}^{\prime}_{y_{i}})|^{k}_{i=1}$. Thus, $A_{o}\!=\!3k+6$. To transform the candidate $\mathbf{\hat{b}}^{\prime}$ into coordinates $\mathbf{\hat{b}}$ relative to the grid cell $\widehat{\mathcal{G}}^{s}_{i,x,y}$, we apply conversions as below:

where $\phi$ is the sigmoid function that limits model predictions in the range of $(0,1)$. Similarly, this detection strategy can be extended to the offset of body-part. A body-part’s intermediate offsets $\mathbf{\hat{d}}^{\prime}$ are predicted in the grid coordinates and relative to the grid cell origin $(x,y)$ using:

In the way, $\hat{d}_{x_{i}}$ and $\hat{d}_{y_{i}}$ are constrained to ranges $\pm 2{B^{w}_{i}}/{s}$ and $\pm 2{B^{h}_{i}}/{s}$, respectively. To learn $\mathbf{\hat{b}}$ and $\mathbf{\hat{d}}$, losses are applied in the grid space. Sample targets of $\mathbf{b}$ and $\mathbf{d}$ are shown in Fig. 4.

For a set of predicted grids $\mathit{\widehat{G}}$, we firstly build target grids set $\mathit{G}$ following formats introduced in Section 4.2. Then, we mainly compute following four loss components:

For the bounding box regression loss $\mathcal{L}_{box}$, we adopt the complete intersection over union (CIoU) across four grids $\mathcal{G}^{s}$. $\mathsf{BCE}$ in both the objectness loss $\mathcal{L}_{obj}$ and classification loss $\mathcal{L}_{cls}$ is the binary cross-entropy. The multiplier $o$ in $\mathcal{L}_{obj}$ is used for penalizing candidates without hitting target grid cells ($o\!=\!0$), and encouraging candidates around target anchor ground-truths ($o\!=\!1$). The coefficient $w_{s}$ is a balance weight for different grid levels in YOLOv5. For the body-part displacement loss $\mathcal{L}_{bpd}$, we utilize the mean squared error (MSE) to measure offset outputs $\mathbf{\hat{d}^{\prime}}$ and normalized targets $\mathbf{d^{\prime}}$. Before that, we apply a filter $\varphi(\cdot)$ with the visibility label of body-part to remove out those false-positive offset predictions from $\widehat{\mathcal{O}}$. Finally, we calculate the total training loss $\mathcal{L}=(\mathit{\widehat{G}},\mathit{G})$ as follows:

where $N_{bs}$ is the batch size. The $\alpha$, $\beta$, $\gamma$ and $\lambda$ are weights of losses $\mathcal{L}_{box}$, $\mathcal{L}_{obj}$, $\mathcal{L}_{cls}$ and $\mathcal{L}_{bpd}$, respectively.

In inference, we process predicted objects set $\widehat{\mathbf{O}}$ to get final results. First of all, we apply the conventional Non-Maximum Suppression (NMS) to filter out false-positive and redundant bounding boxes of body and part objects:

where $\tau_{conf}$ and $\tau_{iou}$ are thresholds for object confidence and IoU overlap, respectively. Considering different recognition difficulties and overlapping levels, we define distinct thresholds $\tau^{b}$ and $\tau^{p}$ for filtering body and part boxes. We fetch confidence of each predicted object $\widehat{\mathcal{O}}$ by $\hat{o}\cdot\hat{c}_{i}$, where $i\!=\!1$ for body object and $i\!>\!1$ for part object. Then, we rescale $\widehat{\mathcal{O}}^{box^{\prime}}$ and $\widehat{\mathcal{O}}^{dis^{\prime}}$ in $\widehat{\mathbf{O}}^{b^{\prime}}$ and $\widehat{\mathbf{O}}^{p^{\prime}}$ to obtain real $\widetilde{\mathbf{O}}^{b}$ and $\widetilde{\mathbf{O}}^{p}$ by the following transformations:

where $x_{o}$ and $y_{o}$ are offsets from grid cell centers. The scalar $s$ maps the down-sampled size of box and offset back to the original image shape.

Finally, we update associated body parts of each left body object in $\widetilde{\mathbf{O}}^{b}$ by fusing its regressed center point offsets ($\widetilde{\mathcal{O}}^{dis}\!\in\!\widehat{\mathbf{O}}^{b^{\prime}}$) with the remaining candidate part objects $\widetilde{\mathbf{O}}^{p}$. Specifically, we search each box $\widetilde{\mathcal{O}}^{box}$ in part objects with its nearest body-part offset $\widetilde{\mathcal{O}}^{dis}$ belonging to body object. The part box that has a large inner IoU ($>\tau^{inner}_{iou}$) with the body box will be selected. Inner IoU here represents the ratio of intersection area on smaller body-part bounding box instead of the larger union of body-part and human body.

We here naturally assume that the smaller body-part box should be completely inside its associated larger human body box if using ground-truth labels. During the prediction stage, detected boxes do not always tightly surround the objects. It may be too large or too small. We cannot simply set $\tau^{inner}_{iou}\!=\!1$. In contrast, a very small $\tau^{inner}_{iou}$ will obviously lead to many wrong body part pair candidates. Therefore, we should assign a reasonable $\tau^{inner}_{iou}$ to keep a balance between the quantity and quality of matching pairs. This experimental discussion is presented in Section 5.2.2. We summarize the overall decoding process in Alg. 1. We report all evaluation results on the final updated body and parts boxes set pairs $(\widetilde{\mathbf{O}}^{b^{\prime}},\widetilde{\mathbf{O}}^{p^{\prime}})$.

To reveal the universality of our proposed extended representation, we here interpret how to integrate it with advanced anchor-free detectors YOLOv5u [61] and YOLOv8 [64]. YOLOv8 is the latest iteration of YOLO series, which employs state-of-the-art backbone [83, 84] and neck architectures. It adopts an objectness-free split head, which contributes to better accuracy and a more efficient detection process. YOLOv5u modernizes YOLOv5 by originating from its foundational architecture and adopting an anchor-free head as in YOLOv8. These similarities allow us to extensively reuse the detailed designs in the previous subsections, with a key focus on representation adjustment.

Specifically, in the anchor-free paradigm, detection is formulated as a dense inference of every pixel in feature maps. For each position $p_{i}\!=\!(x_{i},y_{i})$ in $\mathbf{F}^{s}$, the prediction head regresses a 4D vector $(l_{i},t_{i},r_{i},b_{i})$, which represents the relative offsets from the four sides (left, top, right and bottom) of a bounding box anchored in $p_{i}$. We reformulate the anchor-based extended representation $\widehat{\mathcal{O}}$ in Eqn. 1 into the anchor-free one $\widehat{\mathcal{O}}_{u}$ as below.

where the objectness score $\hat{o}$ is omitted. The meaning of $\mathbf{\hat{c}}_{u}$ and $\mathbf{\hat{d}}_{u}$ are totally the same as $\mathbf{\hat{c}}$ and $\mathbf{\hat{d}}$, respectively. The bounding box $\mathbf{\hat{b}}_{u}\!=\!(\hat{b}_{l},\hat{b}_{t},\hat{b}_{r},\hat{b}_{b})$ is the distinctive design for point-based detectors. In both YOLOv5u and YOLOv8, they adopt the same task-aligned assigner as in TOOD [83] to learn the probabilities of values around the continuous locations of target bounding boxes $\widehat{\mathcal{O}}_{u}^{box}$, which includes a CIoU loss $\mathcal{L}_{u_{box}}$ and a Distribution Focal Loss (DFL) [84] $\mathcal{L}_{u_{dfl}}$. Please refer [83] and [84] for the explicit definition of these two losses. The classification loss $\mathcal{L}_{u_{cls}}$ uses the binary cross-entropy. For the body-part displacement loss $\mathcal{L}_{u_{bpd}}$, we synchronously generate target GTs in one identical box assigner, and optimize corresponding outputs $\mathbf{\hat{d}^{\prime}}_{u}$ using MSE. The final total training loss $\mathcal{L}_{u}$ is as follows:

which is an updating of $\mathcal{L}$ in Eqn. 8. $N_{bs}$ is the batch size. The $\alpha_{u}$, $\beta_{u}$, $\gamma_{u}$ and $\lambda_{u}$ are weights of each loss. In inference, the association decoding part keeps unchanged.

For supporting the downstream application hand contact estimation, we build a new architecture by extending the proposed object representation with additional spaces to estimate the contact state of a detected hand. We rename this new framework as BPJDetPlus.

Specifically, we keep extending the object representation $\widehat{\mathcal{O}}$ with adding $\widehat{\mathcal{O}}^{cts}$ that represents four contact states of two hands . The new typical prediction $\widehat{\mathcal{O}^{\prime}}$ inherited from $\widehat{\mathcal{O}}$ in Eqn. 1 is constituted as below:

where $\mathbf{\hat{h}}$ contains hand contact states. The $\hat{h}^{1}_{s_{j}}$ or $\hat{h}^{2}_{s_{j}}$ is a specific contact state $s_{j}$ ($j\in\{1,2,3,4\}$) of one hand part. Following [39], we define a new hand contact states loss $\mathcal{L}_{cts}$ to be the sum of four independent binary cross-entropy losses corresponding to four possible categories.

where $\hat{h}_{s_{j}}$ is the predicted hand contact state ranging from 0 to 1, and $h_{s_{j}}$ is the ground-truth state being 0, 1 or 2 meaning No, Yes or Unsure, respectively. We do not penalize $h_{s_{j}}$ with Unsure state. The final loss of BPJDetPlus is updated as:

which is an extension of the multi-loss function $\mathcal{L}$ in Eqn. 8. $N_{bs}$ is the batch size. The loss weight $\mu$ of $\mathcal{L}_{cts}$ is empirically set as 0.01 by online searching similar to $\lambda$ in Section 5.2.1. We train BPJDetPlus on the train-set of ContactHands [39] and evaluate its performance on the test-set. In inference, the final predicted hand contact state score is a weighted sum of two probabilities embedded in the hand and its associated body instance. More results are in Section 5.4.2.