HPPS: A Hierarchical Progressive Perception System for Luggage Trolley Detection and Localization at Airports

The main contributions of this article are summarized as follows:

• •

  This article presents a dataset of 13740 images capturing various luggage trolley states and occlusion levels across diverse environments to improve luggage trolley detection accuracy in real-world scenarios.

• •

  This article introduces a novel Hierarchical Progressive Perception System (HPPS) designed for detecting and locating luggage trolleys under partial occlusion.

• •

  Real-world experiments demonstrate the robustness and accuracy of HPPS in complex and dynamic environments, where it detects and locates the target luggage trolley and successfully collects it. This progress enhances the deployment of robotic autonomous luggage trolley collection systems at airports.

The remainder of this article is organized as follows. The related work is reviewed in Sec. II. Sec. III provides the details of the HPPS, including the Detection Module, Keypoints Process Module, Orientation Process Module, Filter Module, and Motion Planner Module. The Dataset, Implementation Details, Experiment Platform, and Results are explained in Sec. IV. The final section, in Sec. V, summarizes this article and considerations for future work.

Object detection and classification are essential for identifying and localizing objects within images. Popular methods include region-based approaches like the R-CNN series [12][13][14] and single-shot detectors such as YOLO (You Only Look Once) [15] and SSD (Single Shot MultiBox Detector) [16]. While R-CNN models are highly accurate, they are computationally intensive. In contrast, YOLO offers a compelling balance between speed and accuracy, making it suitable for real-time applications. YOLO divides the image into a grid, each cell predicting multiple bounding boxes and their confidence scores, which indicate the likelihood of an object’s presence and the accuracy of the box. Each grid cell also classifies the detected object, combining localization with precise categorization. Considering both performance and efficiency, the YOLO model is an ideal choice for real-time object detection. Its ability to process images quickly and with high accuracy meets the demands of most practical applications. Employing the YOLO model will facilitate efficient and accurate visual recognition capabilities for our task.

Keypoint detection is instrumental for accurately identifying and localizing specific parts of objects in images. Several neural network models have significantly advanced this field: OpenPose [17] excels in multi-object scenarios by generating part candidates and analyzing their connections. Stacked Hourglass Networks [18] leverage a repetitive structure that captures and integrates features at multiple scales, ideal for precise single-object keypoint localization. DeepPose [19], developed by Google, applies deep learning to keypoint detection by directly regressing coordinates from images but may struggle with complex backgrounds. Convolutional Pose Machines (CPM) [20] refine keypoint predictions through multi-stage processing, each enhancing the heatmaps’ accuracy. HRNet [21] maintains high-resolution representations throughout the network, enabling simultaneous capture of detailed and contextual information, making it highly effective for complex poses and environments. HRNet maintains high-resolution streams throughout its architecture, effectively capturing fine and coarse features. This design allows HRNet to excel in precision and detail, outperforming other models, especially on challenging datasets like COCO keypoints. Unlike other models that may lose spatial information due to pooling operations, HRNet retains high-resolution features during processing, making it particularly suitable for complex scenarios involving occlusions and various poses.

Accurate orientation estimation is fundamental for autonomous systems. Traditional approaches relied on handcrafted features and machine learning classifiers like Support Vector Machines (SVMs) [22], which often struggled with occlusions and complex backgrounds. Current deep learning-based methods frequently approach orientation estimation through classification tasks. The method in [23] employs a four-layer neural network to regress the orientation directly from the image, providing an end-to-end solution. In the study by [24], the orientation is classified into eight bins and then regressed within those bins for a finer estimate. These methods employ straightforward network structures, leading to models that achieve optimal performance in environments similar to the training data used. Yu et al. [25] present models that detect keypoints and infer orientation based on the spatial arrangement of these keypoints, effectively handling occlusions. Estimating orientation directly from images is valid, as it simplifies the annotation process for training datasets and enhances performance by concentrating on the orientation estimation task [26]. Considering the strengths and constraints of these methods, combining HRNet and ResNet provides a promising solution. HRNet excels in keypoint detection, and ResNet is suited for global feature extraction and orientation estimation. This combined method offers reliable orientation estimation, maintaining precision even when objects are partially occluded.

3D pose estimation is crucial for determining objects’ spatial orientation and position. Current leading methods in this domain primarily utilize deep learning. These approaches employ neural networks to determine the pose of objects from the input images. PoseCNN [27] predicts the 3D rotation and translation directly from images, suitable for single-object scenarios but struggles with complex backgrounds and occlusions. It also requires extensive, precisely annotated 3D pose data, which can be costly and time-consuming to prepare. DeepIM [28] improves initial pose estimates through an iterative process, offering significant precision advantages. However, its high computational cost and slow processing speed make it unsuitable for real-time applications. DeepIM also heavily relies on accurate initial pose estimations. PVNet [29] uses keypoints and a voting mechanism to estimate poses, handling partial occlusions and various viewing angles effectively. However, it demands precisely annotated 2D image positions and corresponding 3D spatial locations, which can be challenging with data collection. NOCS [30] provides an end-to-end solution by mapping objects into a normalized coordinate space, facilitating viewpoint and size-independent representations. While NOCS excels at generalizing to unseen object categories, its reliance on detailed 3D models and precise data alignment during training limits its practical use. MonoLoc [31] utilizes a neural network to identify keypoints within an image, then employs these calculated 2D keypoint positions to get the object’s 3D location through a multi-task neural network. EPro-PnP [32] infers the object’s pose using a 3D bounding box incorporating learnable 2D-3D correspondences. The training datasets for these methods often require complex data collection and annotation, which increases implementation costs. Furthermore, these datasets usually need complete observation of the objects. However, in environments like airports, luggage trolleys are frequently only partially visible, making these methods challenging to handle effectively.

Compared to these methods, our approach separately detects the position and orientation of the luggage trolley. It uses a keypoints detector to provide 2D information for model-based 3D location processing and an orientation detector to estimate the orientation probability for Gaussian regression analysis. The main advantages of our approach include: 1) Utilizing 2D detectors simplifies the 3D pose estimation process by relying solely on RGB image data and offers improved robustness against partial occlusion compared to deep learning-based methods. 2) Our model-based process module efficiently uses a predefined model of the luggage trolley to compute position from well-detected 2D keypoints. 3) Our method decreases the requirement for dataset preparation, streamlines implementation, and enhances processing speed, making it well-suited for real-time applications, especially in robotic systems that are resource-constrained and time-sensitive.

Given an input image $I$ with dimensions $W\times H\times 3$, the idle luggage trolley is identified, generating bounding boxes $[x_{min},y_{min},x_{max},y_{max}]$. From these bounding boxes, we can derive 2D coordinates of the keypoints on the image plane and a probability distribution of the luggage trolley’s orientation. Specifically, YOLOV5 [33] is employed for real-time luggage trolley detection due to its effectiveness in object detection. The next step involves cropping the image within the bounding box to focus solely on the luggage trolley. This produces a cropped image $I_{c}\in\mathbb{R}^{W_{c}\times H_{c}\times 3}$.

Inspiration by human pose estimation, we use HRNet to predict heatmaps of six 2D keypoint (as shown in Fig. 3) coordinates $\bm{p}_{i}=[x_{i},y_{i}]^{T}(i=0,1,...,5)$, leading to the generation of homogeneous 2D keypoints coordinates $\bm{\hat{p}}_{i}=[u_{i},v_{i},1]^{T}(i=0,1,...,5)$. The HRNet and ResNet are combined [34] for orientation detection. The cropped images pass through a backbone network, acting as a feature extractor. These features are then combined and processed via additional residual layers, culminating in a fully connected layer and a softmax layer. The outcome includes $n$ unit orientations, ${\vartheta}(j)=[\vartheta(0),\vartheta(1),...,\vartheta(n-1)](\sum_{j=0}^{j=n-1}{\vartheta}(j)=1.0)$, each indicating the probability of corresponding orientation unit that best represents the luggage trolley’s orientation in the image.

The keypoints detection network identifies the 2D coordinates of the keypoints on the image plane. Assuming the center point of the luggage trolley is on the ground, a coordinate system is established with this center point as the origin. From this reference, the 3D coordinates of the keypoints are determined. With the known 3D heights of the keypoints corresponding to the center point, their precise 3D positions can be calculated [35]. The luggage trolley’s prior model is defined as follows:

where $\bm{\mathcal{Y}}_{i}$ represents the $i$-th keypoint’s 3D position relative to the center point, consisting of coordinates $x_{i},y_{i}$ and height $\zeta_{i}$. We define the position of the luggage trolley as the center point, $\bm{\mathcal{Y}}\in\mathbb{R}^{3}$, within the camera coordinate frame, satisfying the ground plane constraint [36]:

where $\bm{\mathcal{N}}$ ($\bm{\mathcal{N}}\in\mathbb{R}^{3}$) indicates the normal vector to the ground, and $\lambda$ ($\lambda>0$) is the distance from the camera’s optical center perpendicular to the ground, as shown in Fig. 3.

Based on the prior model, the 3D position of visible keypoints in any input image is determined as follows:

The image coordinates of the $i$-th keypoint can be obtained through the keypoints detection network, creating homogeneous coordinates $\bm{\hat{p}}_{i}$. By multiplying the homogeneous coordinates with the inverse of the camera’s intrinsic matrix $\bm{\mathcal{K}}^{-1}$, getting the ray $\bm{\rho}$ that passes through the $i$-th keypoint. Using the geometric relation between the camera’s height $\lambda$ and the height of the $i$-th keypoint $\zeta_{i}$, the 3D position of this keypoint $\bm{\mathcal{X}}_{i}$ is determined. Once the coordinates of $i$-th keypoint $\bm{\mathcal{X}}_{i}$ are calculated, and with the prior model $\bm{\mathcal{M}}$, the center point coordinates of the luggage trolley are derived using the following formula:

where $\bm{\mathcal{C}}$ represents the center point coordinates of the luggage trolley, $N_{vis}$ is the count of visible keypoints, $\bm{\mathcal{X}}_{i,vis}$ refers to the 3D position of the $i$-th visible keypoint and $\bm{\mathcal{Y}}_{i,vis}$ corresponds to the $i$-th visible keypoint in prior model.

The orientation detection network infers possible orientations for the luggage trolley. The number of possible orientations corresponds to the degree of discreteness. After conducting comparative experiments, this article adopts 360 divisions based on findings demonstrating optimal orientation accuracy. Details of the experimental results that support this decision are provided in Tab. II. ${\vartheta}(j)$ indicates the probability that the luggage trolley’s orientation is within the $j$-th unit orientation. This means the orientation falls in the range of $\{\theta\mid j*1^{\circ}-0.5^{\circ}\leq\theta\leq j*1^{\circ}+0.5^{\circ}\}$. The loss function for ${\vartheta}(j)$ is defined as follows:

where $\psi(\mu,\sigma)$ represents the “circular” Gaussian probability, a representation of the ground truth, shown in Fig. 4 (orange curve):

where $\tau$ is the ground truth orientation. This process predicts a Gaussian function centered on the accurate unit orientation. The idea is that a unit orientation closer to the true orientation gets a higher probability score from the model. Therefore, the final orientation, $\theta$, is determined by the highest probability score, represented as:

To ensure the accuracy and stability of the luggage trolley’s central coordinates, it is essential to implement filtering based on the coordinates of the visible keypoints. In this article, the Modified Moving Average Filter (MMAF) is employed, which dynamically adapts to incoming data points, focusing on mitigating the impact of outliers. The core process is described as follows:

1. 1.

   Initialization of the Filter Parameters:

   $$\bm{\mathcal{F}}=(\Delta,\Theta_{z}),$$

   (8)

   where $\Delta$ represents the window size for the moving average, and $\Theta_{z}$ denotes the threshold for z-score outlier detection.

2. 2.

   Dynamic Adaptation and Outlier Filtering: Given a sequence of data points ${\mathcal{O}}_{i}=[x_{i},y_{i},\theta_{i}]\quad(i=1,2,\ldots,N)$, the MMAF updates its state by:

   $${\mathcal{Z}}({\mathcal{O}}_{i})=\frac{|{\mathcal{O}}_{i}-{\mu}_{{\Delta}}|}{{\sigma}_{\Delta}},$$

   (9)

   where ${\mu}_{\Delta}$ and ${\sigma}_{\Delta}$ denoting the mean and standard deviation of the points within the window $\Delta$, respectively. Computing the updated moving average excluding outliers:

   $$\bar{{\mathcal{O}}}=\frac{1}{|N_{{\mathcal{Q}}}|}\sum_{{\mathcal{O}}_{i}\in{\mathcal{Q}}}{\mathcal{O}}_{i},\forall{\mathcal{O}}_{i}\in{\mathcal{Q}},{\mathcal{Z}}({\mathcal{O}}_{i})\leq\Theta_{z},$$

   (10)

   where ${\mathcal{Q}}$ is the set of data points not considered outliers, and $N_{{\mathcal{Q}}}$ represents the count of ${\mathcal{Q}}$.

3. 3.

   Dynamic Adjustment:

   $$\bar{{\mathcal{O}}}_{f}=\begin{cases}\bar{{\mathcal{O}}},&\text{if }|{\mathcal{Q}}|>0\\ {\mathcal{O}}_{\text{latest}},&\text{otherwise}\end{cases}.$$

   (11)

   This equation signifies the filter’s output is the average of non-outlier points if available; otherwise, it defaults to the latest point.

Using this filter, the luggage trolley’s pose is progressively updated by effectively observing visible keypoints. Once the luggage trolley’s pose is determined, the Motion Planner takes over the planning task. The Multi-Risk-RRT [37], our previous work, integrates Multi-directional Searching with Heuristic Sampling. This approach efficiently incorporates heuristic information from dynamic sub-trees into the rooted tree, overcoming the constraints of TBVP solvers. The Multi-Risk-RRT algorithm improves motion planning in static and dynamic environments, enabling the robot to receive control instructions based on the planned trajectory.

where $\upsilon,\omega\in\mathbb{R}^{2}$ are the linear and angular velocity of the robot, $x_{f},y_{f}\in\mathbb{R}^{2},\theta_{f}\in[0,2\pi)$ are the pose of the luggage trolley, and $Map$ is the environment representation.

A detailed dataset of luggage trolleys is created to make the HPPS more accurate. The dataset comprises 13740 images featuring diverse backgrounds, lighting conditions, viewpoints, occlusion levels, and different luggage trolley states. Each image is labeled with 2D bounding boxes, six keypoints, an orientation angle, and an indicator to determine the usage states of the luggage trolley. This dataset is gathered in indoor and outdoor environments to enhance detection robustness. Recognizing that airports often have many luggage trolleys, this dataset includes images categorized by the number of luggage trolleys. To reflect real-world airport challenges, images capture luggage trolleys from various angles and include scenarios where luggage trolleys are either idle or occupied. As shown in Fig. 6, the inner circle highlights the contrast between indoor and outdoor environments, while the outer circle shows images based on the number of luggage trolleys. The label “0” means no luggage trolley is in the image, and a bar chart shows the type distribution of images with occupied luggage trolleys under different occlusion conditions. The red, blue, and green boxes in each bar represent three different luggage trolleys. The transparency of each color indicates the luggage trolley’s visibility level: high transparency shows visibility under 40%, medium transparency denotes visibility between 40% and 80%, and low transparency indicates visibility over 80%. Crosses in various colors on the boxes mark the occupied luggage trolleys. The dataset classifies the visibility and occupancy conditions of the luggage carts into 84 different categories.

The dataset is divided into three parts: 80% for training, 10% for validation, and 10% for testing. Training is performed offline using PyTorch on an AMD EPYC MILAN 7413 CPU and an NVIDIA RTX A6000 GPU. The 2D detection network builds on the official YOLOV5 [33] code. For training, the SGD [38] optimizer assists for 300 epochs with a batch size 16. The training results are shown in Tab. I, which evaluates 1374 images across several metrics. The results show the training model effectively identifies whether luggage trolleys are idle or occupied and accurately determines their bounding box.

HRNet is utilized to detect 2D keypoints, and orientation detection is achieved through a combination of HRNet and ResNet. Training results ranging from 72 equal divisions to 360 equal divisions are analyzed to compare the relationship between angle discrete division and accuracy. Each training is conducted for 100 epochs with a batch size of 64. The training results are presented in Tab. II. Bins refer to equally spaced segments of the circle’s 360 degrees. For example, dividing a circle into 72 equal parts means each bin corresponds to an angle of 5 degrees (360° / 72). Average degree error (ADE) indicates the average error in degrees. Acc.-5°, Acc.-15°, and Acc.-30° specify the rate at which predictions fall within the 5°, 15°, and 30° error margin, respectively. Keypoint detection accuracy (KDA) reflects the model’s performance in identifying and accurately locating keypoints within images. It is evident that as the division becomes finer, the average error of the angle gradually decreases. From the comparison results, a division into 360 equal bins is selected, achieving an average degree error of less than 3° and a keypoint detection accuracy of 99.4%.

To enhance clarity, Fig. 7 displays two types of images: the left column for outdoor environments and the right column for indoor environments. The first row illustrates the detection and classification of luggage trolleys, with “yes” indicating occupied and “no” signifying idle. The second row highlights the detection of visible keypoints on the luggage trolleys, each marked with a white circle and a corresponding number. The third row displays the orientation of luggage trolleys, illustrated by a red line within a 360-degree circle.

As shown in Fig. 8, this article employs a robot designed for luggage trolley collection. The robot measures 0.45 m $\times$ 0.416 m $\times$ 1.2 m, while the luggage trolley dimensions are 0.79 m $\times$ 0.525 m $\times$ 1.01 m. This robot features advanced sensors like LiDAR (Velodyne VLP-16) and a camera (Realsense D435i) powered by an onboard computer. It employs a specialized manipulator for the efficient collection of the luggage trolley. The algorithms integrate with the Robot Operating System (ROS) and function on the robot’s onboard computer in real-time. This system is powered by an i7-1165G7 CPU and an NVIDIA GTX 2060 GPU.