LWIRPOSE: A novel LWIR Thermal Image Dataset and Benchmark

Human pose estimation using RGB images has received extensive attention in recent years [2, 3, 4, 5]. Several datasets [6, 7] in the RGB domain are now available in the public domain, which has been widely used by these algorithms to develop 2D human pose estimation (2DHPE) algorithms. However, research involving thermal data remains scarce due to the limited availability of publicly accessible datasets. This work addresses this gap by introducing a novel dataset specifically designed for thermal pose estimation.

While a few thermal pose datasets exist, they have limitations. Chen et al. [8] proposed a thermal and visible image pose estimation dataset in indoor environments. They collected nearly 24,000 images and annotated the thermal images using the pose points obtained by running state-of-the-art (SotA) models on RGB images. The same pose points are used as ground truth for the thermal image after scaling. However, the dataset has certain limitations. Firstly, the resolution of the thermal images is very low, and the pose annotation is not properly supervised. Smith et al. [9] proposed the UCH-ThermalPose dataset, containing both indoor and outdoor thermal images with annotated key points. However, its size is relatively small, and it only targets static poses. The dataset contains nearly 600 images collected from different thermal image datasets and manually annotated by the authors. Our dataset offers significant advancements over these datasets. It features:

• •

  Larger size: Over 2,400 high-resolution RGB and IR images.

• •

  Diverse activities: Actors performing various daily actions (walking, sitting, eating, etc.).

• •

  Near-paired RGB counterparts: Facilitate comparison and analysis.

Due to the availability of high-quality annotated 2D human pose estimation data in the RGB domain, several algorithms that provide state-of-the-art results were developed. Simple Baseline [10] utilizes a ResNet architecture to extract features and uses deconvolutional layers to estimate poses. HR-Net [11] has proposed the generation of heatmaps using multi-resolution connection subnetworks in parallel and fusing features at multi-scale. HRNet and its variants have been widely used in 2D human pose estimation tasks. OpenPose [12, 4] is another state-of-the-art model that uses part affinity fields (PAF) for heatmap-based keypoint coordinate grouping. Different vision transformer-based [13] models were developed for pose estimation with the advent of transformer-based networks. ViT Pose [3, 2] have utilized [13] to extract feature maps of a person present, then a CNN-based decoder was used to estimate the 2D pose points.

There has been a very limited number of works in thermal human pose estimation. [9] proposed a dataset and, along with it, established a baseline using existing RGB 2D human pose estimation models. [8] proposed a CNN-based feature extraction and PAF-based decoding network for thermal pose estimation.

In this work, we also provide a baseline for our dataset by running different existing RGB-based 2D pose estimation networks.

We leverage a Seek thermal camera with long-wave infrared sensor setup operating at 7-14mm to capture thermal signatures of human poses at a resolution of 640x480 pixels, ensuring detailed representation of each subjects. With the help of advanced features of the camera, we were able to capture both RGB and their corresponding IR images. Each image within the dataset is meticulously annotated with 17 keypoints, precisely pinpointing major body joints such as head, shoulders, elbows, wrists, hips, knees, and ankles. This comprehensive set of pose points enables accurate estimation of various postures and movements, laying the groundwork for robust model training and evaluation. Moreover, to enhance the dataset’s diversity and real-world applicability, we captured images across 12 distinct activity classes like ”Discussion,” ”Smoking,” and ”Walking.” Random pose variations were captured within each class, resulting in a comprehensive representation of human movement.

While our dataset included both RGB and corresponding infrared (IR) images, directly applying the HRNet model [11] to IR images yielded unsatisfactory pose estimation results. To overcome this challenge, we adopted a two-step approach. First, we passed the RGB images through the HRNet model [11], obtaining accurate keypoint predictions. Subsequently, we transferred these predicted keypoints to their corresponding IR images. These key points were inaccurate as there was some mismatch in the registration of the RGB-thermal image pair. Further, many RGB-IR image pairs were not naturally captured together, necessitating manual annotation of the entire dataset. We developed a custom annotation tool to streamline this process, hence achieving robust pose ground truth in the thermal domain.

We captured the whole data in a controlled indoor environment in which the camera remained fixed at a distance of 7 meters for every object, ensuring consistent thermal signature capture across the dataset. This approach of leveraging RGB information combined with custom annotation and controlled data acquisition enabled us to create a high-quality IR pose estimation dataset, paving the way for further research in this domain. Table 2 represents the number of images for each activity.

The collected and annotated data was processed to train the existing RGB-based 2D HPE models. We have segregated the training and testing sets based on the subjects. The Training images included images from subjects S2, S3, S4, S5, S6 and S7 performing various activities. Images of subject S1 are preserved for testing purposes. Since the data size is small, we have omitted the validation set. Random samples from the training set were used to validate the model. The results demonstrated in the paper were obtained on the testing set. The split between the training and testing data and the number of images for each set is presented in Table 2.

The models are trained and evaluated using Mean Per Joint Position Error (MPJPE) loss in pixels, widely used in the Pose estimation tasks. MPJPE is the mean Euclidean distance between the GT 2D Pose $p_{k,i}$ and estimated 2D Pose $\hat{p}_{k,i}$. To pre-train the encoder and decoder of the ResNet, the loss was calculated as:

Where n is the number of joints in a pose. Lower MPJPE represents better model pose point predictions.

Further, the Percentage of correct key points (PCKh) metric was also used to determine the accuracy of the localization of different critical points with a given threshold. Specifically, [email protected] is used as a threshold, which is 50% of the head bone link. A higher PCK value represents better model performance.

The evaluation of various pose estimation models was conducted on our proposed dataset, employing two standard metrics: Mean Per Joint Position Error (MPJPE) and Percentage of Correct Keypoints (PCKh). Table 3 summarizes the MPJPE errors for each activity category, along with the average MPJPE across all activities. Notably, the MPJPE of ViTPose [3] exhibited a considerable reduction compared to the baseline ResNet-50 model [10], indicating superior accuracy in pose estimation. Further, [email protected] metric was used for evaluation. Table 4 shows the PCKh value for different models. There is a noticeable enhancement in [email protected] metrics with the ViTPose [3] model, indicating improved keypoint localization capabilities with attention networks.

However, it was observed that poses characterized by self-occlusion, such as sitting and talking, yielded larger MPJPE values compared to poses with minimal occlusion. This underscores the inherent complexity of our dataset, particularly with respect to self-occlusion and occlusion by external objects. Consequently, there exists substantial room for the development of models specifically tailored to address such challenging poses present in our dataset.

Figure 3 presents the visual results obtained from different pose estimation models applied to our proposed dataset. While the models were able to extract relevant features and formulate the pose structure, their performance fell short of expectations compared to their performance on RGB images. Additionally, Figure 2 showcases the failed cases of ViTPose [3] on images characterized by high occlusion levels.

The challenges encountered in pose estimation on Long Wave Infrared (LWIR) images can be attributed to the inherent differences in texture and contrast compared to RGB images. LWIR images depict thermal intensity rather than visual appearance, rendering occluded poses particularly challenging to identify. Consequently, accurate pose estimation in LWIR images is hindered by the distinct characteristics of thermal imagery.

The evaluation results shed light on the performance and limitations of various pose estimation models when applied to LWIR images in our dataset. These insights guide future research endeavours to improve pose estimation accuracy in challenging conditions characterized by self-occlusion and occlusion in LWIR imagery.