Diffusion Models Enable Zero-Shot Pose Estimation for Lower-Limb Prosthetic Users

The video search was conducted by a single researcher with terms include: gait, walking, amputee, transtibial, and transfemoral. A total of 16 videos containing two subjects with different amputation levels; a female transfemoral and a male transtibial amputee, were downloaded. Essentially, each subject has two sets of four videos (Supplementary Fig. 1), recorded at different camera positions (i.e., anterior, posterior, left and right). Both subjects are amputees on the right side. As the source of videos was publicly available, the etiology for amputation, time since amputation and type of prosthetic used were unknown. The videos were downloaded at a resolution of 1280 x 720 and 30 frames per second. Subsequently, each video was trimmed to 6 seconds (180 frames) as the initial videos included the subject changing walking direction, before following the zero-shot pose estimation pipeline.

The pipeline for zero-shot pose estimation is as follows. Given a video, for each frame, we first apply edge detection using Canny edge detection to generate an edge map. The edge map is then used as the conditional control into a text-to-image generative diffusion model. In this case, we use ControlNet proposed by Zhang & Agrawala [31] which applies conditional controls using inputs such as edge maps, segmentation maps, and keypoints to pretrained large diffusion models to support additional input conditions other than prompt text. The generated image by the diffusion model conforms to the edge map to a large extent but transforms the prosthetic limbs into able-bodied limbs. The generated image is then passed to a standard pose estimation model, for example, OpenPose, trained on large human pose estimation datasets to output the pixel locations of body keypoints. The frames were rescaled to 512 x 512 to adhere to the default ControlNet configuration.

Image generative diffusion models are currently state-of-the-art models for generating synthetic images. They are inspired by the diffusion process where molecules move from high to low-concentration areas, leading to a homogenization of the distribution over time. In image generation, the diffusion process is applied to the image pixels, where random noise is added over a number of time steps until the image is indistinguishable from random Gaussian noise.

The diffusion process is typically implemented using a sequence of invertible transformations, such as Gaussian diffusion or Langevin dynamics. During training, the model learns to reverse the diffusion process given the time step and the noisy image at that time step. During inference, new images can be generated by sampling from random Gaussian noise and applying the learnt reverse diffusion step-wise to obtain a clean generated image.

To generate images conditioned upon other features such as class, text description, or sample image, the conditioning information is provided to the diffusion model as an additional input. The conditioning information is typically encoded into a fixed-length vector representation using a separate neural network, such as a text encoder, which is then used as an input to the diffusion model.

During training, the diffusion model is trained to maximize the log-likelihood of the observed data given the conditioning information. During inference, the conditioned diffusion model is used to generate images by sampling from the diffusion process given the conditioning information. The conditioning information is used to initialize the diffusion process and to guide the sampling process at each time step.

Overall, conditioning diffusion models for image generation involves modifying the training and inference procedures to incorporate the conditioning information, and training a separate network to encode the conditioning information into a fixed-length vector that can be used as input to the diffusion model.

ControlNet is a neural network architecture proposed by [31] that applies task-specific controls to large image diffusion models such as Stable Diffusion. ControlNet does so by changing the inputs of diffusion network blocks. ControlNet makes a copy of the network block and applies a 1x1 convolution layer before and after the block. The corresponding diffusion model network block weights are frozen during training to avoid overfitting to the smaller conditional training set and preserve the model’s pre-trained high-quality image generation capabilities. The control condition is fed to the first 1x1 convolution layer and the output of that layer is added to the original input of the network block. This modified input is then passed through the network block and the subsequent 1x1 convolution layer. The resulting output is added to the original output of the network block [31].

On the Stable Diffusion model, which follows a U-Net architecture, ControlNet is implemented to control each level of the U-Net. It creates trainable copies of the 12 encoding blocks and 1 middle block of Stable Diffusion. The outputs of the ControlNet are added to the 12 skip-connections and 1 middle block of the U-Net. We refer interested readers to the detailed explanation in the original paper.

ControlNets have been trained to perform tasks based on various conditions, this includes generating images that are controlled by Canny edges, Hough lines, HED edges, human pose keypoints, segmentation maps, depth maps and more.

Specifically relevant to this application, the Canny edge conditioned ControlNet was trained with edge maps that were obtained by processing 3M images (with captions) using the Canny edge detector. The positive prompts used to generate images for this study were, “an able-body person walking, intact lower limbs, 2 legs, full-body portrait, realistic”, while the negative prompts were, “cyborg, amputee, panfuturism”. The code for ControlNet is created by [31] and available at https://github.com/lllyasviel/ControlNet.

Human pose estimation models aim to estimate the pose of humans in 2D images, typically by detecting body keypoints. Deep learning has led to significant advances in human pose estimation, and models such as OpenPose and AlphaPose, which have been trained on large datasets (COCO human keypoints dataset), demonstrated good performance on various benchmarks. Several studies also investigated the accuracy of these pretrained pose estimation models for clinical applications including gait analysis. Here we introduce OpenPose in more detail, which is the widely used model in pose estimation for clinical and biomechanics studies [34, 28, 35, 36].

The OpenPose model uses a multi-stage convolutional neural network (CNN) architecture to process images and estimate human poses [24]. The keypoint detection is performed in two stages: the first stage generates a coarse heatmap of body keypoints and parts affinity field, and the second stage refines the heatmap to produce a more accurate estimate of the keypoint locations. The model then uses the key point heatmap and parts affinity field to link up the keypoints into the pose skeleton. The model can detect various body keypoints, including those of the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. OpenPose is designed to work in real-time and is capable of estimating human poses in images or videos with multiple people, even in complex scenes with occlusions and overlapping body parts. It is also able to estimate the pose of people in different orientations and viewpoints.

Another pose estimation model used in this study was DeepLabCut (DLC) version 2.3.0. DLC allows users to label and train with their own key points. Its architecture consists of a residual neural network (ResNet-50) with deep convolutional and deconvolutional neural network layers to predict the keypoints using feature detection. We train a DLC model using videos from the transtibial amputee. Ground truth labels for 8 lower-body keypoints (left and right of hip, knee, ankle and toes) were obtained by manual annotation of every frame for one set of the transtibial videos. Using the labels of these 4 videos as ground truth, we evaluated the accuracy of the custom model (using DLC) trained on 20 labelled frames until the training loss had plateaued.

The data, expressed in mean (and standard deviation in parenthesis), are reported in pixels due to unknown measurements and the lack of a calibration procedure in the video. Results from our preliminary study were deemed satisfactory which showed that the custom model had a mean absolute error (MAE) of 7.89 $\pm$ 3.11 pixels for the coordinates (Supplementary Table 5) and 1.61^∘ $\pm$ 1. 19^∘ for kinematics when evaluated on one set of transtibial videos (Supplementary Table 6). Therefore, a custom model following the same steps was taken to obtain the ground truth coordinates for the other 12 videos. The resulting keypoints were visually examined frame by frame to ensure accuracy before we effectively treated as manual labels.

In certain frames, the visibility of keypoints may be occluded leading to lower likelihood scores given by OpenPose. Consequently, frames with a likelihood below 0.50 were removed. Cubic spline interpolation was applied to address the resulting gaps in the data, and a low-pass Butterworth filter (4th order, 6 Hz) was subsequently applied to attenuate high-frequency noise. These procedures have also been used in previous studies [37, 26].

Assuming that the left and right camera views are positioned perpendicularly relative to the sagittal plane, lower-limb joint angles of the limb closest to the camera can be calculated using the keypoints coordinates via inverse kinematics [10, 28, 26, 27]. The hip angle was calculated involving the keypoint vectors of the hip, knee and a relative vertical line to the hip. Similarly, the knee angle was computed by using the keypoint vectors of the hip, knee and ankle. In both cases, a positive value indicates flexion while a negative value indicates extension. The ankle angle was calculated using the keypoints vectors of the knee, ankle and toe. In addition, the ankle angle is defined by the foot with respect to a 90^∘ line to the tibia. The kinematic data for the entire time-series was initially calculated, followed by time normalization from 0-100% of the gait cycles. Gait cycles were defined as consecutive occurrences of heel strikes by the same foot, which were manually identified. There were at least 4 gait cycles observed for all videos.

To quantify the error of pose estimation, we use the MAE which is the mean Euclidean distance between ground truth and predicted keypoints. Both coordinates and kinematics data were quantified using MAE. There are several previous works in the literature that quantify the accuracy of markerless pose estimation models such as OpenPose for the purpose of gait analysis. Supplementary Table 4 summarizes existing data reported on the accuracy of clinical gait applications of markerless pose estimation on able-bodied individuals. Original values reported are used if provided in the papers, otherwise, the values were estimated by reading off from figures.

Our findings reveal that when OpenPose is directly applied to the original image, a substantial proportion of frames exhibit either an inability to identify or a conspicuous misidentification of lower-limb keypoints (Fig. 2). Supplementary Table 1 provides a detailed breakdown of these occurrences. Specifically, transfemoral images display a higher incidence of such challenges compared to their transtibial counterparts. Notably, our zero-shot method demonstrates remarkable efficacy in mitigating the number of affected frames across all camera perspectives. It is also important to note that the majority of observed keypoint challenges are concentrated in the sagittal camera view, particularly when the prosthetic limb is in closer proximity to the camera.

The OpenPose method exhibited coordinate errors that were about twice as large (Table 1) on the prosthetic limb (right) compared to the intact limb (left). A detailed breakdown of the individual keypoints coordinates errors can be found in Supplementary Table 2.

The zero-shot method consistently outperformed the OpenPose method in estimating keypoints on the prosthetic limb by a large margin for joint coordinates. The mean absolute error (MAE) measured for joint coordinates decreases by 37% for transtibial prosthetic limbs and 76% for transfemoral prosthetic limbs. Even though the OpenPose method yielded slightly lower coordinate errors on the intact limb, the differences observed were relatively minor, with a mean difference of less than 2 pixels.

Similarly for joint kinematics, the errors against ground truth using the zero-shot method consistently outperformed the OpenPose method on the prosthetic limb by a large margin. The improvement is particularly noteworthy for ankle angles and transfemoral knee angles. The comparison of joint kinematics is shown in Fig. 4.

Although the open-source videos lacked explicit calibration or measurement information, an estimate of the subject’s height enabled us to approximate the metric error. Assuming a subject height of approximately 180 cm, we deduced that 1 pixel corresponded to approximately 0.3 cm. These assumptions enabled us to draw comparisons with previous findings. Our results for OpenPose on subjects with intact limbs are consistent with existing literature (Supplementary Table 4). Clinicians aiming to replicate our study may consider incorporating calibration steps at the beginning of the video recording. This would allow them to gather additional gait parameters, such as stride length and step width, which were not analyzed in our study due to the utilization of open-source videos.

The two main types of lower-limb prosthetics, transfemoral and transtibial differ in appearance and in characteristics of gait. The performance of pose estimation and gait analysis on transfemoral and transtibial amputees are compared.

As observed in Table 1, when comparing transfemoral and transtibial amputees, the pose estimation of the prosthetic limb exhibited poorer performance in the transfemoral amputee. Even with the application of the zero-shot method, the coordinate errors on the prosthetic limb were twice as substantial in the transfemoral amputee. Whereas in the transtibial amputee, the zero-shot method was able to reduce the coordinate errors on the prosthetic limb to a comparable level as the intact limb.

The hip angle curve exhibited satisfactory results for both transtibial and transfemoral amputees when employing both the OpenPose and zero-shot methods. Particularly, the zero-shot method displayed the ability to reduce the MAE on the prosthetic limb for the transfemoral condition by about 3 degrees.

In the knee angle, the OpenPose method was able to replicate the kinematic curvature throughout the gait cycle in the transtibial amputee (Fig. 3) but less so in the transfemoral amputee (Fig. 3). Thereafter, the application of the zero-shot method resulted in an improvement in replicating the knee angle kinematic throughout the gait cycle for the transfemoral amputee (Fig. 3).

In comparison to the hip and knee angles measured using OpenPose, the ankle angle exhibited higher error values. This observation is further supported by the higher error in ankle keypoint coordinate (Supplementary Table 3). When the zero-shot method was applied, a slight improvement in replicating the ankle kinematic curve for the transtibial amputee was observed, but no improvement was evident for the transfemoral amputee.

Previous studies have indicated significant kinematic differences between prosthetic and intact limbs in lower-limb amputees [38]. Notably, the kinematic gait cycle observed in our study exhibited a similar pattern to those reported in previous studies [39, 2, 40].

For instance, as seen in Fig. 4, our zero-shot approach revealed that the hip extension of the transtibial amputee was reduced. Additionally, within the 10-50% range of the gait cycle, a decreased range of motion was observed in the knee angle. In the case of the transfemoral amputee as seen in Fig. 5, a noticeable phase shift in both hip and knee angles was observed, indicating a leftward displacement in the kinematic patterns throughout the gait cycle when compared to the intact limb’s kinematics. In comparison, OpenPose fails to capture some of these findings, especially in the transfemoral case due to greater inaccurate joint angle kinematics. This demonstrates the effectiveness of the zero-shot method over OpenPose for identifying and quantifying key anomalies during gait analysis and provides the means for clinicians to uncover and measure key gait markers during rehabilitation.

Gait analysis for lower-limb prosthetic users without the need for specialized equipment has the potential to transform rehabilitation for this population, by providing an automated, consistent and quantitative assessment of the presence and severity of pathological gait. This opens up the possibility of rapid clinical feedback, reliable longitudinal tracking of progress, objective comparison between rehabilitation strategies and more. The method proposed in this work is a step towards this goal through accurate pose estimation from markerless videos acquired from low-cost commercial cameras.

The results obtained in our investigation demonstrate the effectiveness of the zero-shot method in reducing both the coordinates and kinematics errors when compared to applying OpenPose on the original images. Although the ankle kinematics exhibited a relatively higher error value, the hip and knee kinematics generated using the zero-shot method remain valuable for clinicians. Such kinematic information enables clinicians to discern disparities in kinematic patterns between intact and prosthetic limbs, thereby facilitating the planning and monitoring of individualized rehabilitation programs. Presently, it is evident that OpenPose encounters difficulties in accurately detecting images featuring prosthetic limbs, with higher error rates observed for prosthetic limbs compared to intact limbs (Fig. 2). This observation aligns with the study by Cimorelli et al.[29], who similarly reported challenges when utilizing OpenPose for lower-limb amputees, consequently restricting the applicability of markerless motion capture in this population.

Our findings demonstrate that the implementation of the zero-shot method leads to enhanced keypoint detection by greatly reducing the number of problematic frames, subsequently reducing coordinate and kinematic errors. This improvement has practical implications for gait analysis, particularly in deriving kinematic angles throughout the gait cycle. This information is crucial for clinicians to comprehend the gait symmetry of individuals. From a clinical standpoint, the analysis of kinematics throughout the gait cycle provides the ability to discern asymmetrical walking patterns between the prosthetic and intact limb. When compared to OpenPose, we have shown the zero-shot method to be more effective in identifying and quantifying significant anomalies during gait analysis. By incorporating the identified anomalies and utilizing the insights gained from the zero-shot method, the clinician can design an individualized rehabilitation regimen that addresses specific gait deviations and promotes the restoration of balanced walking patterns. This personalized approach enhances the potential for successful rehabilitation outcomes in the lower-limb amputees’ population.

One drawback in the application of the zero-shot method is the increased swapping of lower-limb keypoints in the sagittal plane (Fig. 2). This occurs when the edge map generation process fails to preserve the information regarding the leg’s proximity to the camera. Humans can perceive the relative positions of objects from 2D images based on overlapping regions. For instance, if the left leg is closer to the camera, the left leg will naturally occlude parts of the right leg that are directly behind when viewing from the camera. Subsequently, when the edge detector fails to identify the outline of the leg in the region where the two legs overlap, it will result in an ambiguous edge map. This ambiguity may potentially result in a swap between the two legs in the synthetic image. Fig. 6 provides an illustrative example of the observed phenomenon. Nevertheless, this issue can be readily resolved by manually identifying and correcting the swapped frames or by utilizing other cues, such as the asynchronous movement of the ipsilateral arm and leg swing. This observation is reinforced by the finding that, although the zero-shot method requires a higher number of frames to be swapped compared to OpenPose, it still leads to a decrease in error. It is worth highlighting that the default OpenPose on the original image may also cause the lower-limb keypoints to swap, similar to previous studies [28, 41]. Moreover, the zero-shot method resulted in the reduction of frames where keypoints were not detected in OpenPose due to the non-resemblance of intact limbs. This further demonstrates the efficacy of the zero-shot method in converting the prosthetic limb to a human-like representation, thereby allowing the application of pose estimation.

It is important to acknowledge that not all frames can be successfully transformed into images resembling able-bodied individuals (Fig. 6B). This indicates certain limitations with the edge map. Nevertheless, even when comparing frames at approximately the same gait cycle, the generation of synthetic images yields inconsistent results (Fig. 6C). This inconsistency is more pronounced in the case of transfemoral amputees, likely attributed to the dissimilarity between the prosthetic limb and a human limb, posing challenges for the model in identifying the appropriate image type. Despite inputs of positive prompts into ControlNet, emphasizing the presence of two intact limbs, the model may still encounter difficulties. However, it is worth noting that the majority of frames can still be successfully transformed, as evidenced by the overall reduction in coordinate error, especially for the prosthetic limb. Another limitation pertains to the wide range of prosthetic options available in the market. Our testing of the diffusion model was limited to a specific subset of prosthetic users, which may introduce some variability in the results. Nevertheless, we anticipate that any deviations from our findings will be minor and not substantially different.

The main advantage of the proposed method is its ability to perform zero-shot pose estimation for prosthetic users without the need for any manual labelling and model training. However, the zero-shot ability comes with the limitations of slow processing speed due to the need to run inference on the large image generative diffusion model. Thus, there is a trade-off between inference time versus the time required for manual labelling for prosthetic joints and finetuning or training a custom detection model. With personal computing hardware (Nvidia GTX Titan X), the speed of generating 512 x 512 synthetic images is approximately 8 - 10 seconds per image which presents the main bottleneck for the workflow. This means for a 24 FPS video, the inference time needed is 200x that of the video length. Given a walking speed of 1 m/s, a 10 m walk test may require 10 seconds of video recording and may take more than 30 minutes to process which precludes real-time analysis and presents limitations for practical clinical usage.

To address this, we propose 2 methods to speed up inference. First, clinicians may run pose estimation on the original video to identify poorly estimated frames before employing the image-generative zero-shot workflow on the poorly estimated frames only. Second, clinicians may drop selected frames to reduce the number of images to be generated. For the purpose of gait analysis, the gait cycle is obtained by averaging over many gaits. In such use cases, it is possible to remove selected frames and perform interpolation without affecting the resultant gait cycle significantly. Nevertheless, the zero-shot method is still faster than creating a custom-trained image set which includes manual annotations and a longer training time for custom pose estimation models for each individual.

Another potential alternative for regenerating an intact limb image for lower-limb amputees is inpainting a masked section of the image with diffusion models [42]. This technique involves selectively highlighting specific parts of an image that require modification. However, this option was not explored in our study due to the continuous movement of the limbs in each image, which necessitates manual masking and thereby increases post-processing time. Future research could investigate alternative methods for automating the masking of the prosthetic limb in each frame. By generating images for smaller masked regions, it may be possible to expedite the inference time.