Neuromophic High-Frequency 3D Dancing Pose Estimation in Dynamic Environment

TORE (Baldwin et al., 2021) attempts to mimic the human retina by preserving the membrane’s potential properties. A fixed-length(e.g., $K$) First-In-First-Out (FIFO) queue is adopted to record the relative timestamp of the most recent $K$ events. When a new event comes into a pixel’s queue, its relative timestamp is pushed in, and the oldest event in the queue gets popped out. TORE computes the logarithm of these timestamps in the FIFO buffer. TORE transforms the sparse events stream into a dense and bio-inspired representation with minimum information loss and achieves SOTA in many DVS tasks(e.g., classification, denoising, human pose estimation). An exhaustive comparison of different representations could be found in (Baldwin et al., 2021).

We use a modified TORE volume representation: Normalization, 0-1 Flip, and a range scaling. Firstly, values inside the TORE volume are normalized to the range [0,1] for faster convergence. Then we flipped the normalized maximum and minimum values. By default, the oldest events and pixels with no event recorded are given the maximum value $log(\tau)$, which is counter-intuitive and may harm the convergence speed. We can easily solve this problem by letting the value $v$ in TORE be set to $log(\tau)-v$. With this modification, older events have a smaller value while newer events have a greater value, and the pixel where no event is recorded is set to 0. Lastly, because logarithm is applied in the TORE calculation, the most recent 4.63$\mu$s takes the $[0.7,1]$ range of $[0,1]$, and 4.63$\mu$s is smaller than the DVS camera’s sensitivity time (150ms is used in the original TORE paper (Baldwin et al., 2021)). So here is our modification formula:

where $v$ and $v^{\prime}$ represents the original and modified TORE value, while the notation $x|_{[}a,b]$ means the value of $x$ is clamped within the range $[a,b]$.

TORE holds a first-in-first-out queue separately for each pixel, and the value in the queue drops as time goes on. TORE can keep the history of pixels up to five seconds. These characteristics make it suitable to work in dynamic conditions. TORE can drop redundant histories on the same pixel when in high-lighting conditions, relieving the model’s computational pressure. When in low-lighting conditions, the event histories captured in the past could compensate and help decide the position of some joints that do not move significantly during the last time window and thus help solve the missing torso problem. Fig. 1 shows the details to generate the modified TORE volumes.

In order to filter out events triggered by background activities and event camera hardware noise (e.g., hot pixel and leak noise (Hu et al., 2021)), a mask prediction network is used that can predict a human body mask from the TORE volume representation.

For the mask prediction network, a modified version of U-Net (Ronneberger et al., 2015) proposed by Olaf et al. is adopted. Unlike the original version, our modified version can predict the mask for the current input frame and a series of masks after this frame. Each predicted mask is generated together with a confidence score. The primary rationale behind predicting the human body mask of future frames is that future motion trajectories of different human body parts are generally predictable with information about current and previous motion trajectories. Time-Ordered Recent Event (TORE) volume representation efficiently captures current and previous motion history. It allows the mask prediction network to estimate human body masks of the current and future frames and their corresponding confidence scores. The confidence scores provide an early-exit-like mechanism where the computational pipeline bypass mask prediction of a certain frame if the mask predicted based on a previous frame’s TORE volume achieves a high confidence score. This mechanism significantly saves computational cost and energy consumption (more results in section 7.4). For the predicted mask, the U-Net generates float numbers between 0 and 1, and a binarization process with a small threshold of $0.1$ is applied. The small threshold was used to ensure that the generated mask misses no parts of the human body since failing to cover the entirety will lead to more error than allowing a small amount of noise.

One major challenge for training the mask prediction network in an end-to-end scheme is the lack of a realistic event camera dataset with labeled human body mask sequences for recorded event streams. However, as our proposed motion-to-event simulator (described in section 5) could generate paired pixel-level human masks with a high frame rate, they can be fed into the mask prediction network to train it fully.

The human pose estimation network consists of a ResNet-based feature extractor, a Bidirectional convolutional LSTM (BiConvLSTM) layer, an HPE backbone, and a triangulation module. The head part of ResNet34 serves as the feature extractor, followed by a BiConvLSTM layer with a skip connection. As the human body does not have abrupt changes during a relatively small time window, adjacent frames usually have similar ground truth labels. This continuity in human joint movements makes it helpful to refer to neighboring frames when estimating the joints’ position. BiConvLSTM is a bidirectional version of ConvLSTM (Shi et al., 2015), where ConvLSTM is a type of recurrent neural network for spatio-temporal prediction that has convolutional structures in both the input-to-state and state-to-state transitions. Lastly, the HPE backbone is made up of three hourglass-like CNN blocks. Each block outputs a series of marginal heatmaps to reconstruct the human joints’ coordinates in 3D space. All the intermediate outputs from these three blocks are used to calculate the loss with the ground truth heatmaps, and the last two blocks could be considered refine networks. The feature extractor and the backbone network architecture have been developed based on a model proposed in (Scarpellini et al., 2021). For each joint, YeLan generates three heatmaps showing the probability of its projected position on xy, xz, and yz planes. Then a soft-argmax operator is applied to extract the normalized coordinates of the joint. Lastly, predictions from the xy plane are used as the final prediction for x and y coordinates, while values for z are calculated by averaging the yz and xz predictions.

The ground truth labels used in the training and testing are normalized before being fed to the network. For a specific joint, we first project it to a plane parallel to the camera’s image plane and have the same depth as the depth reference. The head joint’s depth value is used as the depth reference. Then the 3D space in the DVS camera’s view is mapped to a cube whose range is $[-1,1]$. Lastly, as the network does not directly predict the 3D coordinates of a joint but predicts its marginal heatmaps instead, we also extract the joints’ projection on three orthogonal faces of the normalized space cube to generate the ground truth for marginal heatmaps. The final marginal heatmaps are then calculated using a Gaussian filter on these projection images (Scarpellini et al., 2021).

For the mask prediction network, the loss function consists of three components. The first component is the Binary Cross Entropy(BCE) loss calculated between the predicted mask series $\hat{M}$ and the corresponding ground truth masks $M$. This loss is applied to guarantee the accuracy of all the generated masks. Next, a Mean Square Error(MSE) loss is calculated over the predicted confidence scores $S$ and their ground truth. The ground truth score is the Mean Absolute Error(MAE) between a predicted mask and its ground truth mask. Lastly, although we want to predict the mask series for the current frame and frames following it, they are not equally important. We need to guarantee that closer frames’ masks get weighted more, especially for the current frame. Therefore, another BCE loss is calculated between the predicted mask for the current frame ($\hat{M}_{0}$) and its corresponding ground truth $M_{0}$. The overall loss is:

We adopted the loss introduced in (Scarpellini et al., 2021) for the human pose estimation network. As the marginal heatmaps can be interpreted as probability distributions of joint locations, Jensen-Shannon Divergence(JSD) can be applied between the predicted heatmaps by each block ($\hat{H}^{i}$, where $i$ represents the block index) and the ground truth heatmaps $H$ on each projection plane ($xy,xz,zy$). Also, a geometrical loss is calculated between the reconstructed 3D joints’ coordinates $\hat{p}_{xyz}$ and their ground truth $p_{xyz}$. The loss for this stage can be written as follows:

Several significant advantages of using a synthetic data generator are highlighted by a recent work (Goyal et al., 2022). Real-world data collection is expensive and time-consuming. The cost per participant is high, limiting the sample size, the scale of scenes, and the diversity in physical and environmental characteristics. The synthetic dataset generation process is fully controllable. The human models, background scenes, movements, lighting conditions, blur scale, camera distance, and camera view angle can all be precisely controlled, making it practical to modify a specific condition while keeping all the other factors unchanged. The output video file can be rendered with a very high frame per second (FPS) without blur problems caused by under or over-exposure in dynamic lighting conditions. Lastly, the ground truth of human joints’ 3D location can be extracted inside the software for different dances and human models.

Synthetic data generation consists of RGB dance video rendering, human joints’ position extraction, and events generation. MikuMikuDance (MMD), Blender, and V2E are chosen for each step, respectively.

MMD is a freeware animation program that lets users animate and creates 3D animated movies. This software is simple but powerful, with a long history and a big open-source community behind it. Plenty of human models, scenes, and movement data can be easily accessed for free. In addition, it can automatically handle clothing physics and interaction with the body in a decent manner with minimal manual adjustment.

Software Blender is adopted to generate human joints’ ground truth labels and camera matrix. Blender is a free and open-source 3D computer graphics software for creating animated films, motion graphics, etc. It is highly customizable, and all essential information can be accessed during rendering, including the 6-degrees of freedom coordinates of human joints and the camera center. The 13 key points’ ground truth coordinates are extracted at 300 frames per second (FPS).

Lastly, Video to Event (V2E) is used for event generation. V2E is a toolset released in 2021 by Delbruck et al. (Hu et al., 2021). It can synthesize realistic event data from any conventional frame-based video using an accurate pixel model that mimics the event camera’s nonidealities. According to its author, V2E supports an extensive range of customizable parameters and is currently the only tool to model event cameras realistically under low illumination conditions.

As Fig. 3 shows, our proposed motion-to-event generator takes 3D character models, motion files, camera views, and lighting conditions as inputs. MMD renders RGB dance videos and their paired human mask videos given these inputs, while Blender generates corresponding ground truths. For each camera view, a camera intrinsic and extrinsic matrix is calculated by Blender as well.

Then our simulator combines these rendered dance videos with collected background videos by referring to the paired mask videos. According to (Hu et al., 2021), if a video’s temporal resolution is low, generated event stream will be less realistic. However, due to the software limitation and background video quality, the synthesized videos are at 60 FPS. This gap in FPS is compensated using SuperSlowMo(Jiang et al., 2018), which can interpolate videos to a high FPS with convincing results. To reduce the time and computation cost, we only apply SuperSlowMo to dance and background videos before the merging. This way, we do not have to interpolate synthesized videos for each human and background combination.

After the RGB video rendering, merged videos are sent to the V2E events stream generator. Many parameters, like event trigger thresholds, noise level, and slow-motion interpolation scale, can be modified in detail. These features enable us to generate many highly customizable DVS event streams at a meager cost in a short time. To simulate situations in the real world, we increase the noise as the brightness decreases.

For human joints’ ground truth, as mentioned above, we write our customized scripts and inject them into Blender to collect the exact position of all joints while rendering scenes at 300 FPS. Also, scripts help extract the camera’s intrinsic and extrinsic matrix used in the label pre-processing. Besides the advantages mentioned above, 3D human models have even more flexibility initially. Skin color, height, body style, clothing, hair color and style, and accessories are all easily modifiable - which is very difficult to do in real-world data collection.

With the comprehensive motion-to-event simulator, we have generated about 4 million data pieces (more specifically, 3,958,169 snippets). Examples of the synthetically generated RGB frames are shown in figure 5. The total dataset size is about three terabytes. This data was synthesized from 1320 combinations of a few different variables, including 10 human models, 8 pieces of one-minute dance motions, 11 background videos, 4 camera views (i.e., front, back, left, right), and 3 different lighting conditions (i.e., high, medium, low). This dataset contains processed TORE volume, paired labels and masks, constant-time frames with the same time step (20ms), and raw event stream files for generating any other customized representations. Due to the computational resources and training time limitation, only 30% of data instances from selected 330 setting combinations are used for training and validation. They are shuffled and divided with a ratio of 8:2. Then we randomly select 82 setting combinations from the rest 990 unused combinations as the test set.

In addition to the synthetic dataset, we have conducted an Institutional Review Board (IRB)-approved human subject study to collect a real-world human pose dataset from nine participants. Seven of them are male, and the other two are female. The participants were 20-31 years old and recruited from a university campus using a snowball sampling technique. During this study, our participants were asked to play a Nintendo dance game “Just Dance 2021”. Each participant selected five songs to dance to during the study after a short training period with the tutorial dances in the Nintendo dance game. A monitor was used to provide the participants with further guidance/cues for participants to follow along.

A 9-camera-based 3D motion capture system (Qualisys AB, Göteburg, Sweden) is used to obtain ground truth 3D kinematics data of the human body, which provides us with the 3D absolute coordinates for all selected human joints at a frame rate of 200 frames per second (FPS). An event camera (DAVIS 346 (Brandli et al., 2014)) and an RGB camera are simultaneously run to record the participants’ movements. Other necessary equipment includes a flicker-free LED light, IR filter, cardboard background, Monitor, and Nintendo Switch. During the dataset collection, the lighting condition was strictly controlled to induce low-lighting and high-lighting conditions. All the lights except a dim flicker-free lamp are turned off during the low-lighting conditions session. The IR filter is attached in front of the event camera’s lens to filter out the events caused by IR light emitted by the motion capture system. Figure 4 illustrates the data collection settings, hardware, sensor arrangements, and mechanisms to generate background noise.

In the dynamic background condition, a person behind the target participant moved randomly with a large cardboard (that depicts a bus or car). For static background cases, on the other hand, no other movable contents appear in the background during recording. On average, each participant danced for about 20 minutes, and the time was distributed equally to the four conditions sessions mentioned above. We divided the real-world dataset simply by selecting all the events generated by participants seven and eight as test sets. Participant seven is male, and eight is female. All other data is shuffled and divided into training and validation sets with the same 8:2 ratio. We prepared a short video To illustrate further the rich synthetic and real-world data, which can be found at bit.ly/yelan-research.

All models are trained on eight 1080ti or 2080ti graphic cards. The batch size is identical over all the training: Stage one has a batch size of 128, and stage two has a batch size of 16. All stages are trained with the following parameters: 0.001 learning rate, Adam optimizer with $1e{-5}$ weight decay, early stopping with the patience of 10 epochs. A learning rate scheduler is applied to halve the learning rate every $N$ epoch. $N$ is set to 5 in stage one and 10 in stage two, considering the training pattern and convergence speed difference. This project is implemented in Pytorch-Lightning. Stage one and two in YeLan are trained separately. For the mask prediction network, due to the ground truth limitation, it is only trained on synthetic data and used directly in both synthetic and real-world datasets. For stage two, it is pretrained on synthetic dataset at first, and then fine-tuned on real-world dataset.

Following the convention, we used Mean Per-Joint Position Error(MPJPE), the most commonly used metric in HPE, as our primary evaluation metric. Another two popular metrics, PCK and AUC, are also compared in the major comparison. MPJPE is calculated by averaging the Euclidean distance between each predicted joint and its corresponding ground-truth joint, usually at the millimeter level. PCK stands for Percentage of Correct Keypoints according to a threshold value of 150mm, which is commonly used. If the predicted joint is within a 150mm cube centered at the ground truth joint, it is treated as correctly predicted and returns 1; otherwise, it returns 0 for the wrong case. AUC means the Area Under the Curve for the PCK metric at different thresholds. We also use the standard threshold sets, 30 evenly spaced numbers from 0 to 500mm. When calculating the AUC, we will first calculate the PCK at all these threshold values, and their mean value is the target AUC.

Where $p_{xyz}$ and $\hat{p}_{xyz}$ represent the ground truth and predicted 3D joint position, and $J$ means the number of skeleton joints. In PCK and AUC’s formula, $\alpha$ and $A$ mean the threshold and the maximum threshold in AUC calculation. $N$ in AUC stands for the number of the different thresholds used.

The baseline methods selected for this work are (Scarpellini et al., 2021) and (Baldwin et al., 2021), proposed by Scarpellini et al. and Baldwin et al. (originally published in 2021). For (Scarpellini et al., 2021), They used two event-count-based representations: constant event count frames and voxel grid, which have a variable time step and thus lose synchronization with our labels and TORE volumes. Therefore, the original work is modified to take constant time representations as input instead. The time step for this representation is set to 20 ms as well to guarantee that the labels are matched. Baldwin et al. proposed TORE volume, and they evaluated on the model and dataset proposed in (Calabrese et al., 2019) with a replaced representation to show their new representation’s superiority on multi-view human pose estimation task. Since in this paper we are interested in developing a monocular human pose estimation framework, we use the same monocular HPE model in (Scarpellini et al., 2021) with the representation switched to TORE to show the performance of (Baldwin et al., 2021) in this new task.

Table 2 illustrates the performance comparison between the proposed YeLan and baseline models on both synthetic and real-world datasets. The table shows that the baseline models consistently underperform the proposed YeLan model in both conditions. In the synthetic dataset, the model proposed by the Scarpellini et al. (Scarpellini et al., 2021) achieves an MPJPE and PCK of respectively 91.88 and 83.92% while the model proposed by the Baldwin et al. (Baldwin et al., 2021) with TORE representations have a slightly improved performance of respectively 59.34 MPJPE and 93.17% PCK. The proposed YeLan system outperforms the baseline models by a significant margin and achieves the lowest MPJPE of 46.57 and the highest PCK of 96.34% on the synthetic dataset.

Compared to the synthetic data, the performance of all the models on the real-world data is significantly worse. For example, while our proposed model (trained and tested on the same type of data) achieves an MPJPE of 46.57 with the synthetic data, the performance in the real-world dataset is 96.61 MPJPE. The real-world data comes with its own challenges and unique characteristics. Real human motion trajectories are slightly different from simulated trajectories. There is also substantial heterogeneity in the skeletal formations and impedance matching states in the real-world data. Lastly, another researcher is inside the view generating background activities, who sometimes get recognized and masked as an additional human, which could confuse the models. Consequently, the performance of the real-world data will be expected to be lower than that of the synthetic data. However, the proposed YeLan still achieves the lowest metrics compared to the baseline models. The baseline models are designed to work in ideal environments which do not suffer from dynamic lighting conditions and moving background content problems, which contributes to their poor performance in more realistic environments.

As Fig. 7 shows, YeLan has strong stability over the changes in lighting conditions and camera views, while more complicated background contents cast an impact on the results. Different human models also show an impact on performance. From the statistics on human models, we know that human models “Albedo”, “Seele,” and “Xiangling” performs less well than other human models. By looking at the original 3D model and test data composition, the reason becomes obvious: Albedo wears a long coat reaching his knee, while Seele wears a fluffy dress with a complex structure. These factors make their joint position harder to estimate. As for Xiangling, though she does not have clothing-related problems, we find some body deformation that happened during the simulation. Also, the randomly selected test cases for Xiangling contain more difficult backgrounds and lighting combinations, such as two busy city street environments in low-lighting conditions, according to Fig. 7 (a).

Moreover, Fig. 9 shows that our model behaves better constantly in all lighting conditions compared to baseline models. This feature is significant in real-world applications as the pipeline could generate convincing predictions regardless of the lighting conditions.

The mask prediction network in YeLan enables it to reach a similar level of PCK and AUC when the background is dynamic and static. We compared on NiDHP test set with dynamic and static background cases using the model pretrained on MiDHP and fine-tuned on NiDHP. Fig. 10 depicts that YeLan get a comparable result on dynamic background cases with static ones.

As introduced in section 4, there are two most essential modules in YeLan: a mask prediction network and the BiConvLSTM. We do an ablation study by removing one module each time and comparing their performance on the synthetic dataset. If the mask prediction network is removed, the resulting MPJPE is 63.99; if the BiConvLSTM is removed, the corresponding MPJPE is 49.08. In contrast, the complete YeLan pipeline gets an MPJPE of 46.57, which proves the effectiveness of these modules.

Synthetic data from the physics-based simulator is a core contribution of this work which allows us to generate event streams from diverse virtual 3D characters in different simulated settings. They include different clothing, lighting conditions, camera viewing angle, dynamic background, and movement sequence. The generated synthetic data is faithful to physics and can provide essential data for pretraining YeLan since running real-world experiments with too many participants is expensive and time-consuming. To this end, we hypothesize that the pretraining on synthetic data will allow YeLan to learn better feature representation of physical conditions in diverse simulated settings and achieve superior performance on the real-world dataset.

Table 2 demonstrates that our proposed model achieves superior performance (concerning the MPJPE/PCK/AUC metrics) on the real-world dataset if the model is pretrained on the synthetic dataset from the simulator. The model trained only on the real-world dataset achieves 96.61 MPJPE and 81.88% PCK. On the other hand, if the model is pretrained on the synthetic data and then fine-tuned on the real-world dataset, it achieves 90.94 MPJPE and 85.14% PCK, which clearly shows the benefit of pretraining on synthetic data from a simulator.

In the real world, occlusion is also an inevitable problem. In our synthetic dataset, different types of clothes and accessories like hats, long hair, fluffy skirts, and long coats already occluded the human body to a relatively large extent. Moreover, the camera’s side views also introduced many self-occlusion. The excellent performance over all these scenarios shows a solid ability to survive occlusion.

To further prove the occlusion resiliency of the system, we augmented the dataset with more block occlusions. In the eye of the event camera, if a static object is placed in front of a human and shadows him, the corresponding area shoots no event due to the object’s immobility. Regarding the input TORE volume, the occluded area becomes pure black, as no event activity is recorded. To simulate the occlusion like this, we trained a model with the value of random areas set to zero. These rectangular occlusion areas have random sizes and locations, and the occlusion is also applied randomly with a probability of 80% during the training. When testing on the test set with random occlusion enabled, the MPJPE is 96.794. During the test, the occlusion probability is set to 100%. There is an accuracy drop, but considering that the maximum random occlusion area is $80\times 80$ (where the frame size is $260\times 346$), it makes sense as sometimes the majority of the human body could be occluded. Fig. 11 shows samples from the test set.

As is introduced in 4.2, the first stage of YeLan is an early-exit-style human mask prediction network, where a threshold $\beta$ is used to decide when to start a new inference. The selection of $\beta$ is a trade-off. If the threshold is set too high, the mask prediction process will be executed too many times, resulting in a longer inference time. On the contrary, many defective masks will be used if the threshold is too low, which harms the overall accuracy. In order to select the best threshold, we ran an experiment to do human pose estimation on a continuous one-minute event stream with a series of different thresholds. The accuracy and time consumption are recorded and made into Fig. 12. From this figure, we can observe that as we increase the threshold, accuracy goes up and time consumption goes down, while we can achieve a balanced accuracy and time consumption at near 0.99.

Moreover, to understand the error distribution across different joints, we calculated the mean MPJPE for all 13 joints. As Fig. 13 shows, the head joint has the lowest MPJPE, and the shoulder joints enjoy the second-lowest error. On the other hand, the left and right hands have the highest MPJPE, and foot joints come next. From our observation, this happens because the head and shoulders share the most stable contours across different characters and overlap less with other body parts. However, as hands and feet locates farthest from the human center and move the most during the dancing, they are harder to estimate due to more possible movements and patterns. More patterns lead to more uncertainty, which harms the accuracy even more when occlusion happens.

The total model size of YeLan is 234 MB. As for the time consumption, we run a full test on the MiDHP test set with a batch size of one, and the average time cost on each frame is about 29.1 ms when running on a 2080ti graphic card.

In order to show the superiority of DVS in low-lighting conditions, we also compare our result with RGB frame-based human pose estimation algorithm. We use the DVS camera DAVIS346, where both events and RGB frames are recorded synchronously. As the same device records both modalities via the same lens, there is no difference in the quantity of light captured by RGB and DVS. OpenPose (Cao et al., 2019) proposed and implemented by Cao et al. is adopted as our RGB baseline. OpenPose is an accurate, fast, and robust 2D human pose estimation algorithm. Although there are differences in joint number and dimension, we calculated the 2D projection of 3D joints generated by YeLan. We picked the closest 13 joints from all 25 OpenPose output joints to compare. The comparison is conducted on paired DVS and RGB recordings from the dataset collected in the motion capture lab. Both high and low-lighting condition cases are included, and qualitative comparison is shown in Fig. 14. Furthermore, we made a box plot showing the 2D MPJPE of DVS and RGB in two lighting conditions in 15. From this figure, it is evident that YeLan generates better and more stable predictions in all cases. Although the RGB-based method achieves good performance (marked by the low 2D MPJPE) in high-lighting conditions, the performance deteriorates significantly in the low lighting conditions. OpenPose fails to detect any human from frames due to motion blur and low SNR sensor data when the illumination is lower than a certain threshold. On the other hand, YeLan shows strong robustness against lighting conditions changes and constantly generates high-quality predictions (marked by the low MPJPE in both cases).

Besides the RGB camera, the depth camera is also widely used in digital dancing games. (Yun et al., 2014; Kamel Boulos, 2012) These cameras are usually paired with RGB cameras, which grant them information from both domains. Compared to RGB cameras, these cameras have a better understanding of the 3D space, which results in a more accurate human segmentation and joint location estimation. RGBD camera-based human pose estimation is a well-established problem with many commercial products and pipelines (Rallis et al., 2018; Kitsikidis et al., 2014; Alexiadis et al., 2011), like the Kinect from Microsoft and the RealSense from Intel.

However, the depth camera also has several disadvantages. Depth cameras actively emit and recapture the reflected infrared to build the 3D point cloud. This procedure is significantly more power-hungry (about 200 times higher power consumption than the event camera) and suffers from many limitations. Firstly, due to the manufacturing and power consumption consideration, depth cameras’ field of view (FOV) is usually fixed and small. Small FOV limits the sensing space coverage and restricts the dancer movement in a smaller space. Secondly, the distance between the target and the depth camera strongly impacts the detection accuracy. If the sensor is too close to the target (smaller than 0.5m), the two IR receivers have overlapped IR patterns which saturate the IR camera and result in an estimation failure (Jiao et al., 2017). Moreover, when the distance is far, the detection rate drops drastically after a certain point, as the IR receiver cannot receive enough reflected IR light. The first two points together limit the functional dancing area and cause a substantial restriction on relative distance.

Thirdly, since the camera emits light itself, it could also be interfered with by other light sources, which is especially true for strong light sources like stage lights and solar lights. This characteristic restricts the application of depth cameras in outdoor and indoor spaces near the light source.

Fourthly, compared with the event camera, the maximum RGBD camera FPS is usually very low, making it hard to be applied to track fast-paced dances or generate high-fidelity 3D digital dances. The two most commonly used RGBD cameras, i.e., Microsoft Azure Kinect and Intel RealSense, support at most 30 FPS when capturing full RGBD streams. On the contrary, event cameras can easily receive ten-millions level events per second, which gives them overwhelmingly huge advantages over RGBD cameras. We tested our model by generating representations from 20FPS to 150FPS (due to the restriction of ground truth label rate) on all the event streams from test subject eight. The result shows that the estimation accuracy is pretty stable on various frame rate settings (Fig. 16).

Although we wanted to quantitatively compare RGBD and event camera for human pose estimation, we found out that these two types of cameras can not work together. When the depth camera is turned on, its built-in IR projector emits a grid of IR rays 30 times per second. In the eye of the event camera, the IR projection of the depth camera is visible and everything in the surrounding is constantly flickering at a high frequency with meshed dots, which negatively impacts the performance. Therefore, we made a qualitative comparison by separately recording two sessions of human dance by Intel RealSense 435i and DAVIS 346. During these two sessions, the participant moves from 2.5m to 3.5m with some dynamic dance patterns, and the results are shown in Fig. 17. As the figure shows, as the distance between the camera and the human increases, the RGB-Depth camera gradually fails to capture valid human shapes. The human pose estimation program also stops generating valid predictions from a certain point. The RGB-Depth camera-based 3D human pose estimation we chose for comparison is Nuitrack (Nui, [n. d.]), a commercial product designed specifically for RGBD camera-based HPE problems. Although RGB and depth channels are all used, here, for visualization convenience, only the depth channel is shown, and the estimated human masks are shown in green.