AIMusicGuru: Music Assisted Human Pose Correction

The original pose estimation algorithms generate 2D skeletons from monocular image frames [40, 9]. However, due to limited expressiveness and ambiguity, 2D representations are often not sufficiently powerful for downstream tasks. Recently monocular 3D pose estimation approaches, often built on top of the 2D skeletons, have gained success [24, 6, 12, 41]. However, both 2D and 3D pose estimators suffer from issues such as jitter, joint inversions, joint swaps, and misses - issues extensively studied in [22, 27]. In 3D, we observe additional challenges with scale, depth disparity, and joint drift. To overcome these issues, filter-based methods such as simple moving average, Kalman filters, and particle filters have been used [23] with some success, but they struggle to model highly non-linear motion trajectories.

Recently a new paradigm demonstrating the use of multi-modal data has emerged. It has been shown of tremendous success for generating poses based on audio data in dance sequences [21, 18, 32], for speech gesture generation [20, 11], and for learning new multi-modal features from visual and sound data that can be used with a variety of classic vision or audio tasks [2].

While there are plenty of audio-video datasets for the areas of speech [17, 4, 26], dance [21, 35, 1, 42, 30], everyday actions [10, 13, 33], and music [21, 35, 1, 28, 30], there are limited data when it comes to ground truth motion capture data on how people move while playing musical instruments. The table 1 details the comparison of audio-video datasets.

There are other datasets with synchronized audio and motion capture data that are also used in gesture generation, such as AIST++[21] and EA-MUD[8]. These datasets have motion capture data from dances with synchronized music. However, in these scenarios, the audio is not directly produced due to human motion. Instead, people move in response to or with the anticipation of the beats of the music. As shown by prior work [21], they can be used for learning realistic human motions that are consistent with the motion styles. However, generation of such pose trajectories does not have a single unique solution, and approximations to the predictions are made. Therefore, the motion trajectories are not precise and do not conform to the real observations in the videos.

For the case of understanding the relationship between music production and the motion that produces them, we find four potential datasets [14, 19, 28, 39]. FCVID [14] contains YouTube videos from the wild hence lacking 3D pose ground truth. URMP [19] was recorded in the lab but without a motion capture or multi-camera system, so it lacks the motion ground truth. QUARTET [28] consists of motion capture raw marker data; however, it only contains upper body parts, contains limited view angles from a single camera, and only contains 30 video sequences out of the 101 that were recorded, making the data very limited. The TELMI [39] dataset consists of 145 sessions with three different camera angles and full-body motion capture raw marker data of the body, violin, and bow. However, the motion capture marker needs kinematic human body model fitting to generate accurate 3D joint coordinates. TELMI does not have calibration data necessary for this mapping from ground truth to an image frame. Additionally, it has a limited number of subjects, making it difficult for downstream machine learning tasks. Thus we conducted extensive evaluation, synchronization, and post-processing steps to get the data to a machine learning-ready state. This is detailed in section 4.

As shown in Fig. 2, we propose Music Assisted Pose Network (MAPnet) that takes temporally sparse and noisy pose skeleton data and refines them to generate temporally dense pose with the help of the audio. The primary building block of MAPnet are three transformers: (1) Pose Transformer and (2) Audio Transformer learn temporal attention from the input pose and audio signals. Since pose and audio embeddings are of different temporal dimensions, we use novel Rebalancing layers for both pose and audio embeddings. Further, the output pose and audio embeddings will be fused by the (3) Cross-Modal Fusion Transformer to learn the correspondences between pose and audio.

Initially introduced in natural language processing for machine translation, the Transformer [37] is a very powerful technique employing multi-head attention (MHA), which allows a model to learn to pay attention to multiple salient features in sequential data. Transformers’ fundamental building blocks are an MHA followed by a feed-forward (FF) layer with normalization and residual computed after each layer.

As motion features, we directly take the sequence of normalized poses as shown in Fig. 3. The input 3D skeletons have dimension $\mathbb{R}^{T_{in}\times 13\times 3}$, where $T_{in}$ is the number of input frames and is equal to 3 seconds times the input frame rate. We have 13 joints for each person, and each joint is described by its 3D position (x,y,z). We flatten the last two dimensions to form our pose feature $P\in\mathbb{R}^{T_{in}\times 39}$.

Raw audio which are sampled at 44 kH are divided into 3 seconds sliding window with 1 second hop. Each 3 second audio is then divided into 150 time steps to generate 150 audio features. We compared raw audio and audio features used by other audio based networks [21]. After our experiments, we carefully selected the following features: the 1-dim envelope (Fig. 4e), 20-dim MFCC (Fig. 4c), 12-dim chroma (Fig. 4d), 1-dim one-hot peaks (Fig. 4e), and 1-dim rms to obtain a 35-dim music feature at 150 time instances $A\in\mathbb{R}^{150\times 35}$.

Given the pose feature $P\in\mathbb{R}^{B\times T_{in}\times 39}$ and the audio feature $A\in\mathbb{R}^{B\times 150\times 35}$, where B is the batch size, MAPnet first embeds the two features using fixed hidden size $H_{1}$ into $P\in\mathbb{R}^{B\times T_{in}\times H_{1}}$ and $A\in\mathbb{R}^{B\times 150\times H_{1}}$. The two embeddings are then fed into the Pose Transformer and the Audio Transformer, respectively, with positional encoding. The positional encoding ensures the temporal order of the concatenated features. The output embeddings are then passed to the Rebalancing layer to reshape the embeddings into ${P\in\mathbb{R}^{B\times H_{2}\times H_{1}}}$ and $A\in\mathbb{R}^{B\times H_{2}\times H_{1}}$. These are then concatenated to get the combined embedding $C\in\mathbb{R}^{B\times 2H_{2}\times H_{1}}$ and sent to the fusion transformer without positional encoding.

We pass the output of the fusion transformer into three fully connected layers to get our output $\in\mathbb{R}^{B\times T_{out}\times 39}$, which gets reshaped into $\mathbb{R}^{B\times T_{out}\times 13\times 3}$ to calculate the loss with the ground truth, where $T_{out}$ is the number of output frames and is equal to 3 seconds times the output frame rate. The whole network with three transformers is learned in an end-to-end-manner.

In our experiments, simply concatenating the modalities do not work, and the number of layers on individual transformers and fusion transformers make a huge difference. Therefore, our introduction of the rebalancing layer is vital to make the model work, for which we include ablation study (see table 5). The rebalancing layer re-encodes the output from the individual pose and audio transformers. Doing this reduces the load for the fusion transformer to learn the attention mapping distribution of the pose and audio transformer embeddings.

We use the Mean Per Joint Position Error (MPJPE) as our loss function. MPJPE calculates the mean of the l2 loss between the ground truth joint position and the predicted joint position. For our case, the MPJPE is defined as:

$$\mathcal{L}=\frac{1}{13}\Sigma_{n=1}^{13}\left\|j_{n}-\hat{j}_{n}\right\|_{2}$$

where $j_{n}$ and $\hat{j}_{n}$ are the ground truth and estimated 3D joint coordinates of the n-th joint.

The proposed MAPdat dataset is developed based on the publicly available database called TELMI Open Database [39]. The TELMI dataset includes motion capture data, depth data, video data, and sound data of four master practitioners playing different violin techniques ranging from basic techniques such as controlling the bow weight while playing a violin to advanced techniques such as staccato articulation, sautille articulation, etc.

The recordings are of master practitioners. Two to four of the master practitioners performed 41 different techniques as shown in figure 5. Although much significant research has been done by the original owners of the TELMI dataset, it was not in an ideal format to be used for machine learning.

Our current MAPdat is a prototype dataset and uses only a subset of TELMI. Since the TELMI consortium owns the TELMI data, we are releasing our scripts to automate the download, post-process the original data, and generate machine learning-ready MAPdat data for the community to allow replication of our work.

While the TELMI dataset is rich, the dataset was built for a different type of research than our use case (e.g., [3]). The video and motion capture data of the TELMI dataset is 50 frames per second, and the motion capture data is synchronized with audio and video data [38]. The 32 motion capture markers of the body: ariel, right and left forehead, back-head, shoulder, back-shoulder, inner elbow, outer elbow, inner wrist, outer wrist, pinky, thumb, back waist, inner knee, outer knee, toe, and two points on the vertebrae (TS and T10) are included in the original data. We ignore markers that cannot be used for human body kinematic fitting with actual physical measurements such as the head and hand markers and do a kinematic fitting with useful markers using heuristic calculations of the clavicle, shoulders, elbows, wrists, hips, knees, and toes. From this, we compute a final 13 joints for a human body model as described in Fig. 3 consistent with the output of the video-based pose estimation approaches.

Although the motion capture data was synchronized with audio and video data, we found that the video, the audio, and the motion capture markers data had different lengths for many trials. Therefore, after they are temporally synchronized, we trim the data to have the same sizes. We validated the motion capture and audio-video synchronization by calculating the onset and offset of the music from the audio features and bow-violin distances and velocity from the motion capture data and doing a qualitative manual review. In TELMI data, data collection began and ended a few seconds before and after the master practitioners played violin. Since those portions of data where no violin was being played are not relevant to our study, we have also cropped those parts before onset and after offset using the calculated bow-violin distances. We then transformed all the motion capture data with the left toe as the origin.

Each sample is normalized and then modeled with jitters as a gaussian noise and joint inversions as random swaps of joints to create ten variations, detailed in the next section. These synthetic noises are based on the most common types of challenges that pose estimators face [23]. These new 10x sets are randomly divided into 8:1:1 ratio for train:valid:test sets. The resulting sample and its corresponding audio samples are then re-sampled into three-second windows with a one-second hop as a sliding window. This results in 40,230 samples or 120,690 seconds of total data.

From our pilot studies, we find that the two most prominent forms of noise in 3D pose estimation are jitter and joint inversions, which was reinforced by [22, 27]. Depending on the pose estimation network, jitter can vary in magnitudes, and joint inversion can occur with variation in the frequency of swaps and with which joints they get swapped. To emulate the noisy pose estimates of a monocular 3D pose estimator, we augment jitter and joint inversion to our ground truth motion capture data. We determined the noise parameters based on the average error from 3D pose estimates of MediaPipe [24] and VideoPose3D [29] on the TELMI [38] video data. The magnitude of Gaussian noise is based on the average distance of pose data from the ground truth motion capture data. While joint swaps are highly dependent on the camera viewpoint, we randomly distribute joint swaps based on the average occurrence from the total length of the video. In MAPdat, we modeled jitter as Gaussian noise distribution with a standard deviation of 300 mm. We modeled the distribution of joint swapping events over time as a Poisson random variable with an average of five swaps per minute. Once a joint swap occurs, the swap length is uniformly distributed between 0.5s and 3.0s. Using these parameters, we randomly generate ten variants of each video.

We implemented all our methods in Tensorflow. We used four NVIDIA RTX 1080 Ti GPUs to do our training and testing. The number of output frames $T_{out}$ across our experiments is 150 so that our model will predict 3D skeleton poses at 50 fps ($\tau$=1.0). We define $\tau$ as the ratio of output to input i.e. our output frame rate is 50, so the input frame rates at 50, 25, and 17 corresponds to $\tau$=1.0, $\tau$=0.5, and $\tau$=0.33. The input frame rate in the experiments was 50fps ($\tau$=1.0), 25fps ($\tau$=0.5), and 17fps ($\tau$=0.33), which correspond to same, one half, and one third of the output frame rate. The size $H_{1}$ and $H_{2}$ in the hidden layers used to embed the features was set to 160 and 150, respectively. For all experiments we used in training a batch size of 128 using the Adam optimizer [16].

To extract audio input features, we use audio processing toolbox Librosa [25]. Most dance-based pose generation uses the beats feature as a cyclical marker for motions, such as foot-to-ground contact. In the case of the violin playing music, no distinct beats can be detected. Instead, we use changes in bow direction as the one-hot peaks and the RMS feature that better model the bow motion. The details of a sample waveform and audio feature have been visualized in Fig. 4.

We compare MAPnet with and without rebalancing with a Simple Moving Average calculation (SMA), a Long-Short Term Memory (LSTM) network using Pose only (LSTM Po), and one using Pose and Audio (LSTM PA), and to a Transformer based on Pose only (PoT). Table 2 shows the quantitative results of our experiments. We used the Mean Per Joint Position Error (MPJPE) as our metric. For the Pose-only Transformer (PoT), we observed from the results that at 50 fps, the model struggles to filter out large jitter in the input data. PoT loses the motion trajectory data and deteriorates, as the number of input frame rate declines, at both $\tau$= 0.5 (25fps) input rate and $\tau$= 0.33 (17fps) input rate. In contrast, because MAPnet models the music features that have alignment with the actual motion trajectory, it filters the jitter and learns to model the joint swaps. Additionally, as we reduce the input frame rate (from 25 to 17), we do not see further deterioration but rather see a marginal improvement which can be explained by the transformer giving more attention to the audio signal to capture motion trajectories of very fine details, as can be observed in figure 7.

Simple Moving Average(SMA) is a traditional noise filtering method where the sum of a specified number of consecutive data is averaged to flatten out the noise. We have calculated the SMA on the input frames, and since it is a piece-wise linear noise-filtering system, we calculated the midpoints of the SMA results to generate higher frequency output. This method is not suitable for solving our problem of predicting motions at a high frame rate due to two reasons. Firstly, SMA is meant for linear systems, but the fine bow movements or movements of the right wrist are highly non-linear, as shown in Fig. 7 with the ground truth. Therefore, it cannot detect and learn the differences between fine bow movements and noise. Secondly, since it averages the input data, the prediction of the motions between two input frames is always predicted as linear, which is highly inaccurate for non-linear bow movements. As shown in Table 2, as the frame sparseness increases, the performance of a simple moving average algorithm deteriorates, while the MAPnet prediction, with the help of audio features, stays close to the trajectory of the ground truth. The results of the simple moving average from this ablation study were consistent with [23].

Inspired from the LSTM network architecture presented in [32], we present the ablation study by replacing our transformer models with LSTM models. We observe that LSTM struggles to model the non-linearity between the pose and music. From table 2, the MPJPE error is exponentially high when compared to our MAPnet model, and therefore it fails to predict the accurate poses. We also observe the LSTM also struggles to learn different motion difficulty (see table 3). We anticipate this behavior is due to LSTM inherently depending on the sequential structure and being versatile in handling a single modality (either audio or pose). On the other hand, our transformer-base MAPnet is equipped to correlate multi-modal non-sequential data structures.

As shown in figure 7, at low input visual frame rate $\mathbf{\tau}=0.33$, during fine bow movements, The Pose-only model (Fig. 7a and 7d) and SMA (Fig. 7c and 7f) poorly predicts the motion of the wrist joints. However, the MAPnet model, due to coupling relations between the motion and the music, can generate more accurate predictions, as can be seen in the Fig. 7b and 7e. We see that there is a significant error in the motion trajectory of the SMA and Pose-only model prediction, whereas there is a more substantial overlap between the MAPnet prediction and the ground truth.