Pose Trainer: Correcting Exercise Posture using Pose Estimation

We now describe the Pose Trainer application from a technical perspective as a pipeline system, consisting of multiple system stages (see Figure 1). Pose training starts from the user recording a video of an exercise, and ends with the Pose Trainer application providing specific feedback on the exercise form to the user.

First, the user records a video of themselves performing a selected exercise. The video is recorded from a particular perspective (facing the camera, side to the camera, etc) that allows the exercise to be seen. However, there are no requirements on camera type or distance from camera: the user only needs to make sure that their body is visible. Then, the user trims the video such that it includes only the frames of the exercise.

We leave video recording and cropping to the user, so that they can use whatever video recording and editing system they are most comfortable with. All modern computers and smartphones include easy-to-use video recording and cropping software, and since Pose Trainer supports all common video formats, any software can be used.

For pose estimation, we use deep convolutional neural networks (CNNs) to label RGB images. After experimentation with multiple state-of-the-art pose estimators, we choose to use the pre-trained model, OpenPose, for pose detection. OpenPose introduces a novel approach to pose estimation using part affinity fields, which are vectors that encode the position and orientation of limbs. The model is composed of a multi-stage CNN with two branches, one to learn the confidence mapping of a keypoint on an image, and the other to learn the part affinity fields  [3]. OpenPose is both accurate and efficient, while also scalable to multiple people without scaling up the run-time.

Another important factor in our choice of OpenPose is its ease of installation and use for our end-users. Most pose estimators are currently released as Tensorflow or Caffe source code, containing the model architecture, and require challenging user installations and downloading of weights to be usable. In addition, GPU libraries such as CUDA and CuDNN need to be installed for these estimators to run properly. This is far beyond the reach of most average computer users. In contrast, OpenPose is released in Windows and Linux executable format, which means that no installation or knowledge of programming is required at all to use it. In addition, no external installation of GPU libraries are required: OpenPose will automatically use the NVIDIA GPU inside the computer, if present. This means that in order to use our application, Pose Trainer, a user only needs to download OpenPose and install Anaconda to get up and running.

OpenPose output consists of lists containing the coordinate predictions of all keypoint locations, and their corresponding prediction confidence. We consider the predictions of 18 keypoints of the pose, which include the nose, neck, shoulders, elbows, wrists, hips, knees, and ankles.

After running pose estimation, we write our own parser to use the raw keypoint output. First, we read the list of keypoint predictions for every video, and segment the list into Part objects that store the $x$, $y$, and confidence of each keypoint. Additionally, we track if a keypoint is obscured in the image frame (prediction confidence of 0). For every frame in the video, we compose each Part (joint keypoint) to construct a Pose object that represents the full skeletal estimation of a person. For the full video, we compose a PoseSequence object that chains the Pose estimations from each video frame.

We note the need to generalize our application to account for users with different body length measurements, distance from the camera, as well as other relative factors. To account for these differences, we normalize the pose based on the torso’s length in pixels. The torso length is calculated by the average of the distance from the neck keypoint to the right and left hip keypoints. This normalization works extremely well: we observe that the torso length stays very constant through all the frames of input videos. Distances are thus represented as ratios of torso length: for instance, an upper arm length of 0.6 means that the upper arm is 0.6 the length of the torso.

For certain exercises, we resolve the ambiguity in camera perspective. For example, the bicep curl exercise is recorded from the side of the body, and could be performed with either the left or right arm. We identify which arm is performing the exercise in the video by measuring which keypoints are most visible (left or right side keypoints) throughout all frames of the exercise. This accurately detects perspective all of our input videos.

Next, we compute body vectors from keypoints of interest, and use personal training guidelines and our own recorded videos to design geometric heuristics, evaluating on the body vectors.

Our second approach to evaluate the exercise posture given normalized keypoints uses a more data-driven, machine learning approach. Since the recorded videos can be of arbitrary length, this results in a different keypoint vector length for each example. Different feature vector lengths present a challenge for many machine learning models, so we approach this task using dynamic time warping (DTW) [1] with a nearest neighbor classifier.

DTW is a metric used to measure the non-linear similarity between two time series. When two similar sequences are phase-shifted (i.e. shifted in the time dimension), metrics like the Euclidean distance fail because it is a direct comparison of two points at the same time. In DTW, we try to dynamically identify the keypoint in the second sequence that corresponds to a given point in the first.

Given two keypoint sequences $Q\in\mathbb{R}^{m}$ and $C\in\mathbb{R}^{n}$, we construct a distance matrix $D\in\mathbb{R}^{m\times n}$, where $D_{i,j}$ is the Euclidean distance between points $q_{i}$ and $c_{j}$. We iterate through all points in the matrix using dynamic programming, find the optimal match of points in each sequence, and compute the distance of between the matched points. A drawback with DTW is that it’s not robust to noise. When OpenPose generates noisy keypoints, this would affect the performance of DTW. To accommodate for this, we run the a keypoint sequence through a size 5 median filter twice before computing the DTW measures.

We take an input keypoint sequence and compute the DTW distance of it with all the training sequences. Then, we build a binary nearest neighbor classifier that predicts ’correct’ or ’incorrect’ form based on the DTW distances.

The single arm bicep curl is a dumbbell (hand-held free weight) exercise that isolates the biceps, the large muscle in the upper arm responsible for bending in and twisting at the elbow. In the bicep curl, a dumbbell is lifted up from a resting, extended position, with rotation around the elbow, while keeping other parts of the body still. This action targets the exercise towards the biceps muscle. Common mistakes when performing a bicep curl include using the shoulder to help swing the weight up, and thus rotating around the shoulder, as well as not lifting the weight fully up.

In our geometric algorithm, we consider the range in angle between the upper arm and torso (measuring if the user rotates the shoulder when lifting), and the minimum angle between the upper arm and forearm (measuring how high the user lifts). If the range between upper arm and torso angle is above 35 degrees, we flag this as too much shoulder rotation. If the minimum angle between upper arm and forearm is above 70 degrees, this is flagged as not curling the weight all the way up.

Our geometric algorithms correctly classify all good form videos correctly. 80% of bad form exercises are labeled as incorrect, while the remainder are labeled as correct form. Upon visual analysis, these mislabeled examples are on the fringe between proper form and clearly improper form: tweaking our algorithms will, as expected, adjust the decision boundary, and thus our strictness when evaluating user exercise posture.

For our ML algorithm, we split our 16-video bicep dataset into 9 training examples and 7 test examples. The DTW classifier achieves an F1 score of 0.85. The precision, recall, and F1 for all exercises are summarized in table 1.

Figures 2 and 3 illustrate examples of good and bad form, as well as the pose angle statistics that we use when evaluating a user’s exercise.

Below is an example of Pose Trainer’s output on a correctly performed bicep curl:

Below is an example of Pose Trainer’s output on an incorrectly performed bicep curl:

Dumbbell front raise is a free weight exercise targeting the shoulders, specifically the anterior deltoids. In the front raise, the lifter holds weights at their side, and lifts the weights straight in front of them, with arms mostly straight. The lifter should keep the body still to isolate the exercise to the shoulders, and should lift up to slightly above the shoulders to complete the full range of motion. Common mistakes include straining and using torso movement to help lift, as well as not lifting up to above the shoulders (see Figure 4).

Our geometric algorithm for front raise measures two things: the horizontal range of motion of the back, and the maximum angle between the torso and the arm. Back motion is measured by measuring the greatest change in vector difference between frames of the exercise, and is used to determine if the user is swinging their upper body to help lift the weight. Torso arm angle is used to determine if the user is lifting all the way up.

For our ML algorithm, we split our 28-video front raise dataset into 16 training examples and 12 test examples. The DTW classifier achieves a perfect F1 score.

Pose trainer will output the following feedback to users for front raise:

‘Your back shows significant movement. Try keeping your back straight and still when you lift the weight. Consider using lighter weight.’

‘You are not lifting the weight all the way up. Finish with wrists at or slightly above shoulder level.’

Dumbbell shoulder shrug is an exercise that focuses on the shoulders, specifically the upper trapezius. The lifter holds dumbbells at their sides and raises their shoulders as much as possible, while keeping the torso rigid and the elbows in a relaxed position. Common mistakes include not going through the full range of motion in the shoulders, and bending the elbows and using the arms when lifting (see Figure 5).

Our geometric algorithm for shoulder shrug measures the range of motion of the shoulders (normalized by torso length), as well as the angle between the upper arm and forearm. If the range of shoulder movement is too low, we flag as the mistake of not going through the full range of motion. If the angle between upper arm and forearm is too small, this means the user is bending too much at the elbow.

For our ML algorithm, we split our 32-video shoulder shrug dataset into 19 training examples and 13 test examples. The DTW classifier achieves an F1 score of 0.85.

Pose trainer will output the following feedback to users for shoulder shrug:

‘Your shoulders do not go through enough motion. Squeeze and raise your shoulders more through the exercise.’

‘Your arms are bending when lifting. Keep your arms straight and still, and focus on moving only the shoulders.’

Dumbbell shoulder press, or overhead press, is a weight training exercise where dumbbells are pressed straight upwards. The exercise works many muscle groups in the shoulders, chest, and upper back. Many mistakes can be made in this exercise: the lifter could raise the weights too far forward or back, move the torso too much when lifting, or not lift the weights high enough (see Figure 6).

Our geometric algorithm for shoulder press measures the range of motion of the back using the neck and hip keypoints, the range of motion of the arm using the elbow and neck keypoints, and the maximum angle achieved between the upper arm and forearm vectors. If the range of motion for the back is too large, we warn the user to keep their back straight. If the location of the elbow is behind their neck, we warn the user to not roll their shoulders during the lift. If the angle between the upper arm and forearm is too small, we warn the user to lift the weight up all the way.

For our ML algorithm, we split our 36-video front raise dataset into 21 training examples and 15 test examples. The DTW classifier achieves an F1 score of 0.73.

Pose trainer will output the following feedback to users for shoulder shrug:

‘Your back shows significant movement while pressing. Try keeping your back straight and still when you lift the weight.’

‘You are rolling your shoulders when you lift the weights. Try to steady your shoulders and keep them parallel.’

‘You are not lifting the weight all the way up. Extend your arms through the full range of motion. Lower the weight if necessary.’