Prediction of Metacarpophalangeal joint angles and Classification of Hand configurations based on Ultrasound Imaging of the Forearm

Various contact based and non-contact based technologies have been used to understand hand motion. These include sEMG [3], FMG [4], vision based approaches [5], resistive hand gloves [6], depth information based approaches [7] and WiFi sensing [8]. sEMG has been primarily widely used for understanding and decoding the hand movements [9]. It has been researched for its use as a control input to prosthetics and powered orthotics [10]. Various combinations of sensors with sEMG has also been explored for upper and lower limb motion prediction [11, 12, 13, 14]. Araki et al. used sEMG to measure middle finger joint angle and used it to control a robotic finger [15]. Shrirao et al. used sEMG to predict index finger joint angle while performing flexion-extension rotation of the index finger [16]. Wang et al. used sEMG to predict joint angles during different grasps [17]. Deep Learning has also been used for facilitating sEMG based human computer interaction [18]. sEMG is sensitive to muscle fatigue as a consequence of long term muscle movements, and works best for fast movements of the hand which can inhibit human-like control of digital and virtual interfaces, and prosthetics/orthotics [9, 16].

While sEMG can be used to give an estimate of the muscle activation for hand and finger movements, ultrasound imaging of the forearm can be used to visualize the muscles which can be used for hand state prediction using image processing techniques. Ultrasound imaging of the forearm, or Sonomyography, has been explored as an alternative sensing modality that can capture both muscle configuration and movement [19, 20, 21]. It has been shown to be capable of identifying different hand gestures and finger movements by analyzing the image obtained from ultrasound data with a combination of image processing and classification algorithms [19, 22]. Sonomyography has also been used to classify several grasp types and controlling robotic mechanisms [19]. Akhlagi et al. demonstrated classification of various hand gestures using Nearest Neighbor classification algorithms [23]. Wrist motion classification has also been done utilizing ultrasound imaging [24]. McIntosh et al. classified 10 hand gestures based on small ultrasound data sizes using support vector machines (SVMs) and multi layer perceptrons (MLPs) [25].

While hand gesture recognition has been implemented previously, there remains to be a need for continuous prediction of the MCP joint angles for potential usage as a direct or proportional control of systems and environments. With the advances in deep learning, newer algorithms and techniques can be used to make useful predictions from ultrasound images. We can use this to predict finer hand movements primarily the MCP joint angles while attaining hand configurations relevant to ADL. In this paper, we present our work on utilizing a CNN for predicting MCP joint angles based on ultrasound imaging. We supplement our implementation by using an SVC for classifying different hand motions. We also include results from our simultaneous state/angle prediction pipeline. This work is the foundation of our future research in the domain of human-computer and human-robot interaction that can be facilitated by ultrasound imaging.

SVMs are a class of classical machine learning algorithms based on supervised learning which are used for image classification and regression problems. For ultrasound forearm image classification to predict hand configurations, SVC and other classical machine learning algorithms have been explored in the past [19, 25]. A support vector classifier constructs hyperplanes in high-dimensional spaces, wherein a good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class.

MCP joint angle prediction was implemented using a CNN. CNNs are a class of deep learning algorithms popularly used for two dimensional image data regression based prediction. The CNN architecture that was used in the paper is based on VGG16 [26].

The combined system demonstration pipeline uses a single script to use saved SVC and CNN models to predict both the hand configuration and MCP joint angles. First, the ultrasound image files are loaded and SVC model is loaded. The SVC model prediction decides the model to be loaded for the CNN. Once the CNN model is loaded, the CNN prediction is executed and the four finger MCP angle prediction plot is displayed. This is done repeatedly for a given number of frames which can be specified by the user as a way to demonstrating the combined pipeline shown in Fig 1.

The study was approved by the institutional research ethics committee at the Worcester Polytechnic Institute (No. IRB-21-0452), and written informed consent was given by the subjects prior to all sessions. Table I lists the age, sex, height and forearm diameter at the ultrasound probe location for the 6 subjects enrolled in the study.

Velcro straps on both sides of the custom designed 3D printed probe casing were used to strap the ultrasound probe on the forearm. The subject’s arm rests on a rest table which was further secured with another external Velcro extender to the table after verification of the ultrasound images. Figure 5 shows a subject’s right arm strapped to the rest table.

11 hand configurations relevant to the activities of daily living (ADLs) were chosen for this project [29]. The hand configurations are shown and described in the Fig 4. The Open (all fingers extended) was considered as the rest state for all the 11 hand configurations and the subjects alternated between the open hand and selected the hand configuration. The subject was seated and 16 motion capture markers were attached to their right hand using a double sided tape. Motion capture markers were attached on approximate ends of metacarpal and proximal phalanx for each of the index, middle, ring and pinky fingers as seen in Fig. 3 (a).

This marker configuration allowed us to calculate the MCP joint angle while ensuring free movement of the hand for the desired hand movements. The MCP joint angle was measured by taking the inverse cosine of the two vectors formed by each metacarpal-proximal phalanx pair. Ultrasound gel was applied to the subject’s forearm and the imaging surface on the ultrasound probe. The ultrasound probe was then strapped to the subject’s right forearm. Then, the subject’s forearm was fastened to a rest table. The subject then performed a trial run for the 12 hand movements. Auditory beeps were used to indicate the desired rate of hand opening and closing. There was a rest of 1 minute between each data acquisition session and 2 minutes between each speed transition. 56 seconds of data was acquired for each data acquisition session. Per subject, there were 33 data acquisition sessions, for 11 different hand movements at 3 different speeds each.

For each subject for each hand configuration and speed, 4 seconds of the training and 2 seconds of test data were saved. This data was then combined across the speeds for each subject and hand configuration leading to 12 seconds of training and 6 seconds of test data per hand configuration. All the individual training and testing files were combined and then were shuffled and split with a test-train split of 0.3 to minimize any bias. For our SVC implementation, we used a linear kernel because of the large number of features in each frame of the ultrasound image. We shuffled the data and set a test-train split of 0.3.

The CNNs were trained for each hand configuration and speed on 56 seconds of data for every subject. We used the Adam optimizer function as the gradient descent method [30]. A learning rate of $1e-3$ was set for the Adam optimizer. The learning rate decay was set to $1e-3/200$. We used mean absolute error as the loss, which computes the mean of absolute difference between labels and predictions. Each data file was split with a test-train split of 0.3 without shuffling. While training, the validation split was defined as 0.1. The number of epochs was set to 50.

Motion capture markers were placed on the subject’s hand at the base and the head of the proximal phalanges and the metacarpals for the index, middle, ring and pinky fingers. The marker locations on a subject’s right hand can be seen in Fig 3(a). Data from the thumb was omitted and will be the subject of future research.

For the index metacarpal, the points can be defined as $M1$ and $M2$, with the vector between these points, $\overrightarrow{M1M2}$, shown in red in Fig 3(a). For the index proximal phalanx, the points can be defined as $P1$ and $P2$, with the vector between these points, $\overrightarrow{P1P2}$, shown in blue in Fig 3(a). These locations are based on the finger joints on the hand. By taking the dot product of the metacarpal and phalanges vectors extrapolated from the marker data, the MCP joint angles can be obtained. In Fig 3(b), the vectors and markers are shown for four different angles.

For classification performance, classification accuracy percentage $Acc$ was used as the evaluation metric. It is defined as

where, $TP$ is the number of True Positives, $TN$ is the number of True Negatives and $N$ is the Total Sample Size. $TP$ is an outcome where the model correctly predicts the positive class. $TN$ is an outcome where the model correctly predicts the negative class. Root Mean Squared Error ($RMSE$) values were used to quantify the deep learning regression results. Root Mean Squared Error is defined as

where, $N$ is the Total Sample Size, $i$ is the integer value ranging from 1 to the total number of samples, $y_{i}$ is the test data value for the sample $i$, and $\hat{y_{i}}$ is the value predicted by the deep learning algorithm at the sample $i$.

11 classes described in Fig 4 were considered for the classification. We were able to obtain 100% classification accuracy for an SVM classifier trained on all the speeds to reduce bias of our algorithm. Figure 6 (a) shows the confusion matrix of the classifier for subject #1.

MCP joint angle prediction was done using VGG16, a CNN network used for to obtain relevant information from image data. Utilizing VGG16, we were able to generate a good degree of visual correspondence between our test data and the model predictions. This is shown for four second test data in Fig 6 (b) through Fig 6 (h). RMSE values were obtained from each of the hand configurations for each speed, for every subject. For the results, the range is set from 0°-100°to encapsulate the angle data from all subjects. The overall RMSE value averaged over all the hand configurations, three different speeds and all the subjects was found to be of 7.35°.

For the combined prediction pipeline SVC models trained to classify between 11 different hand configurations for a single speed and subject and 11 VGG16 models trained for prediction of MCP joint angles were used. In the combined loop, first the hand configuration state prediction is done using the saved SVC model. Then, based on the SVC prediction, the CNN model is chosen from a dictionary of saved models. For each frame of data, the SVC prediction takes 0.1 - 0.15 seconds. CNN prediction takes 0.004 - 0.007 seconds. The total time ranges from 0.11 - 0.16 seconds for angle and state prediction for one frame of ultrasound information, leading to 6.25 - 9.1 Hz of data processing capability using the combined prediction pipeline.