Enabling Hand Gesture Customization on Wrist-Worn Devices

Leveraging a user-experiment-platform with a large user repository, we recruited 512 participants (133 self-identified female, 378 male, 1 non-binary) with a wide coverage of age range (min=21, max=63, mean=33.1$\pm$10.5). Majority of users were right-handed (N=442, left-handed=70). We used Apple Watch Series 5 and 6 for data collection, with IMU sensors sampled at 800 Hz max, ultimately downsampled to 100 Hz during training ¹¹1We collected raw data with an overly high sampling rate, 800 Hz, to maximize our dataset ability. However, during the model training, we found that 100 Hz already suffices. Thus, in the rest of the paper, our framework only uses 100 Hz data.. Participants wore the watch on their non-dominant hand during the data collection. Data was first stored on the watch and then uploaded to a server for processing and model training. The user study received institutional IRB approval.

We asked participants to follow instructions on the watch throughout a session. In each round, a gesture would appear on the screen, and we asked participants to perform that gesture 10 times, each time following a three-second countdown. Gesture order was randomized and each participant performed at least three rounds of data for each gesture. Throughout this process, a phone was placed directly above the user’s hand, and we recorded video to serve as additional ground truth for annotation purposes.

We also considered other relevant factors: body posture, hand-eye angle, motion, gesture variation, and activity. We randomly picked a subset of participants to perform gestures in different body postures, such as sitting upright (N=224), sitting and leaning back (N=149), standing (N=58), and lying down (N=21). For different hand-eye angles, participants were either asked to put down their hand to abdomen-level (N=193), chest level (N=38), or head-level (N=203). We also asked a few participants (N=40) to walk while performing gestures. Lastly, we asked a small fraction of participants (N=13) to perform gestures at different speed and intensities (i.e., slower-faster, weaker-stronger). Some participants (N=12) performed light everyday tasks (e.g., typing or wrist twisting) while occasionally performing a gesture.

In addition to positive examples, we also asked participants to perform negative (i.e., non-gesture) examples. In this round, we asked participants to perform normal indoor daily activities, such as walking, phone browsing, and typing (among others). Moreover, we asked a small group of participants (N=12) to perform a wide range of behaviors that involved fine-grained hand movement, including tapping on the watch/other surfaces, scratching head/hands, using mouse/keyboard, playing pens, brushing teeth, shaving, washing hands/utensil, showering, driving, using juicer/vacuum cleaners, playing video games, opening bottles, and biking. These negative sessions were not video-recorded.

Each data collection study lasted 30 to 60 minutes. We synchronized videos with our IMU signals and annotated the start and end times of each gesture performance. After annotation, we collected approximately 110,000 gesture samples (30 hours), and 60 hours of negative samples. The average duration of single gestures (i.e., Clench and Pinch) is around 550ms, while the average duration of double gestures (i.e., Double Clench/Pinch) is around 800ms.

We first investigate the direct outcome of the prediction, which is at the window level. The results show an average accuracy of 74.8%, a precision of 93.6%, a recall of 74.8%, and an F1 score of 82.7%. Figure 3(a) visualizes the confusion matrix of the window-level prediction on the testing set.

The confusion matrix shows that there is little confusion among the four gestures, and that the majority of the misclassification comes from the model not recognizing some windows where a gesture actually happens, leading to false negatives. The window-level prediction only focused on every single 1-sec window. However, in real-time scenarios, a gesture is comprised of multiple windows. Therefore, we need to aggregate our window-level predictions to obtain our gesture-level predictions.

The aggregation involves a few hyper-parameters. For the four gestures, we need to decide how many consecutive prediction windows are required before the model predicts a gesture. For negative data, we also need to decide how many consecutive windows with non-negative predictions are required before the model triggers a “false positive”. We performed grid search on these hyper-parameters using our validation set. Our final consecutive window length thresholds are 3, 4, 3, 4 for Clench, Double Clench, Pinch, and Double Pinch, respectively.

Aggregation significantly improves our recognition performance, with an average accuracy of 95.7%, a precision of 95.8%, a recall of 95.7%, and an F1 score of 95.8%. Moreover, the gesture-level false positive rate is 0.6 times per hour. Figure 3(b) presents the confusion matrix of the gesture-level results. These results indicate that our model can accurately recognize gestures on new users’ data and is highly robust to negative data.

Before actual data processing, it is worth noting that there is no readily available training sample. When end-users record data of their new customized gestures, they can either do gestures consecutively in a row (similar to the data collection process in Section 3.1), or follow some instructions to do one gesture at a time and repeat several times, depending on the interaction design. In either way, it is unrealistic to ask users to provide the exact start and end timestamp of the gesture. Therefore, we need to segment the signal sequence to obtain data samples.

We take the output signals of the middle bandpass filters (8-32 Hz) as this is robust to noisy arm movement, and use a peak detection algorithm to identify potential moments of performing hand gestures. Specifically, we calculate the sum of the magnitude of both filtered accelerometer and gyroscope signals, and apply a 1-sec absolute moving average to smooth the data. We then use a simple peak detection method that finds local maxima by comparing neighboring values (with distance threshold as 1 sec). A peak is ignored if it is lower than the overall average of signal magnitude. If any time reference is available (e.g., a countdown mechanism), we can further filter peaks according to the reference. We take a 1-sec window centered at these potential peaks, and feed them into the feature extraction part of the pre-trained model. We then compute a Euclidean distance matrix of the normalized embedding vectors and remove outliers (threshold empirically set as 0.8). In such a way, we can segment out pronounced, repetitive hand movement periods that correspond to the target gestures.

Once the peaks are determined, we take a 1.5-sec window centered at each final peak to ensure that a gesture is fully covered by the window. Our data augmentation techniques are applied to these windows.

After data segmentation, we use several data augmentation techniques to generate a larger number of samples. Three time series data augmentation techniques (Iwana and Uchida, 2021), with all seven combinations ($2^{3}-1$), are used to generate positive samples, enlarging the data size by eight times: 1) zooming, to simulate different gesture speed, randomly chosen from $\times 0.9$ to $\times 1$; 2) scaling, to simulate different gesture strength, with the scaling factor $s\sim\mathcal{N}(1,0.2^{2}),s\in[0,2]$, and 3) time-warping, to simulate gesture temporal variance, with 2 interpolation knots and warping randomness $w\sim\mathcal{N}(1,0.05^{2}),w\in[0,2]$.

Moreover, we also employ three augmentation techniques to generate gesture data that is marked as negative (Xu et al., 2021b): 1) cutting out by masking a random 0.5 sec of signals by zero; 2) reversing signals, and 3) shuffling by slicing signals into 0.1-sec pieces and generating a random permutation. These augmentations are often used in other ML tasks to augment positive data, but we mark the data augmented by these techniques as negative to ensure our model only recognizes valid gestures. We also applied the seven positive augmentation techniques on these negative data to generate more negative samples.

After data augmentation, we take a 1-sec sliding window on these 1.5-sec windows to generate samples to be fed into the pre-trained model. The step size is set as 0.1 sec, leading to five input samples from each 1.5-sec window. In addition, the data collected in Section 3.1 are all added as negative data to improve the robustness of the model against noisy movement.

Although the data augmentation can generate signals with larger variance from the data recorded by end-users, these augmented data may not be close to the actual gesture variance introduced by natural human behavior. Therefore, we further synthesize more data from both the raw signals and the augmented signals that simulate the natural motion variance. Specifically, we train a $\Delta$-encoder (Schwartz et al., 2018), a self-supervised encoder-decoder model that can capture the difference between two samples (i.e., $\Delta$) belonging to the same gesture, and use it to synthesize more new gesture samples.

A $\Delta$-encoder is trained as follows: it takes two samples (sampleInput and sampleRef) from the same class as the input, feeds sampleInput through a few neural network layers to be a very small embedding called $\Delta$-vector (similar to a typical Autoencoder (Goodfellow et al., 2016)), and then use the $\Delta$-vector and the sampleRef to reconstruct sampleInput. The intuition comes from the fact that the size of $\Delta$-vector is so small that it focuses on capturing the difference between sampleInput and sampleRef, which is then used to rebuild sampleInput with sampleRef as the reference (Schwartz et al., 2018). After the $\Delta$-encoder is trained, it can take another sample from the new class as a new sampleRef, and generate a new sample of the same class with a $\Delta$-vector. This $\Delta$-vector can either be obtained via feeding any existing sample from other classes through the encoder, or randomly generated.

In our case, we use the data of the four pre-existing gestures to train a $\Delta$-encoder. During the training, we randomly draw two samples from the same gesture and the same user to ensure that the model captures the within-user variance instead of the between-user variance. We use the feature embeddings of length 120 as the input and the output of the $\Delta$-encoder to save computation cost. Our structure of our $\Delta$-encoder is fairly simple. Both the encoder and decoder have one hidden layer with a size of 4096 and uses Leaky ReLU ($\alpha=0.3$) as the activation function. The size of $\Delta$-vector is set as 5. Using the training set from the four gestures, the model is trained with 200 epochs and has a 0.5 exponential decay on the learning rate every 30 epochs. The epoch with the best results on the validation set is saved. We also calculate and save the $\Delta$-vectors from the four gestures’ testing set, which will be used to generate new samples.

In real-time, when the customized gestures’ data come in and go through the augmentation stage, we use the $\Delta$-encoder to generate extra samples of the customized gestures that contain more natural gesture variance, enlarging both augmented positive and negative data by 10 times.

After the data augmentation and data synthesis, we obtain a large amount of data with appropriate variance to train the prediction head. To further improve the robustness of the model, we adopt the practice of adversarial training regularization (Goodfellow et al., 2015; Ma, 2018) when learning the model.

The main idea of adversarial regularization is to train a model with adversarially-perturbed data (perturbed towards the decision boundary or inverse gradient decent so that the training process becomes harder) in addition to the original training data. It can prevent the model from overfitting and classify the data points close to the boundary more robustly. In Figure 4, the two customized gestures’ data from the same user tend to be blended with the existing four gestures near the boundary. The adversarial regularization can help to enhance classification performance, especially for the purpose of reducing false-positive. We set both the adversarial regularization loss weight and the reverse gradient step size as 0.2.

Through a series of data segmentation, data augmentation, data synthesis, and adversarial training, we can learn a robust prediction head for each new user that can accurately recognize their customized gestures with a low false-positive rate. Figure 5 visualizes the whole training procedure of the prediction head.

To evaluate the performance of our framework, we refer to the taxonomy of dynamic hand gestures (Choi et al., 2014) and choose a set of new gestures (in addition to the four supported by the pre-trained model) that covers a wide range of movement. Other than the existing four gestures (Clench, Double Clench, Pinch, Double Pinch), we pick a set of 12 new gestures that are representative of different wrist/hand/finger movement patterns (Choi et al., 2014): Spread and Double Spread have opposite finger movement (opened v.s. closed) against Clench and Double Clench, PinkyPinch and Double PinkyPinch use a different finger than Pinch, and Peace has two fingers opened and three fingers closed; Slide has opposite motion between the thumb and the index finger; TwistOut/In, RotateOut/In and Extend/Flex involve wrist movement in different ways. Figure 7 visualizes the 16 gestures (12 new + 4 existing). All gestures start from the neutral pose and return back to the neutral pose.

It is worth noting that the main purpose of this gesture set is to evaluate the framework’s performance. The actual gesture set that can be supported by our framework goes beyond these 12 gestures.

20 participants (4 self-identified female, 16 male, Age = 29.8$\pm$4.9) volunteered to participate in the data collection study. 3 participants are left-handed. We employed the Apple Watch Series 6 for data collection, with IMU sensors sampled at 100 Hz. All participants wore the smartwatch on their non-dominant hands.

Participants went through multiple sessions for data collection. Each session is similar to the study in Section 3.1, where participants saw the gesture name on the watch screen, followed a 3-sec countdown to perform the gesture, and repeated five times.

Every participant started with one session for each of the existing four gestures as a warm-up stage. Then, they had five data collection sessions. In each session, they performed each of the 12 new gestures 5 times. A Latin-square design was used to counterbalance the order effect. Participants took a 30-sec break between gestures and a 2-min break between each session to reduce fatigue. Moreover, to simulate the actual use case of taking the watch on and off over time, participants were asked to take off and put on the watch during the break to vary the watch position on the wrist, with a variation within 5 cm. The study was around 30 to 40 minutes. Overall, for each participant, we collected 5 repetitions per existing gesture, 25 repetitions (5 sessions $\times$ 5 repetitions) per new gesture.

We investigated two factors that have important design implications: the number of shots used for training and the number of new gestures that the model is trained to recognize. For the first factor, we went through different numbers of training samples from the original training set (from 1 shot to 10 shots) to train the model. For the second factor, given the number of new gestures, we went through all possible combinations of the gestures, from one new gesture to four new gestures. In total, we trained and evaluated 475,800 models (10 shot numbers $\times$ 3 repetition $\times$ $\sum_{n=1}^{4}{12\choose n}$=793 gesture combinations $\times$ 20 participants).

Prediction Head Evaluation. We first evaluate the performance of the prediction head (the solid lines in Figure 8). When using only one shot to add a new gesture (i.e., users perform the gesture just one time), our framework can achieve an average gesture-level accuracy of 55.3% and an F1 score of 64.6%. The more shots the model has, the better the recognition performance. With three shots of a new gesture, our framework can achieve an average accuracy of 83.1% and an F1 score of 88.9%. The performance further increases to 87.2% and 92.0% with five shots, and 91.0% and 94.5% when using ten shots. Meanwhile, the models are robust to noisy data, with an average false positive rate af 0.02 times per hour when evaluated on daily activity non-gesture data.

Supporting more than one gesture is more challenging, but our framework achieves an average accuracy of 83.3% and an F1 score of 88.8% with three shots of two new gestures. When adding three new gestures, our method has an accuracy of 77.7% and an F1 score of 84.2%. For four gestures, our method still has an accuracy of 77.2% and an F1 score of 83.4%. More new gestures also lead to a slightly higher false positive rate, and we observe the same trend as more number of shots are included for training. This can be explained by the fact that the increased variety of the positive samples raises the difficulty of the classification task. But our models can maintain the false positive rate as low as 0.06 or 0.12 times per hour when adding two or four gestures. The evaluation results show promising potential of the framework.

Moreover, we investigate the individual performance of each gesture. Figure 9 reveals that the majority of the 12 gestures have good performance. Using three shots, 7 out of 12 gestures have F1 scores at least 90%. Spread, RotateOut, and Flex have F1 scores higher than 95%. In contrast, Slide has relatively lower performance. The difference can be explained by the fact that the sliding gesture has larger variation, or is hard for the accelerometer/gyroscope to capture the motion.

To evaluate how these new gestures are confused against each other, we also look into the confusion between each pair of gestures when both are added as new gestures. Figure 10 shows that Slide and Peace are relatively more easily confused with other gestures.

Combining Prediction Head and Pre-trained Model. After the prediction head is trained and applied in real-time, it works with the pre-trained model together for recognition. Therefore, we also evaluate the performance on the new gestures when combining the two models, as shown by the dashed lines in Figure 8. The results are very close to those tested solely on the prediction head, with a minimal performance drop of 1.7% on F1 scores. This indicates that the two models have a distinguishing focus on different gestures (i.e., new gestures v.s. existing four gestures) and barely confuse each other.

In addition to evaluating the performance of our framework on new gestures’ data and negative data, it is also important to measure how much the additional prediction head influences the recognition performance on the existing four gestures. Both the average accuracy and F1 score achieve 97.5% and 97.7% when applying the combined model on participants’ data of the existing four gestures. This shows that the recognition outcomes of the original four gestures are not impacted by the additional prediction head.

The real-time system in actual usage can recognize both new gestures and existing gestures. Therefore, we also evaluate the two models on all gestures, as shown by the dotted lines in Figure 8. When using one shot to add a new gesture, our framework can achieve an average F1 score of 84.3% on the five gestures. With three, five, and ten shots, the F1 score increases to 93.1%, 94.4%, and 95.4%. When adding two, three, or four gestures with three shots, the average F1 scores achieve 91.0%, 86.6% or 84.9% on the whole six, seven, or eight gestures. We summarize detailed results in Appendix Tab. 4

The results from Section 5.2.1 to Section 5.2.3 suggest that our framework can include new gestures and achieve good performance with only a few shots, without degrading the recognition accuracy on existing gestures.

We also compare our framework against a few other methods. Some of them are traditional computational methods, while some of them are deep-learning-based:

• •

  DTW. In some prior work (Liu et al., 2009), DTW can be used to recognize new hand waving gestures with only one sample as the template. We re-implement the algorithm in uWave (Liu et al., 2009) and test it using our datasets.

• •

  Traditional ML models. As deep learning methods are usually data-hungry, an alternative solution is to use off-the-shelf traditional ML models to lower the data requirement. We test both SVM and random forest as they are commonly used on wearable gesture recognition systems (e.g., (Zhang and Harrison, 2015; Georgi et al., 2015; Iravantchi et al., 2019a)). The input for these traditional models are the feature embeddings from the pre-trained model.

• •

  Fine-tuning on the pre-trained model. This method is one of the common transfer-learning-based solutions. Specifically, we remove the final layer of the pre-trained model and add a new layer with more output nodes (five original classes plus the number of new gestures). We copy the weights from the old layers for the old five nodes, and randomly initiates the weights for the new nodes. Then, we fine-tuning the model using new data.

• •

  Ablation study. In addition to other methods, we also remove one of the data augmentation, data synthesis, and adversarial regularization methods to evaluate each of their individual contribution to the final results.

To make a fair comparison, we use three shots and two gestures for consistency, and negative data is available in all methods. Table 2 presents both the window-level and gesture-level results. Our method significantly outperforms the traditional methods and the fine-tuning method by at least 12.1% on accuracy and 9.2% on F1 score. Moreover, the ablation study results show that each of the techniques helps improve the model performance.

The real-time performance of the system is similar to the offline results in Section 5.2. The average recognition accuracy and F1 score on the four existing gestures are 96.7% and 98.1%. For new gestures, the overall average accuracy and F1 score are 91.1% and 95.1%, respectively. The customized gestures defined by participants were diverse. Examples include snapping fingers, flicking fingers, making spiderman pose, etc.. Some gestures have zero misclassification during the study, such as Snap, Spiderman, Clap, and FiveFingerBend.

Meanwhile, false-positive rate was kept low. A few participants tried diverse non-gesture motion and the real-time system was robust to noisy data. On average, participants had 0.6 times of false positive throughout the whole study.

Table 3 summarizes the proposed gestures and their recognition performance. These results indicate the scalability and the robustness of our framework.

During the study, all participants added the RotateOut gesture smoothly. As for adding their own gestures, most participants succeeded at the first trial when adding their own gesture. This indicates that the system’s sanity check does not pose much restriction on users’ creativity.

A small number of participants’ new gesture did not went through due to the similarity check. Two participants’ first gestures (middle finger pinching and hang loose) were recognized as close to Pinch and Clench, respectively. P15 attempted to add a slow waving motion as the second gesture but it got recognized as being close to common daily activities. P18 first added PinkyPinch as the first new gesture, and then tried to add a thumb-tapping on index finger knuckle. It did not go through as it was recognized as being close to PinkyPinch. Moreover, some participants deliberately tried to confuse the system. After adding the first gesture, P2 intentionally added another similar gesture but it did not go through. The accelerometer signals of these new gestures and the wrongly recognized gestures were indeed similar. On the one hand, this indicates the robustness of the system; On the other hand, this reveals the room for improvement of our framework to distinguish closer gestures.

As for the model performance check, P1 and P19 got an accuracy below 80% when adding their gestures. Both of them chose to collect more shots and completed the addition.

The questionnaire results also suggest positive feedback from participants. Following the score calculation method (Bangor et al., 2008), we obtain the average overall SUS score as 87.2$\pm$8.3 out of 100. This indicates the high overall usability of our system. SUS has two sub-scale scores (Q4 and Q10 for “Learnability” and the rest for “Usability”). The learnability score was 84.5$\pm$14.3 (out of 100, high) and the usability score was 87.9$\pm$8.6 (out of 100, high). Both sub-scale scores further indicate that our system is easy-to-learn and easy-to-use. The details of each SUS questions can be found in Appendix Figure 12.

Moreover, results of the NASA-TLX questionnaire (on a 7-point Likert scale) indicate that participants had low task load when using the real-time system, which is in line with the SUS outcome. Participants reported low task mental load (1.9$\pm$1.1), physical load (2.7$\pm$1.3), and temporal load (2.3$\pm$1.3). They considered themselves paying low effort (2.0$\pm$0.9). Moreover, participants agreed that they had a good performance during the study (6.2$\pm$1.0) and there was little frustration (1.5$\pm$1.1). Both the SUS and NASA-TLX questionnaires’ results indicate good usability of our system.

In addition to the questionnaire results, participants provided positive comments about the system. 8 out of the 20 participants explicitly mentioned that they would love to use the system in daily life. A few participants were impressed by the system: “It works amazingly well! That’s beyond my expectation.” (P11). Some participants were happy to see how the system can be robust against noisy motion. P2 was excited when their confusing gesture was not accepted by the system. “The system could tell that they are too close and it did not let me add it. This is a very good design, [making the system] much more robust!” (P2). P1 liked the feedback when the system has sub-optimal performance. “Know that the system is not perfect is absolutely fine! It only took three samples! I also feel good that the system can inform me about the performance so that I can have a better expectation.” (P1). Some participants discussed a few potential use cases of our system. We will have more discussion in Section 7.2.

Overall, the recognition performance and subject feedback of the user study illustrate the good usability and promising future of our framework.