Forearm Ultrasound based Gesture Recognition on Edge

The data comprised 4 gestures, namely, open hand gesture, and finger pinch gestures. The latter was defined to be an index-thumb pinch, middle-thumb pinch, and ring-thumb pinch. The gestures are shown in Figure 2 with the corresponding ultrasound image output on a screen. Each image was 640 x 640 pixels with total 2400 frames corresponding to 8 rounds of data acquisition (300 frames per round) with an average frame rate of 10 Hz. The data was subjected to 25% test-train split, leading to 1800 train samples and 600 test samples.

The network architecture is based on the CNN proposed in [4, 5]. The network uses 5 cascaded convolution layers with normalization and max-pooling, followed by flattened and dense layers. The output is 4 class probabilities for each of the gestures for a given ultrasound image. Adam optimizer and a learning rate of 1e-3 were used. The models were trained for 25 and 50 epochs for baseline performance analysis, and 20 epochs for deploying and demonstrations. All the development for training was done on a Windows 10 system, with the models trained on an NVIDIA GeForce RTX 2070 SUPER GPU using TensorFlow and Python [7].

We utilized a Raspberry Pi 4 running Debian for our on-device deep neural network system. The trained models were converted to TensorFlow Lite (TFLite) models using TensorFlow Lite [10] to facilitate efficient on-device inference. Communication between the Raspberry Pi and the Windows system was achieved using the User Datagram Protocol (UDP). This setup allowed ultrasound images to be initially captured on the Windows system, downsampled, and then transmitted to the Raspberry Pi via UDP. The original 640 x 640 images were downsampled to 80 x 80, reducing the data size by a factor of 8. This downsampling was necessary to ensure smooth communication within the constraints of UDP’s data transfer limits, avoiding buffer overflow and minimizing data loss and transfer time. In this configuration, the Raspberry Pi acted as the server, while the Windows machine functioned as the client, capturing and sending ultrasound data to the Raspberry Pi as illustrated in Figure 1.

To implement model compression, we employed quantization as the primary approach, exploring three distinct techniques: Float16 quantization, Dynamic Range Quantization, and UInt8 quantization [11]. Initially, the input tensors to the network were in Float32 format. During Float16 quantization, weights were converted to 16-bit floating point values during the conversion of the model from TensorFlow to TensorFlow Lite’s flat buffer format. In Dynamic Range Quantization, weights were quantized post-training, while activations were dynamically quantized during inference. Notably, the model weights were not retrained to mitigate quantization-induced errors in this technique. UInt8 quantization approximates floating-point values by subtracting the zero-point value from the int8 value and multiplying the result by a scaling value. Each quantization method exhibits specific behaviors in terms of accuracy degradation and model compression.

The size of the models is described in bytes and megabytes (MB). Accuracy percentage was used as the metric to evaluate the performance of the trained model for both offline and online estimation.

Using the CNN, the baseline performance obtained on the GPU is shown in Figure 3(a) and (b). After 25 epochs, the training data achieved an accuracy of 92.3%, while the accuracy on the test data reached 91.5%. Upon extending training to 50 epochs, the accuracy on the training data increased to 96%, yet the accuracy on the test data slightly decreased to 90%. Despite the improvement in training performance, there was a marginal decline in test performance, indicating potential overfitting to the training data, considering the independence of the train and test sets.

For the final analysis and demonstration, we employed a separate model trained for 20 epochs, which was subsequently optimized for UDP-based transfer. The results for this model are shown in Figure 3(c). For this model, the train accuracy reached 93% and the test accuracy was 92%.

It was observed that upon employing the tflite converter, the model size decreased significantly from $\sim$1.5 MB to $\sim$0.46 MB. Subsequently, with the implementation of Float16 quantization, the size was further reduced to approximately 0.24 MB. Dynamic range quantization and UInt8 quantization resulted in an additional halving of the model size. However, the training and testing accuracy significantly dropped with UInt8 quantization. Conversely, for the other techniques, the accuracy remained high at 96% for training and 92% for testing. Notably, dynamic range quantization led to a marginal decrease in accuracy by 1

In terms of inference latency, the highest latency was obtained for dynamic range quantization and the lowest for the UInt8 quantization. Although UInt8 quantization exhibited the least latency for inference on the test set, its poor performance necessitated consideration of the next best option, which was Float16 quantization. These results are summarized in Table I.

To illustrate our approach, we utilized the tflite model deployed on the Raspberry Pi to evaluate training and testing accuracy as well as inference times. For the training data, we selected a total of 600 frames. The inference time averaged at 9.2 ms per sample. The accuracy percentage achieved was 86%, slightly lower than the baseline performance observed over 20 epochs (refer to the confusion matrix in Figure 3(c)). For the test set, the inference time remained consistent, while there was a marginal accuracy improvement of 0.1%. In terms of power consumption, our desktop system (with GPU) has a wattage of 420W, compared to <5W for a Raspberry Pi [12].

There are two demonstrations which can be seen at this link: https://youtu.be/tHTCzyHD7ak.

In the first demonstration, ultrasound data is downsampled on the Windows end before being transferred over UDP to the Raspberry Pi. To ensure reliable UDP data transfer, an artificial delay of 0.1 s was introduced. We achieved an accuracy of 85.3% on the test set with an end-to-end inference time of 172 ms per frame.

In the second demonstration, we plotted 10 arbitrarily selected ultrasound images on both Windows and Raspberry Pi before and after the transfer. While this increased the latency significantly, it was essential to visually demonstrate the transfer of ultrasound data from Windows to Raspberry Pi. Despite the increased latency, the demonstration clearly shows the frames being transmitted. Of the 10 sampled transfers, 2 misclassifications resulted in an 80% classification accuracy. On average, it took approximately 2 seconds per sample.

The significant reduction in model size achieved through quantization techniques is crucial for wearable applications, where memory and computational resources are limited. Our findings show that it is possible to maintain high accuracy and low latency, even with substantial model compression. Compared to previous studies that required powerful desktop GPUs, our approach demonstrates that real-time ultrasound-based gesture recognition is feasible on edge devices, making it a viable option for practical applications in healthcare and human-computer interaction.