HEAR-YOUR-ACTION: HUMAN ACTION RECOGNITION BY ULTRASOUND ACTIVE SENSING

Action recognition is one of the important technologies that is used for many applications such as robotics [MOLLER2021], healthcare [SUBASI2020, NWEKE2019, HAQUE2020], elderly behavior monitoring [DEEP2020, BUZZELLI2020] and suspicious behavior detection [AMRUTHA2020]. Many of these techniques utilize visual clues, such as RGB videos and images. Images contain a wealth of visual information about people and scenes. However, privacy concerns limit the use of scenes that may include identifiable information, such as faces.

To consider the privacy concern, the methods that using radio frequency (RF) signals [WANG2016, LI2019, LI2022], Wi-Fi signals [GENG2022, BIYUN2020, YAN2020], and acoustic signals [GAO2020listentolook, IRIE2019SeeingThroughSound, GINOSAR2019, DAS2017GestureUSMicroSoft, MELO2021GestureUS, SHIBATA2023ListeningHumanBehavior] are proposed. Although RF and Wi-Fi signals can detect fine-grained human postures due to their short wavelengths, the accuracy would degrade due to interference from the electromagnetic waves emitted by electronic devices. Acoustic signals are also interfered with ambient noise. However, if the frequency of the target sound is known, it can be restricted by filters. Furthermore, when using ultrasonic active sensing, the influence of environmental noise is less compared to audible sounds.

The sensing of a person through acoustic signals can be classified into two streams: passive and active sensing. Passive sensing involves capturing sounds emitted by objects. Recognition tasks, such as segmentation [IRIE2019SeeingThroughSound] and pose estimation [GAO2020listentolook, GINOSAR2019], have been performed based on audible sounds, particularly focusing on the voices. However, voices contain person-identifiable information and can be considered sensitive data from a privacy perspective. On the other hand, active sensing methods analyze the reflected signals from a person in response to the sounds emitted by the device. Therefore, these methods allow the acquisition of human movements without using personally identifiable information. Although the active sensing methods have been applied to gesture recognition [DAS2017GestureUSMicroSoft, MELO2021GestureUS], segmentation [TANIGAWA2022ECCVW], and pose estimation [SHIBATA2023ListeningHumanBehavior], human action recognition by non-invasive ultrasound active sensing has not been well established. In particular, there are no published action recognition datasets that focus on ultrasonic active sensing.

We propose a new task for human action recognition by ultrasound active sensing. We build our own datasets. To construct this dataset, we define 8 basic motion patterns, which include upper-/lower-/whole-body motions and motionless postures. Then, feature extraction was performed based on time series amplitudes of reflected waves, and action classification was evaluated using a support vector machine and a convolutional neural network. Our contributions are as follows: (1) We tackle a new task: action recognition by contactless ultrasound sensing; (2) Since there are no previous methods to handle this task, we create a new dataset; and (3) We conduct a comparison of features for classification.

In this section, we present the related work on recognizing action from acoustic sensing. The related works are summarized in Table 1. The acoustic sensing methods are categorized as follows: (1) passive sensing and (2) active sensing. Passive sensing method is used for action recognition [GAO2020listentolook], segmentation [IRIE2019SeeingThroughSound], 2D hand and arm pose [GINOSAR2019]. These methods use clues of ambient and subject sound including voices. Voices have been used for speaker recognition tasks therefore passive sensing has privacy concerns due to the usage of voices. Moreover, passive sensing methods cannot detect human information when the person does not emit any sound such as voices and footsteps.

On the other hand, active sensing allows for the acquisition of human information irrespective of whether a person is emitting sound or not. The active sensing method is used for gesture recognition [DAS2017GestureUSMicroSoft, MELO2021GestureUS] and 3D pose estimation[SHIBATA2023ListeningHumanBehavior]. In active sensing, single-frequency burst waves or chirp signals with temporally varying frequencies are employed as acoustic signals for the sound source. The burst waves are used to measure the distance to objects in [MELO2021GestureUS] and to detect spatial power distribution of reflected waves in [TANIGAWA2022ECCVW]. The method employing chirp signals enhances the accuracy of Time-of-Flight (ToF) by incorporating signals from multiple frequencies [DAS2017GestureUSMicroSoft]. Additionally, utilizing the frequency characteristics as features enables the estimation of complex tasks such as 3D pose estimation [SHIBATA2023ListeningHumanBehavior].

In our approach, we use active sensing for human action recognition to avoid obtaining privacy-related information. Chirp signals are used for the sound source of our active sensing because they enable us to obtain spatial propagation features in multiple frequencies and are used for other tasks such as gesture recognition and 3D pose estimation. The frequencies are limited to ultrasound range to avoid capturing voices in the audible range.

Human Action Recognition By Ultrasound Active Sensing: We propose a new task that estimates human action from ultrasound signals. To realize this task, the information we need to know is the changes in the emitted ultrasound from a speaker when it is captured by microphones. This is the same manner of the room impulse response when using audible sound. The ultrasound transfer features are changed along with the environment of the space. If a person moves inside the space, the transfer features of ultrasound would be changed. Therefore, we established an ultrasound active sensing system for human action recognition and considered feature extraction methods suitable for the action recognition task. Our concept is described in Fig. 1. We closely set a speaker and microphone opposite to humans and captured reflected signals from the environment and humans. Analyzed reflected signals to extract features are used for action classifications.

Active Sensing System: The schematic diagram of the active sensing system used in this research is described in Fig. 2. We used a tweeter to emit ultrasound. As a receiver, we used two MEMS microphones. The microphone was placed 38 mm below and 22.5 mm away from the center of the speaker. The sampling frequency was set to $f_{s}=96$ kHz enabling to capture of ultrasound range.

Signal Design: We design a linear chirp signal for the active sensing. The linear chirp signal $x$ is a signal where the frequency increases linearly over time and calculated as

$$x(t)=\sin\left(2\pi\left(\frac{\beta}{2}t^{2}+f_{0}t\right)+\phi_{0}\right),$$

(1)

where $t$ is the time, $f_{0}$ is the lower bound frequency, $\phi_{0}$ is the initial phase. $\beta$ is the coefficient determined by the time length of the chirp signals $\tau$ and the lower and upper bound frequencies $f_{0}$ and $f_{1}$, respectively: $\beta=(f_{1}-f_{0})/\tau.$ In our approach, we set the frequencies to $f_{0}=20$ kHz and $f_{1}=40$ kHz. To determine the time length of the chirp signal, we considered the minimum distance of the sensing area. We need to avoid interfering the direct ultrasound and reflected ultrasound when we capture the reflected ultrasound of human. Therefore, the time length of the chirp signal is limited as $\tau\leq(2d_{\mathrm{min}})/c$, where $d_{\mathrm{min}}$ is the minimum distance of the sensing area and $c$ is the speed of ultrasound in air. We set $d_{\mathrm{min}}=0.30$ m; therefore, we set $\tau=1.5$ ms.

To observe the transfer feature changes over time, cyclic chirp signal emission is required. To do so, we determined the cycle time $T$ of the chirp signals based on the maximum distance of the sensing area $d_{\mathrm{max}}$: $T\geq(2d_{\mathrm{max}})/c$. We set $d_{\mathrm{max}}=2.0$ m; therefore, we set $T=11.8$ ms.

Feature Extraction: While common feature extraction methods have been well-established in audible passive sensing, such as Short-Time Fourier Transform and Mel-Frequency Cepstrum Coefficient, there is limited research in the context of active sensing, and the feature extraction methods are not well-established. Hence, we considered the following four types of features: (1) time-series reflected waves, (2) time-series envelopes of reflected waves, (3) time-series impulse response of direct and reflected waves, and (4) time-series envelopes of impulse response of direct and reflected waves.

Time-series reflected waves: The first feature value is time-series reflected waves. Let the received signal of a microphone as $\mathbf{y}$, it includes direct signal and reflected signal. The direct signal is the wave that directly approaches from the speaker, and the reflected signal is the wave propagated to the space and reaches the microphones. Since the reflected waves propagate through the space occupied by a person, we extract each reflected wave from a single chirp signal and concatenate them as follows: $F_{\mathrm{ref}}=[\mathbf{y}_{1},\mathbf{y}_{2},\ldots,\mathbf{y}_{N}]$, where $F_{\mathrm{ref}}$ is the time-series reflected waves and $\mathbf{y}_{i}\,(i=1,2,\ldots,N)$ is the $i$-th reflected wave: $\mathbf{y}_{i}=\mathbf{y}[N_{\mathrm{dir},i}+N_{\mathrm{min}},\ldots,N_{\mathrm{dir},i}+N_{\mathrm{max}}]$, where $N_{\mathrm{dir},i}$ is the index of $i$-th direct wave, $N_{\mathrm{min}}=2f_{s}d_{\mathrm{min}}/c$ is the index corresponding to the minimum distance of the sensing area $d_{\mathrm{min}}$, and $N_{\mathrm{max}}=2f_{s}d_{\mathrm{max}}/c$ is the index corresponding to the maximum distance of the sensing area.

Time-series envelopes of reflected waves: The second feature value is the time-series envelopes of reflected waves. Since the phase of reflected waves greatly depends on the shape of the reflecting object, it is considered as environment-dependent information. To focus on the amplitude of reflected waves, the envelopes of the amplitude of each reflected wave are extracted and concatenated as the time-series envelopes of the reflected waves $F_{\mathrm{renv}}$: $F_{\mathrm{renv}}=[\mathbf{\hat{y}}_{1},\mathbf{\hat{y}}_{2},\ldots,\mathbf{\hat{y}}_{N}]$, where $\mathbf{\hat{y}}_{i}\,(i=1,2,\ldots,N)$ is the $i$-th envelope of reflected wave.

Time-series impulse response of direct and reflected waves: The third feature value is the time-series impulse response of direct and reflected waves. Both the direct wave and reflected wave are emitted from the same speaker and received by the same microphone. Therefore, by calculating the impulse response of the direct wave and reflected wave, the signals can be obtained that eliminate the characteristics of the speaker and microphone. When $Y_{\mathrm{dir}}$ and $Y_{\mathrm{ref}}$ represent the frequency-domain signal of the direct and reflected wave, respectively, the transfer function $H_{\mathrm{rd}}$ of the direct wave and reflected wave can be expressed as $H_{\mathrm{rd}}(\omega)=Y_{\mathrm{ref}}(\omega)/Y_{\mathrm{dir}}(\omega)$, where $\omega$ is the angular frequency. The feature value of the time-series impulse response of direct and reflected waves is formed by concatenating the $i$-th time-domain signal $h_{\mathrm{rd},i}$ of the transfer function $H_{\mathrm{rd},i}$: $F_{\mathrm{ir}}=[\mathbf{h}_{\mathrm{rd},1},\mathbf{h}_{\mathrm{rd},2},\ldots,\mathbf{h}_{\mathrm{rd},N}]$.

Time-series envelopes of impulse response of direct and reflected waves: The last feature value is the time-series envelopes of impulse response of direct and reflected waves. The value is formed as $F_{\mathrm{ienv}}=[\mathbf{\hat{h}}_{\mathrm{rd},1},\mathbf{\hat{h}}_{\mathrm{rd},2},\ldots,\mathbf{\hat{h}}_{\mathrm{rd},N}],$ where $\mathbf{\hat{h}}_{\mathrm{rd},i}$ is the $i$-th envelope of the impulse response of direct and reflected waves.