An Interactive Tool for Simulating Mid-Air Ultrasound Tactons on the Skin

We made several assumptions about mid-air ultrasound stimulation in order to lower the computational complexity of the simulation tool. In the literature, focused mid-air ultrasound at a single focal point creates a central oval-shaped vibration area with maximum amplitude, followed by four smaller vibration areas located at four directions apart 15 $cm$ from the center with about less than 30 percent of the center amplitude (side lobes) if the ultrasound device located at the 20 $cm$ distance (Carter et al., 2013; Wilson et al., 2014). The amplitudes of side lobes are nearly below the detection threshold at the maximum stimulation (Howard et al., 2019), while the side lobes vary on the distance between the device and the focal point due to changes in the angles of the ultrasound transducers. Therefore, our model assumes the ultrasound stimulation occurs at a target single circular point on the skin. In addition, we assumed that the skin of the user’s palm is a 2D plane, and the propagation of waves along the skin does not occur, considering the computational complexity of the model. We discuss the implications of these assumptions in Section 4.

Our model provides two configuration spaces for rendering mid-air ultrasound Tactons (Figure 1): (1) The temporal configuration includes five parameters of amplitude, AM frequency, envelope frequency, superposition ratio, and total duration; and (2) the spatiotemporal configuration includes three parameters of shape, size, drawing speed of a focal point trajectory. We selected these parameters based on the most frequently used parameters in Tacton design literature (Dalsgaard et al., 2022; Hajas et al., 2020; Obrist et al., 2015, 2013; Hwang et al., 2017a; Seifi et al., 2015; Park and Choi, 2011). Thus, users can observe the vibration waveform at any point on the skin by controlling these eight parameters of mid-air ultrasound Tactons.

1. Temporal configuration: We defined the temporal configuration of mid-air ultrasound Tactons using the mathematical equation $p(t)$, which controls five parameters:

where $U(t)$ represents the continuous ultrasound at 40 kHz (typical frequency from commercial mid-air ultrasound haptic devices), $M(t)$ is an AM sinusoid of $sin(2\pi f_{AM}t)$ with AM frequency $f_{AM}$, $E(t)$ is an envelope sinusoid of $sin(2\pi f_{e}t)$ with envelope frequency $f_{e}$, and $w_{AM_{1}}$ and $w_{AM_{2}}$ is the weights of two AM sinusoids (i.e., $M_{1}(t)$ and $M_{2}(t)$).

Amplitude ($A$) corresponds to the peak acoustic pressure at the focal point, ranging between 0% and $\pm$ 100%, as provided by a mid-air ultrasound device. While $A$ represents a commanded amplitude on the device, its relative intensity varies with height, which is the vertical distance between the focal point and the mid-air ultrasound device. The amplitude is known to have an inverted U-shaped relationship with height, reaching a maximum at 200 $mm$ (Raza et al., 2019) above the device. We applied the above findings to calculate relative intensity of $A$ as in Figure 2(b). AM frequency ($f_{AM}$) represents a temporal frequency that modulates $U(t)$ (Obrist et al., 2013, 2015). Here, $f_{AM}$ = 0 Hz indicates no AM rendering (i.e., $M(t)=1$), suggesting that STM rendering is necessary to create tactile sensations. Envelope frequency ($f_{e}$) refers to a frequency modulating $M(t)$ (Park and Choi, 2011) and $f_{e}$ = 0 Hz denotes a constant envelope (i.e., $E(t)=1$). Superposition ratio ($w_{AM_{1}}$:$w_{AM_{2}}$) is the mixing ratio of two sinusoidal signals for creating a superimposed signal. The simulation tool currently provides five ratios: 1:0, 0.75:0.25, 0.5:0.5, 0.25:0.75, 0:1 (Yoo et al., 2022; Hwang et al., 2017a; Lim and Park, 2023). Total duration ($t_{d}$) is the maximum $t$ of the vibrations. While the computational model can handle any duration, we set 10 seconds as the maximum in the current tool, in line with the Tacton durations common in user applications (Seifi et al., 2015).

2. Spatiotemporal configuration: We defined the spatiotemporal configuration as the spatial properties of the temporal formula $p(t)$, consisting of three parameters, shape, size, and drawing speed of a focal point trajectory in a 2D plane above the mid-air ultrasound device (Figure 1). In this configuration, shape represents the trajectory of a focal point in the 2D plane. Based on the literature (Rutten et al., 2019; Hajas et al., 2020), we provide five shapes: point, horizontal line, circle, regular triangle, and square. Size ($d$, in $mm$) refers to the length of the line, diameter of the circle, or the side length for the regular triangle and square. We set the maximum size as 60 $mm$, considering the typical size of a user’s palm (Shen et al., 2023; Hajas et al., 2020). Drawing speed ($v$, in $m/s$) denotes the velocity of a focal point’s movement. Combinations of these three parameters derive a drawing frequency ($f_{d}$), defined as the number of completions or revolutions of a trajectory per second (Freeman and Wilson, 2021; Wojna et al., 2023; Rutten et al., 2020), creating a spatiotemporal tactile sensation (i.e., STM) (Frier et al., 2018). The three parameters of shape, size, and drawing speed determine the 2D position of a focal point on the moving trajectory at $t$, represented as $\vec{x}(t)$ = ($x(t)$, $y(t)$). For example, when shape is the point, $\vec{x}(t)$ is a constant at (0, 0). When shape is the circle, $\vec{x}(t)$ can be expressed as ($\frac{d}{2}\cos(2\pi f_{d}t),~{}\frac{d}{2}\sin(2\pi f_{d}t)$). With the trajectory $\vec{x}(t)$ from a total of the three parameters, we denoted $p(t)$ at the focal point as $p(\vec{x}(t),t)$.

At a selected point $\vec{a}=(a,b)$ on the 2D plane above the device, we can define the Euclidean distance between $\vec{x}(t)$ (location of the focal point) and $\vec{a}$ as $D(\vec{x}(t),\vec{a})$. The distance determines the intensity of stimulation at $\vec{a}$, which we defined as $S(D(\vec{x}(t),\vec{a}))$ (Figure 2(a)). Thus, we defined the final intensity of stimulation as $A(\vec{x}(t),\vec{a})=A\cdot S(D(\vec{x}(t),\vec{a}))$ and expressed $p(\vec{x}(t),t)$ at $\vec{a}$ as:

With this implementation, our model facilitates tests for the temporal and spatiotemporal configurations, comprising five and three parameters, respectively (Figure 2(c)). In the interactive simulation tool, we provide the stimulated areas on the skin by $p(\vec{x}(t),t)$ (Figure 2(d) Left). Our tool also presents the temporal plot of the commanded Tacton (i.e., Equation 1) considering $U(t)$ as 40 $kHz$ sinusoid. For the temporal plot for the signal at $\vec{a}$ (i.e., Equation 2), we substituted 1 to $U(t)$ because the ultrasound stimulation acts as a constant pressure (Frier et al., 2018; Shen et al., 2023). Then we applied Fourier transform to the temporal waveform to estimate the frequency spectrum (Figure 2(d) Right).

We selected ultrasound Tactons for the measurements, focusing on AM frequency ($f_{AM}$) in the temporal configuration and size ($d$) and drawing speed ($v$) in the spatiotemporal configuration, as these three parameters mainly determine the AM frequency and the drawing frequency of the ultrasound Tactons which affects vibration spectrum. We did not use the other temporal parameters and we kept total duration at 1 second. In addition, we maintained shape as the circle and height at 200 $mm$. We used four $f_{AM}$ values: 0, 80, 140, and 210 Hz. We selected $f_{AM}$ = 0 Hz to test pure STM rendering ($PureSTM$) and $f_{AM}$ ¿ 60 Hz, as the power lines in our country introduces a constant 60 Hz measurement noise. We also used four combinations of size and drawing speed as ($d$ in $mm$, $v$ in $m/s$): (0, 0), (10, 6), (20, 12), (30, 18). We chose (0 $mm$, 0 $m/s$) to test pure AM rendering ($PureAM$), and maintained the same ratio of $\frac{v}{d}$ for the other three combinations to have the same drawing frequency at 191 Hz.

We rendered the ultrasound Tactons using the STRATOS Explore device by Ultraleap and measured the Tactons using a 1D laser vibrometer (Ploytec IVS-500) on paper (Figure 3(a)). Initially, we tried to measure the displacements induced by the ultrasound device on the skin of palm but the induced displacement was too weak and lower than the vibrometer’s minimum resolution. After much experimentation, we configured a system to measure displacements induced by the ultrasound Tactons on a thin paper. For the Tactons varying on AM frequency, size, and drawing speed, we measured vibrations at 5 points inside, on the trajectory, and outside the shape, spaced at 5 mm intervals (Figure 3(b)).

The preliminary measurement results showed similar results to those reported in (Chilles et al., 2019), although we measured the vibrations on paper. $PureAM$ introduced frequency harmonics (multiples of AM frequency or $f_{AM}$) (Figure 4). $PureSTM$ rendering showed frequency harmonics (multiples of drawing frequency or $f_{d}$), which appeared consistently for different drawing speed and size as long as the drawing frequency (the ratio of speed to size) was the same. The combinations of AM and STM ($AM{+}STM$) also resulted in frequency harmonics at multiples of both AM frequency and drawing frequency. Also, the highest magnitude occurred at the inputted AM frequency among all harmonic frequencies, regardless of the STM parameters.

Our model for $PureAM$ and $PureSTM$ simulated the exact AM frequency and drawing frequency ($f_{AM}$ and $f_{d}$) as observed in the measurements (Figure 4). In particular, simulations for $PureSTM$ showed the same frequency harmonics at multiples of the drawing frequency. However, for both $PureAM$ and $AM{+}STM$, the simulations and measurements showed less correspondence, perhaps due to the different characteristics between paper and human skin. The simulated waveform for $PureAM$ included a single component at the inputted $f_{AM}$ in the spectral domain, while the measured waveform for $PureAM$ introduced frequency harmonics at multiples of $f_{AM}$ in the measurement. In $AM{+}STM$, both the simulations and measurements showed the highest magnitude at the inputted AM frequency, regardless of the STM parameters. However, the simulations showed the $f_{AM}$ component and pair frequency components at multiples of $f_{d}\pm f_{AM}$, while the measurements included frequency harmonics at multiples of both $f_{AM}$ and $f_{d}$. These frequency harmonics have a much lower magnitude than the AM frequency, so the extent of their impact on user perception is not fully known.