Can a Machine Feel Vibrations?: A Framework for Vibrotactile Sensation and Emotion Prediction via a Neural Network

Over the decades, extensive research have explored myriad design parameters for Tactons to effectively convey and communicate information and emotions in various haptic user interfaces [27]. Prior studies have investigated low-level parameters of Tactons, such as carrier frequency [28, 29], envelope frequency [21, 20], duration [30, 14], and amplitude [28, 29], as well as superposition of multiple sinusoids [31, 32] and combinations of multiple parameters [14, 20]. Other studies have proposed high-level parameters of Tactons, such as rhythmic structure [22, 33, 34], which include the evenness of pulses and note length, interval between vibrations [35], and sound waveform or timbre [9, 36]. Designers can create Tactons by systematically varying these parameters (i.e., parameter-based design approach). For constructing a haptic dataset, we select four common low-level Tacton parameters – carrier frequency, envelope frequency, duration, and amplitude – and one high-level parameter, rhythmic structure.

In addition to creating Tactons from scratch, haptic designers can create new Tacotns by transforming libraries from other modalities into vibration libraries or by modifying template Tactons from existing vibration libraries [37]. Past studies have proposed various vibration libraries together with their associated user feelings or metaphors (i.e., metaphor-based design approach). For example, van Erp and M.A. Spapé created 59 Tactons by transforming auditory melodies into vibrations and examined their perceptual impacts [38]. Disney Research introduced FeelEffects, a library of over 40 Tactons, and investigated the semantic and parametric spaces of these Tactons [39]. Seifi et al. proposed VibViz, a library of 120 Tactons with subjective ratings and descriptive tags on their physical, sensory, emotional, usage, and metaphoric attributes [16]. These libraries consist of Tactons with complex waveforms, such as intricate rhythmic structures in the time domain, varying frequency spectra over time, and various durations. We include 40 Tactons by modifying existing Tactons from the open-source vibration library VibViz for rendering on iPhones, to enhance the diversity of our haptic dataset.

Investigating how humans perceive vibrations is a fundamental aspect of the haptics field. Prior research has studied the neurophysiological processes underlying tactile perception, identifying four types of mechanoreceptors in human skin that contribute to the perception of touch [40]. These four mechanoreceptive tactile channels have different characteristics, such as perception properties, spectral sensitivities, and sensory adaptation rates [41, 24]. Among these four mechanoreceptive channels, Meissner Corpuscle (RA1) and Pacinian Corpuscle (RA2) are primarily activated by vibrotactile stimuli compared to Merkel Disk (SA1) and Ruffini Ending (SA2). Drawing on the properties of the two mechanoreceptive channels (RA1 and RA2) most related to vibrations and their spectral sensitivities, we propose a mechanoreceptive processing approach that converts vibration signals into two-channel spectrograms.

Previous studies have also explored the perceptibility and discriminability of vibrations. Psychophysics researchers have examined the detection thresholds, or Absolute Limens (AL), for vibrations and uncovered a U-shaped threshold curve across frequencies, with the highest sensitivity occurring around 200 Hz (between 150 Hz and 300 Hz) [42, 43]. Other studies have investigated the discrimination thresholds, or Just Noticeable Differences (JND), between vibrations [29, 44, 45, 46]. JNDs for vibration intensity generally range from 10% to 30%, while JNDs for vibration frequency typically fall between 15% and 30% [24]. Some studies have also explored the relative impact of these parameters; increasing the duration of vibrations has been shown to decrease the JNDs for vibration intensity [47, 48]. We use these established findings on detection and discrimination thresholds for vibrations to augment our vibration signals while ensuring that the augmented vibrations remain perceptually indistinguishable to users.

The sensations and emotions elicited by haptic stimuli are also important research topics in HCI. Past research has developed sensory and emotional lexicons associated with haptic stimuli [49, 50, 51, 52]. Researchers have collected user ratings based on Russell’s circumplex model of affect, which defines two key emotional dimensions: valence and arousal [53]. Guest et al. identified 26 sensory attributes and 14 emotional attributes for describing touch and demonstrated that valence and arousal are primary factors in haptic emotional experience [54]. Among sensory attributes, researchers identified roughness as the primary dimension for capturing the perceptual properties of real textures or materials, with low variability among individual users. Studies on Tactons reported analogous findings: participants associated Tactons with roughness effectively but often faced challenges in associating hardness with Tactons, and sensory attributes like wet/dry or hot/cold were not relevant to vibrations [15, 16]. Additionally, while temporal attributes such as tempo, energy, and rhythm are relevant for describing certain vibration patterns, they are not considered primary sensory dimensions, as they are context-dependent and less generalizable across different tactile scenarios. Building on these established frameworks, we select roughness as the primary sensory dimension for Tactons and valence and arousal as the two primary emotional dimensions.

To inform the design of Tactons that convey specific sensations and emotions, previous studies have sought to derive sensory and emotional spaces for Tactons and identify effective design parameters. After creating one or more sets of Tactons, designers typically recruit participants to gather subjective responses to these Tactons. The sensory and emotional spaces are then visualized using averaged ratings for each Tacton, or statistical tests are applied to determine which design parameters significantly impact the sensations and emotions. Seifi and Maclean designed 14 Tactons and demonstrated that rhythmic structure influenced all three dimensions (roughness, valence, and arousal) and that carrier frequency affected arousal [15]. Yoo et al. explored emotional spaces with three different Tacton sets (25, 36, and 24 patterns) and provided design guidelines for four sinusoidal parameters [14]. Seifi et al. collected roughness, valence, and arousal ratings for 120 Tactons and developed the VibViz visualization to assist haptic designers [16]. While these studies have examined emotional and sensory spaces through controlled laboratory user studies (i.e., offline studies), recent research has explored the efficacy of crowdsourcing user studies (i.e., online studies) to reduce the effort and time needed to collect sensory and emotional ratings of Tactons [55, 26]. However, these studies still rely on the efforts of researchers and designers in evaluating Tactons, which limits their scalability. To address this limitation, we construct sensory and emotional ratings of 154 Tactons, the largest Tacton set ever studied, and develop a model that predicts roughness, valence, and arousal ratings to enable the rapid prototyping of effective Tacton candidates.

Computational models that predict subjective evaluations of haptic stimuli or compare different haptic stimuli can help designers and researchers enhance user experiences and accelerate the design process in various applications. Previous research has developed models to predict haptic perceptions and sensations for objects or textured surfaces. These studies used robots or rigid tools to collect haptic data generated from physical interactions, such as accelerations, forces, and speeds. The resulting models predict sensory attributes elicited from objects [56, 57, 58], textured surfaces [59, 60], or perceptual similarities between textured surfaces [61]. While these models primarily focus on real objects or textured surfaces, we propose a model that predicts the sensations and emotions elicited by Tactons conveyed through a single vibrotactile actuator in haptic interfaces, which generates 1D accelerations.

Past studies have also proposed computational models for predicting subjective evaluations for vibrations rendered on a vibrotactile actuator. Park and Kuchenbecker introduced algorithms to transform three-axes accelerations collected from human interactions with textured surfaces into one-axis accelerations that can be played on a vibrotactile actuator [62]. They then developed a model to assess the perceptual similarities between the haptic stimuli generated from human interactions with textured surfaces and the converted one-axis accelerations. Others researchers proposed models for evaluating and comparing perceptual qualities between original and compressed vibration signals [63, 64, 65]. Recent work introduced a model to predict perceptual dissimilarities between Tactons by simulating the neural transmission from mechanoreceptors in the skin to the brain [20]. Previous studies primarily focused on developing models to predict perceptual distinguishability between real-world haptic stimuli and vibrations or between different Tactons. In contrast, we propose a computational model that predicts sensations and emotions elicited by Tactons. We note that unless perceptual differences between Tactons are extremely subtle (i.e., Tactons are not easily distinguishable), users may perceive their sensory and emotional attributes differently. This highlights the need for vibrotactile sensation and emotion prediction models, as perceptual distinguishability models alone may not capture the nuanced mappings between vibrations and user-rated sensation and emotions. Our approach enables efficient prediction for new and unique Tactons that may not closely resemble existing vibrations in the dataset.

We created a set of 154 Tactons to provide haptic dataset to the neural networks (Figure 2). The design of this set was informed by prior literature on Tacton design, utilizing sinusoidal parameters (54 Tactons), rhythmic structures (60 Tactons), and complex Tactons that evoke multiple attributes such as sensations, emotions, and metaphors (40 Tactons).

The 54 Tactons (V1–V54) systematically varied in signal parameters, including three carrier frequencies (80 Hz, 155 Hz, and 230 Hz), three envelope frequencies (0 Hz, 4 Hz, and 8 Hz), three durations (300 ms, 1000 ms, and 2000 ms), and two amplitudes ($half$ and $full$) (Figure 2a). We selected these parameters to cover most of the emotional space of sinusoidal vibrations [14] while ensuring compatibility with iPhone playback. We rendered the sinusoidal vibrations using the mathematical formula for temporal envelopes $E(t)$ and temporal frequencies $F(t)$ for generating Tactons on iPhones, as follows:

Here, $A$ represents the amplitude in the Apple Haptic and Audio Pattern (AHAP) format, where $half$ and $full$ correspond to 0.5 and 1, respectively. $f_{e}$ denotes the envelope frequency, where $f_{e}=0$ Hz represents a constant envelope (i.e., $E(t)=A$), and $f_{c}$ denotes the carrier frequency.

The 60 Tactons (V55–V114) varied on 10 rhythmic structures from [22] and two signal parameters of three carrier frequencies (80 Hz, 150 Hz, and 230 Hz) and two amplitudes ($half$ and $full$) (Figure 2b). We selected these rhythmic structures to cover most of the perceptual space of rhythmic Tactons, considering primary parameters such as note length and evenness. All 60 Tactons lasted for 2 seconds.We rendered the rhythmic Tactons using the following formula for temporal envelopes $E(t)$ and temporal frequencies $F(t)$:

Here, $A$ and $f_{c}$ represent the amplitude in the AHAP format and the carrier frequency, respectively. $R_{t}$ is a list of binary pulses, each with a length of 31.25 ms, as shown in [34, 66].

Lastly, we designed 40 complex Tactons (V115–V154) using the Vibviz library [16] (Figure 2c). We divided the original emotional space of Tactons in the library into nine segments and selected 40 Tactons based on their distribution in the original sensory and emotional spaces. We rendered the Tactons by transforming original vibration waveforms into temporal envelopes $E(t)$ and temporal frequencies $F(t)$ suitable for iPhones. These Tactons featured more complex waveforms compared to the 54 sinusoidal and 60 rhythmic Tactons. In addition, these complex Tactons used variable $F(t)$ over time, whereas the 54 sinusoidal and 60 rhythmic Tactons used constant frequency (i.e., $F(t)=f_{c}$). The durations of the 40 Tactons ranged from 0.43 to 5.38 seconds.

We recruited 36 participants (18 women and 18 men; 18–31 years old (mean: 22.8, SD: 3.5)), including three left-handed and 33 right-handed users. All participants had no sensory impairments in both hands. The participants took 65 minutes on average to complete the study and received $30 USD as compensation.

We used three iOS smartphones (iPhone 13 mini, iPhone 14, and iPhone 11 Pro Max) by Apple Inc. to collect acceleration data as well as sensory and emotional ratings of Tactons on various consumer phones. These phones varied in size and mass: iPhone 13 mini ($64.2\times 131.5\times 7.65$,mm, 141,g), iPhone 14 ($71.5\times 146.7\times 7.8$,mm, 172,g), and iPhone 11 Pro Max ($77.8\times 158.0\times 8.1$,mm, 226,g). We attached a 3-axis accelerometer (Analog Devices; ADXL354z) on the right to the smartphone camera (Figure 3a) and measured the accelerations of the vibrations using a DAQ board (National Instruments; USB-6353) with the sample rate of 10 kHz. We collected three-axis accelerations but used one axis (left-right) aligned with the vibration direction of the Taptic Engines in the iOS smartphones when training a neural network, as the other two axes consisted of noise unrelated to the Tactons. We collected participants’ responses using a graphical user interface (GUI) on the smartphones (Figure 3b). Participants placed both hands on a table in front of them and maintained the same holding posture with their left hand and a natural grasp force while interacting with the GUI application using their right hand. The participants wore noise-canceling headphones with white noise to block any environmental sounds.

The study process included a sequence of sessions: introduction, training session, and main session. Participants first completed the consent form and a demographics pre-questionnaire. Next, participants began the experiment using the GUI application on one of three smartphones, with participants evenly distributed by biological sex across the devices. The training session displayed a set of buttons, each randomly corresponding to a Tacton (Figure 3b left). Participants experienced all the Tactons before the main session, while we collected acceleration data from their interactions using the accelerometer. If a participant played a Tacton multiple times, the acceleration data from the last interaction was stored. In other words, we measured the accelerations from 12 participants (six women, six men) per smartphone, resulting in acceleration data for 154 Tactons from 36 participants across the three smartphones(Figure 4a).

In the main session, participants rated roughness (smooth/rough), valence (unpleasant/pleasant), and arousal (calm/exciting) of 154 Tactons using sliders ranging from 0 to 100 (Figure 3b right). The application presented Tactons in a random order. Participants could play the Tactons multiple times but were unable to modify ratings for previously rated Tactons. Throughout the study, participants could take breaks as needed, and we maintained the room temperature between 20–23 degrees Celsius.

We collected 5,544 acceleration data (154 Tactons $\times$ 36 participants) and downsampled all the data from a 10 kHz to a 1 kHz, a sample rate that captures the range of human tactile receptors. This was done before augmenting the vibration signals and inputting them into the neural network to improve the efficiency of model training.

In the comparisons in ratings between the three devices (averaged for 12 participants per device), roughness and arousal ratings showed very strong correlations. For roughness: Pearson’s correlation $r$ = 0.91 (iPhone 14 vs. iPhone 13 mini), 0.90 (iPhone 14 vs. iPhone 11 Pro Max), and 0.90 (iPhone 13 mini vs. iPhone 11 Pro Max), all with $p<0.01$. For arousal: $r$ = 0.87 (iPhone 14 vs. iPhone 13 mini), 0.88 (iPhone 14 vs. iPhone 11 Pro Max), and 0.88 (iPhone 13 mini vs. iPhone 11 Pro Max), all with $p<0.01$. In contrast, valence ratings showed low to moderate correlations between the iPhone 14 and the other two devices ($r$ = 0.38, $p<0.01$ for iPhone 13 mini and $r$ = 0.47, $p<0.01$ for iPhone 11 Pro Max), while the correlation in valence ratings between the iPhone 13 mini and iPhone 11 Pro Max was strong ($r$ = 0.72, $p<0.01$). Therefore, we used the averaged ratings for each device to train the neural network. Overall, valence ratings showed a negative relationship with arousal ratings (mean $r=-0.79$ across the three devices), while arousal ratings had a positive relationship with roughness ratings (mean $r=0.89$), as shown in [16]. The standard deviations for roughness, valence, and arousal ratings of the 154 Tactons across 36 participants were 17.2, 19.1, and 17.3 (out of 100), respectively (Figure 4b).

In contrast to the extensive research and techniques available for data augmentation in images or audio, limited techniques exist for augmenting haptic data. To address this, we propose three vibration augmentation techniques informed by human haptic perception and audio data augmentation methods, particularly for waveforms in the time domain, while ensuring that users cannot distinguish between the original and augmented vibrations (Figure 5). Five users confirmed that accelerations augmented by the following three techniques and their four combinations (=$\binom{3}{2}$+$\binom{3}{3}$) were indistinguishable through 3AFC (Three-Alternative Forced Choice) testing, where participants were presented with two identical stimuli (A, A; both being the original vibration) and one different stimulus (B; the augmented vibration) and asked to identify the different one. For the seven augmentation cases, the average probability of participants correctly identifying the augmented vibration converged to around 0.33, suggesting the perceptual invariance of the augmented vibrations.

To better capture human vibrotactile processing, we supplemented the 1D waveform haptic data, which contains only amplitude values and lacks explicit frequency information, with additional data in a different form. Previous research on developing neural networks for audio data introduced a method to convert the audio classification/prediction problem into an image classification/prediction problem by transforming waveform audio data into a spectrogram [68, 69]. This conversion can complement the waveform data by segmenting its duration into smaller time intervals and representing the frequency content of these segments in a single plot through the STFT. In addition, since the converted spectrogram is an image, it is well-suited for input into CNN-based architectures designed for image processing.

When processing the conversion, we applied two bandpass filters based on the spectral sensitivity of mechanoreceptors involved in coding touch information for perception. In vision research, neural networks are typically developed using three-channel (RGB) images [67, 70], as the cone photoreceptors in the retina, which are responsible for visual processing, are classified into three types, each exhibiting different spectral sensitivities and therefore activated by photons of specific wavelengths [71]. Similarly, each of the four types of mechanoreceptors involved in tactile processing have distinct spectral sensitivities and activation patterns. However, we used two bandpass filters corresponding to the spectral sensitivities of Meissner Corpuscle (RA1) [3 Hz, 100 Hz] and Pacinian Corpuscle (RA2) [40 Hz, 500 Hz], as these two mechanoreceptors are most relevant to vibration perception [72]. In addition, we also tested neural networks using four mechanoreceptive filters but did not find significant improvements in performance compared to those using two filters (see Section VI-B and Table I).

We applied the mechanoreceptive processing technique to 83,160 augmented 1D waveforms, creating two-channel 2D spectrograms with raw floating-point numbers, where time is on the x-axis and frequency on the y-axis. To ensure consistency in data sample length, we first added zero padding to the end of each acceleration data to make all lengths identical at 6,000 samples (i.e., 6-second durations), accounting for the maximum duration of the Tactons in our dataset, which varied in duration. We then applied STFT to the two signals from the above mechanoreceptive filters, using a 0.5-second window and a 0.05-second hop size to achieve a spectral resolution of 2 Hz in the spectrograms. As a result, we derived two-channel spectrograms with dimensions of (251, 121), and then fed both the 83,160 1D acceleration data entries and their corresponding two-channel spectrograms into the neural network.

VibNet consists of two parallel neural network streams, each designed to capture different aspects of the vibration data (Figure 6). The first stream employs Gated Recurrent Units (GRU) layers [19]. GRU networks use a reset gate and an update gate in each unit to capture temporal dependencies and are known for their training efficiency compared to Long Short-Term Memory (LSTM) networks, as GRUs have fewer parameters. Given the significant impact of temporal and rhythmic structures on human vibrotactile perception [20, 21, 22], we selected GRU networks to effectively capture the sequential features of Tactons. This stream consists of two layers, each with 1024 GRU units. The first layer processes the 1D waveform data as input and passes it to the next layer. The final GRU layer flattens the output before passing it to the concatenation layer.

The second stream employs a ResNet architecture [23], a type of Convolutional Neural Network (CNN), to process the two spectrograms from each acceleration signal. ResNet uses residual connections to allow the training of very deep networks by addressing the vanishing gradient problem, and it has demonstrated state-of-the-art performance with fewer parameters compared to traditional plain CNNs, such as VGGNet [73]. We used a 154-layer ResNet with bottleneck blocks to capture the temporal-spectral features from the two-channel 2D spectrograms, effectively transferring the model’s capabilities from image processing to haptic data processing. As in the first stream, the final layer of this stream flattens the output and passes it to the concatenation layer.

The concatenation layer receives the outputs from the final layers of both streams. This is followed by three fully connected layers with 1024, 128, and 16 neurons, respectively, applying the ReLU activation function and a dropout rate of 0.5 between layers. The final layer, consisting of 16 neurons, regresses the values to provide three outputs corresponding to predictions for roughness, valence, and arousal ratings.

We implemented VibNet using the PyTorch library, and trained it on the abovementioned haptic dataset using an NVIDIA GeForce RTX 3080 Ti GPU. We employed the Adam optimizer with a batch size of 32 and used Mean Square Error (MSE) as the loss function. The learning rate was set to 0.001, and the model was trained for 100 epochs, following commonly-used values in DL [67]. To ensure robust performance, we applied 5-fold cross-validation during the training process, balancing training efficiency and model accuracy given the complexity of the dataset and architecture.

We verified VibNet against five baseline models, including conventional machine learning models and standard neural networks in the haptics field [59, 58]. Additionally, we conducted an ablation study [74] to assess the impact of the multi-channel spectrogram inputs and the corresponding CNN layers derived from the proposed mechanoreceptive processing (Section IV-B). We compared the proposed framework — which utilizes two-channel spectrograms processed by two mechanoreceptive filters (RA1 and RA2) — with neural networks that applied (1) STFT without mechanoreceptive filters and (2) STFT with four mechanoreceptive filters (RA1, RA2, SA1, and SA2) in the second stream constructed by CNN layers. We specifically tested the approach (2) because Merkel Disk (SA1) and Ruffini Ending (SA2) are known to contribute less to vibration perception compared to the Meissner Corpuscle (RA1) and Pacinian Corpuscle (RA2) [72, 24]. To include these two additional channels, we applied a lowpass filter at 5 Hz and a bandpass filter of [15 Hz, 400 Hz] considering the spectral sensitivities for Merkel Disk and Ruffini Ending.

We used two new Tacton sets from a previous study [26], which included sensory and emotional ratings for 24 Tactons in each set. The first set of 24 Tactons was designed using sinusoidal parameters, while the second set consisted of 24 Tactons with complex waveforms. None of the 48 Tactons in the two sets overlapped with the 154 Tactons used for training. We collected acceleration data from 36 new participants for the 48 Tactons, following the same procedure described in Section III-C. We compared the RMSE averaged across the 36 participants for roughness, valence, and arousal ratings between the predictions and the ground truths. We also report the proportion of our predictions that fall within the standard deviation of user ratings across the two test sets. We used individual acceleration data from each of the 36 participants, with each data entry consisting of 48 vibration measurements, to evaluate the model’s performance for variations in user measurements. However, we averaged the roughness, valence, and arousal ratings across participants and used these as the ground truth to ensure the results were generalizable and applicable to real-world haptic design, reflecting aggregate perceptions across diverse users.

The RMSE values averaged across 36 users showed that all three of our proposed methods, regardless of whether mechanoreceptive filters were used in the second stream, outperformed the five baselines across both test sets (Table I). Among all tested models, linear regression yielded the highest RMSE for test set 1 (17.58). The 1D-CNN model [58] showed a reduction in RMSE for test set 1 but had the highest RMSE for test set 2 (20.05) among all tested models. The LSTM model [58] performed similarly to the 1D-CNN model for set 1 but showed improved performance for set 2, with a reduced RMSE of 14.95. The CNN-LSTM network from [59], which used parallel streams of CNN and LSTM, performed better than using either a single stream of CNN or LSTM [58]. However, the transformer regression model [75] did not yield accurate predictions for the roughness and emotions of unseen Tactons, despite being well-trained. Our proposed methods outperformed the CNN-LSTM network across all three tested approaches. Within these approaches, using two mechanoreceptive filters led to a lower RMSE for test set 1 (RMSE = 12.48) compared to using a single spectrogram (i.e., without a mechanoreceptive filter, RMSE = 13.63). The average RMSE for test set 1 was higher when using two mechanoreceptive filters compared to four mechanoreceptive filters (RMSE = 12.12). For test set 2, the RMSE was lowest when using two mechanoreceptive filters (RMSE = 11.35) compared to using a single spectrogram or four-channel spectrograms (RMSE = 11.81). Overall, VibNet with the two-channel mechanoreceptive filters performed the best over both datasets.

On average, 82% of the predictions generated by VibNet fell within the standard deviation of the ground truth user ratings in both test sets across 36 participants. Specifically, in set 1, 18.5, 23.6, and 16.9 out of 24 predictions for roughness, valence, and arousal, respectively, were within the standard deviation. In set 2, 19.4, 23.6, and 16.2 out of 24 predictions for roughness, valence, and arousal, respectively, met this criterion. Overall, across the two test sets, the highest proportion of predictions within the ground truth’s standard deviation was for valence (98% for both sets 1 and 2), followed by roughness (set 1: 77%, set 2: 81%) and arousal (set 1: 70%, set 2: 67%).

We used dual streams in our neural network to capture sequential and temporal-spectral features, drawing inspiration from human vibrotactile perception and the cognitive mechanisms [41, 20]. Our evaluation demonstrated that this architectural choice enabled VibNet to achieve the lowest average RMSE across the test sets. The 1D CNN model [58] performed better than linear regression for test set 1, which consisted of Tactons designed with sinusoidal parameters that varied the spectral aspects of vibration signals. However, this model showed the worst performance for test set 2, which consisted of Tactons with complex waveforms in the time domain. In contrast, the CNN-LSTM model [59], which combined 1D CNN and LSTM in dual streams to predict haptic attributes of real textures, showed lower RMSEs for the test sets compared to using a single stream alone. Our findings further confirm that predicting vibration cognition requires capturing both sequential and temporal-spectral features, in line with the results presented in [20]. Moreover, the results imply that using 2D spectrograms with 2D CNN layers instead of 1D waveforms with 1D CNN layers can enhance the network’s ability to capture the spectral features of vibrations.

In the second stream designed to capture the temporal-spectral features of Tactons, the use of two-channel mechanoreceptive filters — based on the spectral sensitivities of the Meissner Corpuscle (RA1) and Pacinian Corpuscle (RA2) — led to a lower average RMSE of 11.91 across the two test sets compared to two alternative approaches: (1) using a single spectrogram (RMSE = 12.72) and (2) using four-channel mechanoreceptive filters (RMSE = 11.97). Specifically, while using more channels in the spectrogram improved performance for test set 1, the best performance for test set 2 was achieved with VibNet (RMSE = 11.35), followed by both the single spectrogram and the four-channel mechanoreceptive filters (RMSE = 11.81 for both). These results underscore the efficacy of mechanoreceptive filters while also highlighting the need to balance the two streams within the network to maintain generalizability for predicting vibrotactile sensations and emotions. Overall, the predictions generated by VibNet achieved an average generalization accuracy of 82% across Tacton sets and participants, as measured by the proportion of predictions falling within the standard deviations of the ground truths, irrespective of using any individual participant’s acceleration haptic data.

While VibNet demonstrated promising prediction performance and generalizability across different Tacton sets and users, our biomimetic framework has several limitations. Although we trained VibNet on a wide range of Tactons designed using a variety of parameters, the model may be less accurate on Tactons outside the design space used for training. For instance, the training data included Tactons with durations ranging from 300 milliseconds to 6 seconds. If a user inputs a Tacton lasting 20 seconds, VibNet may require adjustments to maintain accuracy. Similarly, the frequency bandwidth of iPhones is in the range of 80 Hz to 230 Hz, with a maximum intensity around 0.3 G. Thus, making reliable predictions for Tactons outside these frequency and amplitude parameters may require additional data collection and further enhancements to the neural network. Lastly, our focus was solely on haptic stimuli, with no auditory or visual stimuli provided. Future research should aim to expand the dataset to cover a broader design space and incorporate multimodal stimuli to further enhance VibNet’s utility.

We outline how our framework can inform future research and influence haptic design practices.

Designers can utilize VibNet to estimate the user’s sensations and emotions when prototyping Tactons. Exploring the sensations and emotions elicited by Tactons is often a time-consuming and resource-intensive process, as it requires conducting multiple user studies while navigating an extensive Tacton design space. However, VibNet requires only acceleration data produced by a haptic device, enabling designers to efficiently prototype and refine Tactons to achieve the desired sensations and emotions. By reducing reliance on trial-and-error prototyping and extensive user testing, VibNet streamlines the development of haptic applications.

Researchers can integrate our model into Tacton design tools. Designing effective Tactons for real-world applications is often a complex and challenging task. To address this, previous studies have proposed graphical user interfaces to assist designers in creating Tactons for diverse applications and scenarios [37, 76]. VibNet can further enhance these design tools by enabling designers to quickly evaluate the sensations and emotions elicited by the created Tactons, leading to a more efficient and streamlined design process.

Our framework can inform the development of future machine learning models to predict various attributes conveyed by haptic stimuli. First, we provide a unique haptic dataset consisting of acceleration data collected from 36 participants, along with roughness, valence, and arousal ratings for 154 Tactons designed using a wide range of design parameters. This dataset serves as a resource for training future prediction models in haptics domain. Additionally, researchers can use this data to explore which components of neural network architectures contribute to capturing specific features of vibrations, providing deeper insights for Tacton design. Second, our framework offers a method for augmenting haptic data offline while ensuring perceptual invariance of the signals. This method provides a foundation for developing deep learning or generative models that require large haptic datasets. Researchers could also explore the effects of haptic data augmentation on model robustness, performance, and training efficiency, as has been done in vision and auditory research [77, 78]. In addition, they could develop online augmentation techniques, such as using Generative Adversarial Networks (GANs) [79], similar to those employed in vision and auditory research [78, 80]. Such advancements could accelerate the development of computational models for haptics by enabling real-time data generation and refinement. Finally, our demonstration of mechanoreceptive processing and dual-stream neural networks offers a foundation for future prediction models of Tactons. Future work can expand these models to predict specific metaphors conveyed through Tactons, such as heartbeat or tapping, with the collection of metaphor ratings [30]. Researchers could also extend VibNet to predict sensations and emotions elicited by vibrotactile grids [81, 82] or by multi-modal feedback systems, including thermal feedback and force feedback. Moreover, future studies could expand the scope of prediction models beyond mechanical vibrations to encompass other stimuli in emerging haptic technologies, such as surface electrovibrations or mid-air ultrasound vibrations. Such models would not only improve in versatility and applicability but also enhance their relevance to a broader range of haptic interfaces and experiences.