An EEG-based Experiment on VR Sickness and Postural Instability While Walking in Virtual Environments

• •

  To the best of our knowledge, this paper presents the first EEG-based study investigating VR sickness in a mobile VR setup, i.e., the user navigate a virtual environment with natural walking while wearing a VR headset.

• •

  The experiment recorded both EEG and full-body motion capture data, our preliminary results can contribute new evidence to the ongoing debate about the validity of postural instability theory and inspire future work in this area.

• •

  The related work section presents a detailed literature review of EEG-based experiments for VR sickness. The review summarises the experiment protocols and apparatus used in previous works and the observed brain dynamic changes associated with VR sickness.

Recent advances in BCI hardware and signal processing algorithms have enabled novel BCI applications and paradigms [1]. Passive BCI [91], which continuously monitors the users’ cognitive states, has been proven particularly valuable in many human-computer interaction scenarios. For example, Peck et al. [61] used BCI to evaluate visual interfaces. Frey et al. [20] proposed an EEG-based framework that estimated the user’s mental workload and attention from EEG signals to evaluate user experience. Yuksel et al. [90] developed a BCI-based learning system that adaptively provides music learning tasks with different difficulty levels to enhance the user’s learning efficiency.

There is a growing body of research on monitoring motion sickness and VR sickness from the VR community using the EEG-based passive BCI approach. (Table LABEL:tab:EEG). One of the major benefits of using passive BCI for monitoring VR sickness is its capability to measure the sickness level continuously. Most VR works use questionnaires, such as Simulator Sickness Questionnaire (SSQ) [33], Virtual Reality Sickness Questionnaire (VRSQ) [34], or Cybersickness Questionnaire (CSQ) [81], to measure the VR sickness symptoms during VR uses. Despite its many advantages [24] , the natural constraint of questionnaires is that they can only be conducted when the user is not actively performing the task. Some previous works used an additional joystick [9, 11, 39] or a physical dial [51] to allow frequent reports of the level of sickness. However, these additional input modalities still introduce different levels of distraction. In contrast, the passive BCI approach provides continuous measurements and does not incur additional workload or distract users from their current tasks.

Table LABEL:tab:EEG lists research works that investigated VR sickness using the passive BCI approach. Because these works used a wide range of experiment protocols to induce VR sickness, we sorted this list of works accordingly to their experiment designs and apparatus. The most common protocol for inducing VR sickness is to expose the participants to video content with strong motions, such as virtual navigation [52, 36, 59, 53, 43] or virtual flythrough [44, 31, 39, 56, 88]. The virtual navigation videos mostly involved the urban virtual environment with streets, roads, and sometimes obstacles. The virtual flythrough videos show a wider range of content, from flying to spacewalking, which exhibit stronger roll and pitch motions. The participants were mostly instructed to stay in a stationary sitting posture and passively watched the video content throughout the experiment session. Keeping the participant stationary is common in EEG-based experiments because it will reduce the potential noise in the EEG signals due to body movements. The participants’ passive watching of identical video contents also ensures they perceive similar visual stimuli, allowing more precise time-locked EEG analysis if required.

Several works extend the 2D video setup by incorporating a motion platform to provide visual and vestibular stimuli. A series of works [45, 9, 46] used an experimental setup that included a CAVE and a vehicle mounted on a 6-degree-of-freedom motion platform. During the experiment, the participants behaved as a passenger in the vehicle as the vehicle automatically drove through a winding tunnel. The same group later updated the protocol to include an additional driver actively driving the vehicle through the tunnel [27]. Recent work from Recenti et al. [66] simulates seasickness with a virtual environment of a small boat floating on the sea. The participants wore an HMD while standing on a motion platform that generated synchronized vibrations to the virtual wave.

As shown in Table LABEL:tab:EEG, most previous works reported the fluctuations of low-frequency band power when the participant experienced VR sickness. However, some changes in the power spectrum and the locations of brain activities are contradictory among these works. For example, among works that induced VR sickness with videos on a 2D display, some works [36, 44, 31] reported a low-frequency power increase while others [52, 59] reported a power decrease. Considering the difference in experiment protocols, hardware (displays and EEG headsets), and EEG signal processing protocols, such inconsistency in frequency band power is unfortunately not uncommon in BCI research. As a result, recent work [88] explores the reactive BCI paradigm [91], which uses more reliable brain dynamic features (e.g., event-related potentials). However, such brain features require strict time-locked events, which significantly limit the design of VR scenarios. There are also works [43, 46] focusing on using a machine learning approach to train classifiers that can detect or predict the onset of the VR sickness while putting less emphasis on the changes of spectrum powers and the underlying brain dynamics.

Complementing these previous works, our work investigates EEG changes of VR sickness while the participant actively navigates the virtual environment with natural walking. As most modern VR headsets allow users to move freely in the physical space, we believe our work represents a meaningful contribution to the understanding of VR sickness. Note that the mobile VR setup inevitably introduces additional noise into the EEG signals [49] and requires additional post-processing of the EEG signals. We describe our EEG data processing pipeline in detail in Section 4.2.

We used an experimental protocol [84] inspired by the research works on redirected walking, originally developed for users to explore large virtual environments inside a limited physical space [80, 82, 55]. These works carefully control the amplification below the user’s detection threshold [79, 21, 40]. In contrast, we intentionally increase the translation gain so that the users can perceive the disparity. The protocol hypothesizes that the disparity of movements between the virtual and physical worlds would require the user to adjust their gait strategies to maintain postural stability. As the disparity gradually increases in the experiment, the users would need to consistently change their posture to fight the postural instability. According to different studies on postural instability [77, 4] , as the users remain unstable, the level of VR sickness would also increase.

There is a significant amount of works and evidences both for and against the postural instability theory, and we would like to direct our readers to a recent review [58]. Here we would like to focus the review on works that use VR and BCI to measure and understand human balance control. The main applications of these research works are in the domain of fall prevention for elderly adults or performance enhancement in rehabilitation settings [87].

The seminal works from Slobounov et al. investigate brain functions and behaviour outcomes using experimental methods such as self-initiated postural sways [74, 73] and postural instability induced by rotating the virtual environment [76, 75]. These works agreed on theta power increases in the frontal areas during postural instability. Chang et al. [6] induced postural instability with a virtual bus scenario on a motion platform to compare the brain activity of elderly adults with low and high fall-risk potential. The result suggested a strong correlation between postural-related cortical regions for the low fall-risk group. Peterson et al. [62] found that transient visual perturbations, i.e., a short roll of the virtual environment while the user is walking on a treadmill, could boost short-term motor learning, supported by the increased theta power and decreased alpha power in parietal and occipital regions and better balance performance. Many previous works also correlate the EEG signals with biomechanical measurements, such as the centre of pressure with force plate [76] or gait kinematics with motion tracking [62].

Building on the methodologies of these previous works, we contribute the knowledge of the brain dynamic related to posture and gait when the user experiences VR sickness in a mobile VR setup.

Twenty-one healthy adults (17 males and four females) participated in the experiment. The mean age was $25.73$, with a standard deviation of $3.6$. All participants gave written informed consent and were compensated for their participation. All participants had a normal or corrected-to-normal vision. Among all participants, 13 had experience with 3D computer games, nine had experience in VR, and 15 had previous experience with motion sickness.

The physical space of the experimental environment was 3 meters by 5 meters (Figure 1). In each trial, the participant walked from point A to point B and back to point A. Points A were marked with black tape on the ground to ensure the participant stood at the proper initial position. The distance between A and B was 5 m.

The virtual size of the entire virtual scene was 100 m by 100 m. The virtual walking took place on a straight street 70 meters long and 5 meters wide. Figure 2 shows the side view of the road, and Figure 1 shows the participant’s first-person perspective while performing the walking task. We marked both ends with a virtual utility hole and a virtual red arrow.

The virtual environment was developed using the Unity 3D game engine, version 2019. The virtual environment images were transmitted to the VR HMD via cable.

The user’s full body motion was captured with the OptiTrack system with 12 Flex 13 cameras at the capture rate of 120 frames per second. We used the Baseline + Toe marker skeleton template with 41 tracked markers in the Motive:Body software for all participants. The HMD used in the experiment was the Oculus VR CV1, which has a 110-degree field-of-view resolution of 1080 x 1200 pixels per eye. The cable of the VR headset hung on a ceiling hook to avoid affecting the participants’ movements. The OptiTrack Unity Plugin, which only supported Oculus VR CV1 at the time of the experiment, was used to synchronize the data between the motion caption system and the virtual environment.

The EEG data were recorded from 64 Ag/AgCI slim active sensors positioned according to the extended 10-20 electrode placement system. A portable and wireless LiveAmp system (Brain Products GmBH, Munich, Germany) was used to minimize the interference of natural walking in the virtual environment. The wireless LiveAmp receiver was on a waist bag, as seen in Figure 1. The EEG data were acquired at a sampling rate of 500 Hz. The channel locations were assigned based on standard location, distributed by the EEGLAB toolbox [13, 14]. Contact impedance was maintained below $5k\Omega$ for the EEG recording.

The experiment induces VR sickness by gradually increasing the level of translation gain. The experiment starts from TG level $1x$, where the user’s movements in the real world are directly mapped to the virtual environment without amplification. In each trial, the participant walks from the utility hole to the mid-point indicated by a red arrow, then back to the utility hole in the virtual city (Figure 1). The TG level increases every 5-trials (a $block$), in the order of $1x,2x,4x,6x,8x,10x$. The experiment consists of six blocks, each corresponding to one TG level, and there are 30 trials for each participant. The participant walks the same physical distance in the real world in each trial, regardless of the TG level. The TG level augments the virtual walking distance and thus the different mid-point locations, indicated by the red arrows (Figure 1).

An informal pilot test led to our decision of using $10x$ gain as the upper bound of the TG level. The VR engineer in our team, who has extensive experience in different redirected walking tests, considered TG levels larger than $10x$ impractical and too difficult for the participant to complete even one trial. Our implementation applied motion magnification to all axes, which is different from the traditional implementation [29], which amplifies the motion in the forward direction only. The decision was intended to ensure that participants were under constant motion magnification and postural instability throughout the experiment.

Participants were split into the VRS and NoVRS groups based on the SSQ after the experiment. The VRS group consists of participants who experienced severe VR sickness symptoms while interacting with the virtual environment. The NoVRS group consists of the participants that experience minor or null VR sickness symptoms during the experiment. Each group had two female participants.

Note that our experimental design does not randomize the order of TG levels as we do not intend to investigate the relationship between the TG level and the level of VR sickness. Instead, similar to previous EEG experiments that used multi-block designs [52, 59, 56], we compare the biomechanical and EEG data between VRS and NoVRS groups in each block to understand the effect of VR sickness at different experimental stages. We want our outcomes to be the preliminary results of one of the first experiments to address VR sickness on a mobile VR setup.

The supplemental video shows two trials with the TG levels of $1x$ and $2x$. Please note that our participants did not agree to be recorded during the experiment. Thus the video is demonstrated by a researcher.

Figure 3 shows the complete experiment procedure. The session started with a pre-experiment questionnaire (PEQ). The PEQ inquired about the participants’ familiarity with 3D and VR technologies and their subjective assessment of their susceptibility to different types of motion sickness. After completing the PEQ, the researcher helped the participant put on the motion capture suit and ensured the participant’s full-body motion within the experiment area could be successfully captured and displayed in the software. After completing the motion capture setup, the researcher proceeded with the EEG headset setup. The researcher injected conductive gels into each of the 64 electrodes and ensured its impedance shown on the EEG capture software was below $5k\Omega$. The EEG signals of the closed-eye resting and eyes blinking were also visually inspected after the setup.

After the EEG and motion capture systems were ready, the participant performed the baseline walking trials (without wearing the VR headset). The participant was instructed to walk between the two ends of the walking area, i.e., from position A to B, then back to A in Figure  1. After the baseline walking, the researcher helped the participant put on the VR headset and adjusted the interpupillary distance. Wearing a VR headset would inevitably cause physical contact with the electrodes. Thus, re-calibration and re-gel were usually required to correct the position and impedance of the EEG electrodes. The setup was completed at this point, and the setup process took around 40 to 50 minutes on average.

As stated in the experiment design section above, the main experiment consists of 6 blocks, each with 5 trials of one TG level. The TG level increases from $1x$ in the first block to $10x$ in the last block. In each trial, the participant walks toward a red arrow and back to the utility hole (Figure 2) in the virtual city. Once the participant reaches the utility hole, a virtual message notifies the completion of the trial. It then prompts the participant to report their level of sickness from 1 to 10, where ten corresponds to severe VR sickness symptoms.

Besides the PEQ, this experiment uses two types of questionnaires. The first questionnaire is the Between-Trial Questionnaire. At the end of each trial, each participant reports, on a scale from 1 to 10, their feelings of dizziness, discomfort, nausea, fatigue, headache, and eyestrain. We chose these symptoms following previous studies [46, 86], which also used a sub-set of symptoms in trials to shorten the report time.

The second questionnaire is the Post-Experiment Questionnaire. Upon completing the VR session, we asked each participant for a full SSQ, followed by a semi-structured post-hoc interview session where the researchers engaged with the participant and recorded their subjective experience during the experiment.

The following subsections describe our methodologies to process and analyze the EEG data of our experiment. Mobile VR with EEG experiments have well-established procedures to ensure that EEG data gets the highest signal-to-noise ratio. To ensure this, the following sub-sections briefly describe the same procedures as Do et al. [15]. The extensive work of these authors describes the methodologies of mobile experiments with VR and EEG.

Our work also follows the procedures from studies that describe how to remove gait-related artifacts [17, 47]. These procedures help minimize gait-related activity and do not affect any of the conclusions of this study. All the EEG data analysis steps were performed in EEGLAB [13, 14] version 14.1.2 in Matlab (Mathworks Inc., Natick, Massachusetts, USA). Figure 7 shows a summary of the EEG processing steps.

Figure 5 shows the comparisons of biomechanical measurements between the two groups. Statistical differences were found on CoM in blocks 2 and 6 (see Table 1, last two rows). At both blocks, the VRS group showed a greater CoM displacement than the NoVRS group. No statistically significant differences were found in any of the other biomechanical measures.

Figures 6, 8, 9, and 10 show the group ERSP results at the Fz, Cz, Pz and T8 channels respectively. Each figure contains the following information: the top rows show the ERSP of the VRS group. The middle rows show the ERSP of the NoVRS group. Finally, the bottom rows show the statistical difference between the two groups after the permutation test.

In the rows showing ERSP labelled with VRS and NoVRS, the red color represents power increases, while blue represents power decreases. In the last row, the red color denotes that the VRS group has a statistically larger power than the NoVRS. The blue color denotes that the VRS group has statistically lower power. A summary of the increases and decreases at each frequency appears in Table LABEL:tab:EEG, at the bottom row.

Figure 6 shows how the VRS group had a significantly lower Fz power at the delta frequencies at blocks 1 through 4. The VRS group had a significantly higher theta power at experiment blocks 1, 2 and 5. The alpha and beta frequencies were lower at experiment blocks 2 and 5, and the gamma frequencies were lower at experiment blocks 2, 3, 4, and 5.

Figure 8 shows a significant decrease in Cz delta power through the experiment. A decrease in theta power in blocks 2, 3, 4, 5, and 6. A low-alpha power decrease from the VRS group at experiment blocks 2, 3, 4, 5, and 6. A beta decrease at experiment blocks 5 and 6. And finally a significant decrease in the gamma power at experiment block 5.

Figure 9 shows a significant decrease in Pz delta power from the VRS group at experiment blocks 1 and 3. The VRS group also had significant theta decreases in experiment blocks 1, 3 and 6. Finally, there as a significant beta decrease in experiment block 2.

Finally, Figure 10 shows a strong decrease in T8 delta frequency from the VRS group throughout the experiment. There is also a significant decrease from the VRS group at the theta frequencies at experiment blocks 1 through 3. There is an alpha frequency decrease at experiment blocks 2 and 3. A beta frequency decrease at experiment block 2, and a decrease at the gamma frequency in experiment blocks 2 and 5.

No biomechanical measurements presented statistical differences but the difference in the Center of Mass. CoM showed differences within blocks 2 and 6. Block 2 has a corresponding TG level of $2x$ and was the first block when the participant experienced a perceivable disparity in body motion between real and virtual worlds. The result suggests that the VRS group initially had trouble adapting to the new perceivable TG level. Indeed, during the post-hoc interview, multiple users in the VRS group reported being surprised by the change in the TG level and commented that more time was needed to adapt to the sudden change in the virtual environment.

The statistically significant differences in the CoM appeared at Block 2, one experimental block before we found statistical differences in the BTQ. These findings confirm the primary assumption of the postural instability theory, which states that postural differences will appear before the onset of subjective symptoms [67]. Our EEG results in the central-theta frequency corroborate these findings. Different studies with VR HMD have reported these changes in participants that struggle to adjust posture or to maintain upright balance [72, 28, 16]. Given the relationship between the changes in posture and this signal, observation of the central-theta frequency can be used as an alternative method to identify if the participant is suffering from postural changes that lead to VR sickness.

Our EEG preliminary results yielded more important insights into the participants’ cognitive performance. The EEG result suggests that VRS group did maintain postural control but at the cost of a higher mental effort. The VRS group has a significantly lower alpha power through blocks 2, 3, and 4. It has been well-established that alpha power (in the parietal brain region) was inversely correlated with mental workload [37, 83]. The suppression of alpha power was also reported when the participant consciously adjusted their posture while standing [73, 87] and while walking [5]. Interestingly, when walking with a prosthesis, a similar reduction in alpha and gamma bands have been reported when the cognitive-motor task demands increased [71, 64]. In addition, the increased theta power in the frontal area of the VRS group also concurs with a series of works from Slobounov et al. [74, 73, 76, 75] which found the theta power increased when the participant experienced postural sways. In sum, the lower alpha power (in frontal and parietal) and higher theta power (in frontal) of the VRS group seem to suggest that although VRS group could complete the tasks without significant changes in postural stability or gait parameters, they still found the task mentally challenging, which took more effort to complete them.

For the NoVRS group, their higher delta power also supports the argument that postural instability theory plays an important role in the mobile VR setup. The increased delta and theta powers were found when the participant was capable of engaging in cognitive-motor tasks, such as visual information processing and postural adjustment [22, 38]. Presacco et al. [63] also observed an increased delta power when the participants engaged in precision walking tasks. These findings suggest that the NoVRS group exhibited a better cognitive capability to continuously update their postural control strategy when compared to the VRS group.

In sum, despite both VRS and NoVRS groups having similar biomechanical measurements, the EEG data provides additional insights into the cognitive performance differences. The brain activities of VRS group exhibited similar patterns that were previously linked to higher cognitive workload and postural instability. In contrast, the NoVRS group showed brain activities that were expected when the participant is capable of performing complex motor tasks such as postural balance control and fine walking movements.

Table LABEL:tab:EEG shows a list of works that used EEG to measure and understand VR sickness. Our experiment used a novel experimental protocol in a mobile VR setup and resulted in some contradictory EEG results from some previous works. As discussed in the previous section, the difference might be attributed to the cognitive activities related to postural control. In contrast, the EEG difference in previous works might represent the brain activities related to the sensory conflict. Nevertheless, most works, including ours, seem to agree that the fluctuation of low-frequency power along the frontal and parietal midline correlates with VR sickness.

This finding suggests that the ideal locations for an EEG add-on for VR sickness detection on a VR headset might be at the frontal and parietal midline region. Because the changes in the low frequency seem to be a common phenomenon in most VR sickness experiments, it might be possible to train a single classifier that can be generalised to different virtual environments and VR setups (stationary or walking).

The result also suggests that postural instability theory is worth further investigation as most consumer VR headsets allow natural walking in the virtual environment. In addition, the biomechanical measurements alone might not be able to capture the full picture of the postural instability theory. Additional measurements such as EEG or other biosignals might be necessary for the future investigation alone this direction.