A Virtual Reality Game as a Tool to Assess Physiological Correlations of Stress

There are many examples of GWAP, and most of them were designed to collect data from a large player base to build on research or improve programs(4). An example of such a game is the Airport Scanner game, which makes players find airport contraband using an X-ray scanner. This data set was used by researchers to investigate how well baggage screening can be performed. Games have also been created for studying performance in cognitive tasks. An example of this is Luminosity, where players play simple games that train various aspects of cognition. Scientists hope to use this database to gain better insights into how people think and what factors affect cognitive ability. In addition, games like Foldit (5) or Eyewire (6) use players to advance scientific studies, such as folding complex proteins or mapping out the synapses of the brain, respectively. Using these games while adding neural and physiological measurements is expected to introduce a new dimension to user interaction and monitoring experiments (7).

This study modified a popular VR game, Keep Talking and Nobody Explodes (8), to test participants in a controlled environment. Keep Talking Nobody Explodes is a VR game where one player disarms a bomb while communicating with another player, a bomb-defusing expert, who has the bomb defusing instructions. The modules or puzzles for disarming the bomb include certain wires that must be cut, buttons that must be pressed at certain times, and keypads that must be pressed in a certain order. The player will need a VR headset (e.g. Oculus or HTC Vive) and controllers to play the game. Only the player can see the puzzles and bomb in VR, while the bomb-defusing expert can only check a manual to find instructions to solve the puzzles. The player and the bomb-defusing expert must communicate effectively to defuse the bomb.

The ultimate goal of this research is to find methods to reduce stress and improve performance at a certain task. The present study aims to find physiological markers under stressful and harsh conditions relating to potential real-life situations. Finding ways to measure stress and how it affects performance is difficult in the way that it is difficult to control a subject’s stress level. This study develops a VR game framework that can be used to measure and synchronize physiological signals, behavioral data, and the timing of game events. The framework needs to be flexible enough to expand to multiple data modalities and it needs to be able to include tools to assist in solving the tasks. In essence, this game was meant to be a different version of an already existing game meant for entertainment. It was a recreation of an existing game and re-purposed as a game with a scientific purpose so that researchers could use this game as a tool for studying stress in-depth in a controlled environment. Furthermore, this game needs to be enjoyable enough so that its players are willing to return to play the game with different parameters multiple times. The scientists then can acquire sufficient data over different days/times to allow the researchers to study the variability in the stress-performance associations across different individuals and times within an individual.

The game we created originates from a popular social game called Keep Talking and Nobody Explodes, where one player disarms a bomb based on another player’s instructions. The player giving the instructions has a bomb-defusing manual that has instructions to solving each puzzle module, and when all the puzzle modules are solved, the bomb is disarmed. The player who has the instructions, or the bomb-defusing expert, cannot see the bomb or the puzzles, and the player who can see the bomb does not have the instructions. Therefore, the two players must collaborate and communicate effectively to disarm the bomb.

In our version of the game, we implemented the following five types of modules: Wires, Keypad, Simon Says, Button and Venting Gas. Figure 1 shows the image of the bomb with the five types of modules. The timer at the top of the bomb indicates the time before the bomb explodes, and how many mistakes the player has made (marked as an x per mistake, which is not shown in the figure). The timer was implemented using the Unity time.time function. For the Wire module (shown in the top left, top right, and two in the middle right), the player is required to cut a certain wire based on the color of the wires and how many there are. For the Keypad module (shown at the bottom right), the player must press the keypad in a certain order depending on the symbols displayed on the keypad. For the Simon Says module (shown in the top and bottom middle), the player must press the buttons in a certain order based on which button is blinking. For the Button module (shown in the leftmost middle row), the player must press the button at a certain time to solve it. The Venting Gas module, in the bottom left, is a “needy module” that captures the player’s attention and causes more stress.

These modules are solved based on our version of the bomb-defusing manual, part of which is shown in Figure 2. When playing this game, one player with the VR headset can see the bomb, as shown in Figure 1, while the other player reads this manual to instruct the player who is disarming the bomb.

In our initial implementation, there were three difficulty levels: easy, with four modules to solve in five minutes, medium, with six modules to solve in two minutes, and hard, with nine modules to solve in one and a half minutes, as shown in Table 1. Because this study focuses on the physiological correlates (e.g. HR and HRV) of stress levels (easy vs hard), we simplified the difficulty levels of our GWAP so that the players were instructed to solve a random puzzle in two minutes (easy) or twenty seconds (hard) to simulate non-stressful and stressful conditions, respectively. We describe this experiment and its results in more detail in the following sections.

We used Unity (11), a popular game engine tool because it uses a well-known programming language and has a simple UI. We also used Blender (12), an open-source rendering tool, because of its popularity and vast amounts of resources online.

We made 3D models such as the bomb itself in blender and imported them into Unity. Afterward, the textures were applied to the game objects in Unity. After applying these textures found from the Unity store and online, we started to create the codes in the game. We first implemented the five types of modules (Wire, Button, Simon Says, Keypad, and Venting Gas, which are described in the previous section) in C#.

To convert our game into VR, we used VR Toolkit (13), a VR library compatible with Unity. Some of the functions (such as grabbing an object and moving it) were already included in the VR library, but we had to modify others so that our game would work with those VR functions that allowed the player to use the game objects.

Lab Streaming Layer (LSL) (10) is a scientific tool used to synchronize streaming data for live analysis or recording. We used LSL Markers to indicate when certain events triggered, like when the player pressed a specific button or cut a specific wire. LSL is a cost-efficient data acquisition and synchronization framework for distributed systems with a focus on reliability, near-real-time access, and time-synchronization. We used the Unity LSL library to mark and broadcast certain events and output timestamps with those events. In our GWAP game, the event markers included when the player started each session (resting, hard, easy sessions), when the player pressed a button or cut a wire, when the player solved a module, and when each session ended.

Figure 3 shows a sample code used to output an event marker when the button module is pressed. Lines 73 to 79 define the function StartUsing, which is called when the button is pressed in VR. Line 76 is where the event marker itself is sent out. Lines 82 to 91 define the function StopUsing, which is called when the button is released and checks whether the player has solved the module or not. The rest of the codes can be viewed through the GitHub link (14).

This section presents how physiological data can be used in conjunction with this game. To measure the electrical signal of the heart, we used a wearable ECG device called Heartypatch and its recording software(15). Figure 5 shows a sample ECG recording obtained from the device and its software.

The trials we recorded were split into two-minute or twenty-second sessions, with a one-minute break in between sessions. Each session required the player to solve a random puzzle module, from cutting wires to pressing buttons. The two-minute sessions were the easy sessions with little stress, while the twenty-second sessions were meant to be stressful (hard) sessions that tested the player’s ability to solve puzzles under a very short time limit. Figure 4 shows ECG data synchronized with event markers from the game, revealing the various levels of ECG signal depending on how stressful the player was at key points in the game.

Heart rate is the number of heartbeats per minute. Heart rate variability is the fluctuation in the time intervals between adjacent heartbeats (16). To calculate HRV, we first find the time intervals between the consecutive heartbeats using the findpeaks function in MATLAB. Figure 5 shows heartbeats and time intervals between consecutive heartbeats of a sample ECG recording. We then compute the differences between the adjacent time intervals and then square them. Finally, we add all those squared differences, divide it by the total number of the squared differences to find the mean, and then take the square root of the mean to find the HRV. This process is succinctly shown in the following equations:

where the $t_{i}$ is the $i$-th time interval (the difference in time in milliseconds) between two consecutive heartbeats and $n$ is the total number of the time intervals.

The goal of this pilot study was to explore how variability in time stress relates to variability in physiological responses (e.g. heart rate and HRV) of healthy individuals. To this end, we recorded the event markers, the players’ ECG, and their behavioral data under three conditions: stressful (hard), non-stressful (easy), and resting conditions. The stressful (hard) condition required the player to solve a puzzle module in twenty seconds, while the non-stressful (easy) condition allowed the player to solve a puzzle module in two minutes and the resting condition allowed the player to take a break for one minute between two sessions. In each experiment, the participants played four hard sessions, four easy sessions, and eight resting sessions. The easy and hard sessions were played in random sequence in the experiment. Two healthy individuals participated in this pilot study, resulting in the ECG signal data for a total of eight hard, eight easy, and 16 resting sessions.

After collecting the data through LSL from our experiment, we split the data using the event markers from the game. We identified which sections of the ECG data correspond to easy, hard, or resting sessions. We then analyzed the data from the separated and categorized (easy, hard, or resting) sessions to find the average heart rate and HRV values for the easy, hard, or resting conditions using MATLAB as shown in Figure 6.

Figure 7 shows the average heart-rate values under the three (resting, hard, and easy) conditions. The hard (stressful) sessions have the highest average heart rate of 96 bpm. The resting and easy (non-stressful) sessions have average heart rates of 84 and 82 bpm, respectively. The difference between the average heart rates of easy and resting sessions is small, but the error bar for the easy sessions is larger, indicating that the variation of heart rates in the easy sessions is larger.

Figure 8 shows the average HRV in milliseconds, along with error bars, under the three (resting, hard, and easy) conditions. As shown in the bar graph, the average HRV value under stressful (hard) condition is around 30 milliseconds while those under the non-stressful (easy) and resting conditions are around 40 milliseconds and 50 milliseconds, respectively. Thus the difference between the average HRV values of the resting and easy conditions is around ten milliseconds, and the difference between those under the easy and hard conditions is also ten milliseconds, which makes the difference between the average HRV values under hard and resting conditions to be about 20 milliseconds. We also note that the average HRV value under stressful (hard) condition is the lowest.

We used statistical tests to examine if the differences in HRV between conditions are statistically significant. To calculate statistical significance, we determined the null hypothesis and an alternate hypothesis. Since the level of statistical significance is often expressed as a $p$-value between zero and one, we calculated $p$-value to show that our results are statistically significant. Typically, a $p$-value less than 0.05 is considered statistically significant. We thus used a one-tailed hypothesis with a significance level of 0.05.

To calculate the $p$-values, we first calculated the standard deviation for both sets of data under the statistical comparison. Then, we calculated the $t$-score and the degrees of freedom using the number of samples in each data set (17). Finally, we used a $p$-value calculator to find the $p$-value using the $t$-score and the degrees of freedom (18).

For example, to find the $p$-value between hard (group 1) and easy (group 2) sessions, we used the following steps:

First, we determined the null hypothesis and the alternative hypothesis as follows: The null hypothesis: "There is no significant difference in the data sets (HRV values) of group 1 (hard sessions) and group 2 (easy sessions)." The alternative hypothesis: "There is a significant difference in the data sets (HRV values) of group 1 (hard sessions) and group 2 (easy sessions)."

Second, we calculated the standard deviation $\sigma$ of the HRV values (data sets) for the hard sessions and easy sessions using the following formula:

where $x_{i}$ is an individual data value in the given group (hard or easy), $\mu$ is the mean of the data for the given group (hard or easy), and $N$ is the total sample size of the given group (hard or easy).

Third, we calculated the standard error $s$ between group 1 (hard) and group 2 (easy) sessions using the following formula:

where $\sigma_{1}$ is the standard deviation for the data in group 1 (hard), $N_{1}$ is the sample size of group 1 (8 for hard), $\sigma_{2}$ is the standard deviation for the data in group 2 (easy), and $N_{2}$ is the sample size of group 2 (8 for easy).

Fourth, we calculated the $t$-score using the following formula:

where $\mu_{1}$ is the mean of the data for group 1 (hard), $\mu_{2}$ is the mean of the data for group 2 (easy), and $s$ is the standard error between group 1 and group 2.

Next, we calculated the degrees of freedom $d$ using the following formula:

where $N_{1}$ is the sample size of group 1 (8 for hard), and $N_{2}$ is the sample size of group 2 (8 for easy).

Finally, we used the $p$-value calculator (18) to find the $p$-value based on the $t$-score and the degrees of freedom $d$.

As shown in Figure 8, the $p$-value for the null hypothesis that there is no significant difference between the average HRV values in hard and easy sessions is $p=0.00244$, which is less than the significance level of 0.05.

This means that the difference between these values is statistically significant and unlikely to happen by chance. The HRV differences between resting and hard groups and between resting and easy groups are also statistically significant since their $p$-value is much less than 0.05 ($p<0.00001$).

Hence, our experimental results showed that the average HRV value is the lowest under the stressful conditions with statistical significance.

We compared our findings of the lower HRV values under stress to prior work such as (19), which analyzed the influences on HRV Values in athletes, and (9), which analyzed the relationship between HRV and regional cerebral blood flow. Both studies showed the stress in athletes and higher blood flow was correlated with lower HRV values.

In the study (20), researchers evaluated the effect of real-life stress on the cardiac response of the subjects. They concluded that stress increases arterial pressure and impairs cardiovascular homeostasis.

Furthermore, in another study (21), participants performed tasks that either had a physical, mental, or combined load, while their HRV was measured. The study concluded that HRV is affected by changes in physical or mental states, and they were also able to differentiate between resting, physical and mental conditions through the characteristics of HRV.

Researchers surveyed London-based civil servants (aged from 35 to 55) (22) to evaluate the stress levels due to their work. They reported correlations between high work stress, low physical activity, poor diet, and most importantly lower heart rate variability.

To showcase our concept, we decided to use a readily available speech recognition system in the Unity library called DictionRecognizer. This tool in Unity listens to speech input and returns a string of words it recognizes in the speech input.

To provide just the precise information to the player, we used a dictionary of keywords that are used for solving the puzzle to check if the recognized spoken words are part of the keywords. This game-specific dictionary filters out unimportant sentences and words to display only keywords such as “wire”, “bomb” or names of colors, so that the player can focus on the important instructions only. We came up with 20 keywords that were important to solve the puzzles and stored them in an arraylist. Each recognized word is then compared against the words that are in the arraylist to determine if they should be shown in the headset display. Figure 9 shows the sequence of this speech-to-text program, the keyword filtering, and how the words appear in the display.

For time-sensitive tasks such as the tasks in this game, we believe that the use of the speech recognition system with a keyword filter can significantly facilitate the communication between the player who disarms the bomb and the player (the bomb-defusing expert) who gives the instruction.

To demonstrate how the speech recognition tool can be used in conjunction with the heart-rate measurements, we developed the above-described speech recognition tool as a feature in the game that can be enabled or disabled while playing the game.

We recorded the player’s ECG signals while the player tried to solve the puzzles and disarmed the bomb and at the same time, we also recorded the audio signal of the other player who gives the instructions. Afterward, we overlaid the event markers with both the audio and ECG streams, and synchronized them on the same $x$ axis as shown in Figure 10.

This illustration is intuitive and we expect this tool to improve performance during the gameplay and also believe that it can result in reduced stress which can be measured by the HRV values. Further experiments and studies with multiple subjects are required to quantify our measurements and verify this conjecture.