AI-enabled Prediction of eSports Player Performance Using the Data
from Heterogeneous Sensors

Indeed, there is a lack of prior research utilizing sensors data to predict eSports player behavior. It happens due to young research domain in eSports. Recent research on using sensor data in eSports is limited to predicting the overall player skill or finding simple dependencies in the data.

In [6] authors investigate the correlations between psychophysiological arousal (heart rate, electrodermal activity) and self-reported player experience in a first-person shooter game. Similar research about the relation between player stress and game experience has been investigated in the MOBA genre [7]. The connection between the gaze and player skill is investigated in [8]. Mouse and keyboard data is a natural source of information about the players. Its relation with player performance in first-person shooters is covered in [9] for Red Eclipse and [10] for Counter-Strike: Global Offensive (CS:GO). Player performance can also be predicted by activity on a chair during the game [11] or in reaction to key game events [12].

However, there is extensive research work carried out on data collection and activity recognition in other applications including sports, medicine, and daily activity monitoring. In sports, wearable sensing systems are used for detection and classification of training exercises for goalkeepers [13], assessing header skills in soccer [14]. Also, wearable systems were designed to classify tricks in skateboarding [15], classify popular swimming styles using sensors [16], and other activities in sports [17]. In terms of daily activity monitoring and medical applications, they have been studied for nearly three decades with the use of wearable sensors. Many medical studies deal with the investigation of human gait, for example, for patients with the Parkinson’s disease [18].

Most of the current research in eSports analytics relies exclusively on the in-game data collection and further analysis. It has been shown that information about kills, deaths, and other game events can help predict a game outcome in Multiplayer Online Battle Arena (MOBA) discipline for Dota 2  [19], League of Legends [20], and Rocket League [21]. Another opportunity to predict the game outcome in the MOBA related disciplines is based on the features extracted from players’ match history, as well as in-game statistics [22]. Players match history can also be used to create a rating system for predicting the matches outcome in the First-Person Shooter (FPS) genre [23].

As noticed earlier, the in-game data in eSports is widely used for analytic studies in the area. Drachen et al. consider clustering a player behavior to learn the optimal team compositions for League of Legends discipline to develop a set of descriptive play style groupings  [24]. Research by Gao et al. [25] targets the identification of the heroes that players are controlling and the role they take. The authors have used classical machine learning algorithms trained on game data to predict a hero ID and one of three roles and achieve the accuracy ranging from 73% to 89% which depends on features and targets used. Eggert et al. [26] has continued the work by Gao et al. [25] and applied the supervised machine learning to classify the behavior of DOTA players in terms of hero roles and playstyles. Martens et al. [27] have proposed to predict a winning team analyzing the toxicity of in-game chat. In [28] authors used pre-match features to predict the outcome and analyze blowout matches (when one team outscores another by a large margin). The research reported in [29] describes the cluster evaluation, description, and interpretation for player profiles in Minecraft. The authors state that automated clustering methods based on game interaction features help identify the real player communities in Minecraft.

In many domains the skills and performance can be assessed and/or predicted based on sensors data [30]. In sport, the data from the Inertial Measurement Unit (IMU) can be helpful for estimating volleyball athlete skill [31]. Tennis player performance can be assessed from the IMU data on the hand and chest [32], or on the waist, leg, and hand [33]. Similar techniques have been investigated for skill estimation in soccer [13], climbing [34], golf [35], gym exercising [36], and alpine skiing [37].

Another popular domain for skill assessment based on the sensor data is surgery. In [38] authors use IMU data to create a skill quantification framework for surgeons. Ershad et al. [39] have shown the connection between the surgeon skill and behavior information collected from IMU. Ahmidi et al. [40] developed a system using motion and eye-tracking data for surgical task and skill prediction. The authors have used hidden Markov models [41] to represent surgeon state which is similar to the method proposed in this work in Section III. The connection between the surgeon actions and pressure sensors data has been investigated in [42].

Physiological data have also been used for predicting skill level and skill acquisition in working activities, such as mold polishing [43] and clay kneading [44], as well as dancing [45].

In this research, we use a number of wearable and local unobtrusive sensors used for data collection during the game session and further data analysis. It was carried out with respect to the players needs.

In our work, we use three groups of sensors: physiological sensors, sensors integrated into a game chair, and environmental sensors. The sensor network architecture is shown in Figure 2. The list of sensors used, their locations, and sampling rates are presented in Table I. Further issues associated with sampling rates are discussed in Section III-D1.

Physiological data recorded:

• •

  Electromyography (EMG) data as an indicator of muscle activity. EMG data are related to physiological tension affecting player’s current state [53].

• •

  Heart rate data received by a heart rate monitor on a chest. High values correspond to mental stress and arousal [54], which might affect the rationality of player decisions.

• •

  Electrodermal activity (GSR) or skin resistance data as a measure of person arousal [55]. This value is also connected with the stress level.

• •

  Eye tracker data of player gaze position on a monitor in pixel values. The player must check the minimap and other indicators on the screen to have relevant information about the game and, thus, make effective decisions [8].

• •

  Mouse movements captured by a custom python script as a measure of the intensity of a player input. This data is an indirect indicator of the hand movement activity as well as the player skill [9].

The sensors integrated into a game chair are presented by a 3-axial accelerometer and a 3-axial gyroscope. We illustrate axes orientation for the chair in Figure 3. Recorded data includes:

• •

  Linear acceleration of a chair. It captures the player movements to the game table, parallel to it, chair height changes, and small oscillations possible in stress conditions. Behavior on a chair is connected with the player skill [12, 11].

• •

  Angular velocity of a chair. This data provides the information about the person’s wiggling and spinnings on a chair.

Environmental data recorded:

• •

  $CO_{2}$ level. High $CO_{2}$ level results in the reduction of cognitive abilities [56], thus directly affect the gaming process.

• •

  Relative humidity. High level of relative humidity results in the reduction of neurobehavioral performance [57]

• •

  Environmental temperature. Too warm conditions may affect the human performance [58].

Apart from a number of heterogeneous sensors, the sensor network has a dedicated storage server (based on Intel NUC PC), gamer PC (a high-speed Intel I7 PC with the DDR4 memory, and an advanced GPU card). For synchronization reasons, we have an NTP server with GPS/PPS support. A high-speed wireless router connects all the devices in the network. PC with strict requirements to ping value (gamer PC, NTP server) has a wired connection to the router (LAN). The sensors have a wireless connection to the network as they are placed near the eSports athletes (WLAN). The router has a low latency connection to the Internet (WAN). Proper synchronization of the sensors and gamer PC is essential for further data collection and analysis.

We invited 21 participants to play FPS Counter-Strike: Global Offensive (CS:GO) for 30-35 minutes. We note here that six pro-players took part in this experiment. All the participants were informed about the project and the experimental details. Every participant signed a written consent form which allowed recording physiological and in-game data. Then players were equipped with the sensors for data collection. We did not receive any complaints about the uncomfortable gaming experience because of the sensors. The experimental testbed is snown in Figure 4.

Players needed to play Deathmatch mode of CS:GO. The example of the game screen interface is shown in Figure 5. In this mode the goal of each player is to achieve as many kills of other players as possible and to minimize the number of their own deaths. When the player is killed it immediately respawns in a random location in the game. This mode is often used by eSports players in their training routine. After the game had been finished, we saved the replays for the future game events extraction.

Collected data samples for the 5-minutes intervals for two players are shown in Figure  6. For the reader convenience, we color the intervals in the 1-second vicinity of kill and death events.

It is clear that for the data from some physiological indicators, e.g. skin resistance or heart rate, there are global and local trends, which might align with changes in the player’s efficiency in the game. Another point is that the players usually do not move a lot on a gaming chair. However, they can change their posture from time to time, and this event is captured by the IMU on the chair and also might be connected with the game events and player performance.

To get rid of the noise and occasional outliers, we have clipped all the data by 0.5 and 99.5 percentiles and smoothed the data by 100 $ms$ moving window. We have also reparametrized gaze, mouse, and muscle activity signals. Mouse signal has been converted from $x$ and $y$ increments to Euclidian distance passed to reflect the mouse speed; gaze data has been transformed from $x$ and $y$ coordinates to Euclidian distance passed; muscle activity signal has been changed to L1-distance to the reference level for the player in order to represent the intensity of muscle tension. For 3.7% data missed we have used linear interpolation to fill in the unknown values since it provides a stable and accurate approximation.

There is no generic player effectiveness metric for the majority of eSports disciplines. The most popular evaluation metric for FPS and MOBA games is Kill Death Ratio (KDR) [60]. It equals the number of kills divided by the number of deaths for the time interval. If KDR ¿ 1, it means that the player performs well, or, at least, better than some players on a game server. Otherwise, the player most likely performs bad compared to other players.

KDR takes values from $0$ to $+\infty$ which is not a clear range for prediction. When the player performs very well and has many kills and few deaths, KDR is fluctuating drastically because of division by a small number. In opposite, if there are many deaths and few kills, KDR is around 0 and changes slowly. This drastic inconsistency in the target creates difficulties for training machine learning algorithms. One possible solution could be to apply logarithm to KDR, but this does not solve the issue with the scale, because logarithm takes the values from $-\infty$ to $+\infty$.

We propose a more numerically stable target value which equals the proportion of kills for the player. More precisely,

where $p_{\tau}(t)\in\mathbb{R}$ is the proportion or performance and equals the proportion of kills for a player at the moment $t\in\mathbb{R}$ considering the kills and death in the next $\tau\in\mathbb{R}$ seconds. In other words, it is the ratio of kills in the next $\tau$ seconds. $K(t)\in\mathbb{R}$ and $D(t)\in\mathbb{R}$ are the total number of kills and deaths at the moment $t$; therefore $k_{\tau}(t)\in\mathbb{R}$ and $d_{\tau}(t)\in\mathbb{R}$ equals the number of kills and deaths within the interval $[t,t+\tau]$. $p_{\tau}(t)$ varies from 0 to 1 and has higher values for well-performing players. Bounding by 0 and 1 helps efficiently train the machine learning algorithms that are sensitive to the target scale.

The important hyperparameter introduced above is $\tau$. Essentially, it is a window size for which the information about the future player performance is aggregated. Small values of the hyperparameter like 1 $s$ lead to noisy target values, while large values like 10 minutes neglect the subtle yet important changes in the player performance. $\tau$ is commonly referred to as a forecasting horizon.

$p_{\tau}(t)$ is a well defined metric for evaluating the player effectiveness, and it is possible to predict this value directly, thus considering the problem as a regression problem. However, it is unclear how to interpret the quality of regression results in an understandable and interpretable way. Formulating the problem in classification terms helps measuring the quality of prediction by more comprehensive classification metrics like accuracy, ROC AUC, and others. These metrics are much easier to compare with results obtained on other data or by other models.

The natural way to claim if a person plays well or bad at the moment is to compare the current performance with his or her average performance in the past. It is important to consider the past events only to avoid overfitting. Formally:

where $y_{\tau}(t)\in\{0,1\}$ and equals 1 in case of good game performance and 0 otherwise; $\overline{p_{\tau}(t)}$ is an average player performance in the past.

Figure 7 demonstrates how the kills ratio $p_{\tau}(t)$ and corresponding binary target $y_{\tau}(t)$ changes over time for three forecasting horizons.

The substantial advantage of using $y_{\tau}(t)$ instead of $p_{\tau}(t)$ is target unification between the players. $0$ and $1$ values of $y_{\tau}(t)$ have the same meaning for all players and imply bad and good current performance, respectively. That is not the case for raw values of $p_{\tau}(t)$, because the same values of $p_{\tau}(t)$ may be good for one player, but bad for another one. For example, 0.5 kills ratio may be an achievement for a newbie player, but a failure for a professional player.

Another justification of using $y_{\tau}(t)$ as a target is robustness to the skill of other players on a server. Player’s score $p_{\tau}(t)$ may be too low in absolute value because of the strong opponents, but target $y_{\tau}(t)$ is robust because it evaluates the performance within one game. The motivation for this target is to unify the target variable for all players and to provide in advance an immediate feedback for a player or for a manager that something is going wrong.

Predicting future player performance $y_{\tau}(t)$ is essential for coaches and progressing players as it provides the quick feedback on players’ actions. That might help identify the inevitable failures in player performance, e.g. burnout, fatigue, etc., in advance and take measures to help the player to recover or even to change the player during the eSports competition. This target is also helpful for learning purposes: although a person plays very well or very poor, it helps find the moments when the player performs a bit better or worse than average.

Despite we formulate the performance metric in terms of kills/deaths, the metric is directly applicable to the majority of First-Player Shooters, as well as other games including kills/deaths. The performance metrics for these and other games can be calculated in other terms (such as gold, scores, progress, etc.), while the data processing and algorithms may be the same.

We trained four models for predicting a player performance using the data from sensors: baseline model, logistic regression, recurrent neural network, and recurrent neural network with attention. In this section, we describe these methods in detail. The output for all models is the probability that the person will play better in some fixed period in the future. All the models are evaluated by ROC AUC score discussed in section III-G2.

We have trained the predictive models for several time steps $\Delta{t}$ (see Section III-D1) and forecasting horizons $\tau$ (see Section III-E) combinations. According to our experiments, reasonable ranges are from 5 to 30 $s$ for time step, and from 60 to 300 $s$ for the forecasting horizon. The average results for the neural network with attention are shown in Table III.

According to Table III, the best time step value is about 20 $s$, and the best forecasting horizon for the model is from 3 to 5 minutes. In other words, the optimal way to predict the player behavior is to aggregate the sensors data every 20 $s$ and make a prediction for the next 3-5 minutes.

We have compared the performance of the algorithms described in Section III-F with respect to the time step values with fixed forecasting horizon $\tau$ equal 180 $s$. The results are shown in Figure 9. There is a clear peak near 20-25 $s$ for all the methods. The neural network consistently outperforms the logistic regression and the baseline model. The use of the attention block helps increase the model score.

Figure 10 demonstrates the relation between the forecasting horizon and algorithms performance with the time step $\Delta{t}$ equal to 20 $s$. The neural network outperforms other methods and achieves the maximum performance for forecasting horizons in the range from 3 to 5 minutes. The attention block helps improve the model.

In order to interpret the neural network predictions, we calculated the feature importances and visualized predictions of a pretrained network and its internal state for the discretization step equal to 20 $s$ and the forecasting horizon equal to 3 minutes. Figure 11 shows how attention, network hidden state, target, and network prediction change over time. Clearly, the importance of different features varies over time, and periodically the data from some sensors is non-relevant. The network hidden state, which can be treated as a player state, also varies during the game.

To calculate the feature importance, we trained 100 instances of the neural network with random weight initializations and the train/val/test splits, and the averaged attentions on the test set for the best epoch of each neural network. Afterwards, we averaged the results across all networks. The results are shown in Table IV and Figure 12.

Information about physiological activity such as heart rate, muscle activity, hand movement, and so on is the most relevant for the network. It is worth noting that all the feature groups have considerable importance, thus contribute to the overall prediction. Since the features in each feature group are mixed, we can not estimate the feature importance for every raw feature.

Experimental results have shown practical feasibility of predicting the eSports players behavior using only data from sensors. The system updates the prediction several times per minute, so this interactivity is enough for potential users like eSports managers or professional players to understand that something goes wrong or to get a quick feedback. Feedback from the algorithm about the performance prediction and feature importance may suggest the users to change their gaming behavior. For example, an eSports manager may understand in time that bad results of the team are connected with the stuffy environmental conditions and adjust the air conditioning. Users may find information from the system useful to make the radical decisions, e.g., changing a player or gaming equipment (computer mouse, display, game chair, etc.).

We have found that 20 $s$ time step and 3-5 minutes behavior forecasting horizon are the most natural parameters for CS:GO, but potential users can set up any other hyperparameters depending on their scenario. Significant advantage caused by the diversified training dataset is universality for the player skill level enabling a wide range of potential users. Negligible model inference time and data collection time on a PC proves the principal feasibility of deploying the model on the edge devices, some of which might be designed specifically for neural network operations. However, the model retraining still requires high computational capability. Larger and more diversified dataset could improve the performance of prediction.

The limitations of the study is a small number of participants involved, the fuzziness of the definition of player’s performance, and overrepresentation of data from males and young people in the dataset. Future work includes more diverse and extensive data collection with more subjects recorded, and the investigation of better metrics of players’ performance. These would allow researchers to utilize more complex machine learning methods and to develop more reliable and robust models.