Robust Vision-Based Cheat Detection in Competitive Gaming

Data mining techniques can examine the memory of the host machine (Philbert, 2018) to detect the presence of specific cheating software, but these solutions raise privacy concerns, as the anti-cheating software can access the entire memory of the host. To prevent a privacy breach, some propose server-side cheat-detection based on game logs to find anomalies in-game events and performance metrics (Alayed et al., 2013). Given the large amount of data in the logs, machine learning techniques like decision trees, naive Bayes, random forests, DNNs, or support vector machines (Galli et al., 2011)(Kotkov et al., 2018) are a good fit for cheat detection, but currently available anti-cheating systems mainly focus on non-visual features (FACEIT, 2021; Punkbuster, 2000; Wells, 2020; Cox et al., 2019) (Islam et al., 2020). For example, Punkbuster (Punkbuster, 2000) scans the memory of the player machine, HestiaNet (Wells, 2020) analyzes data using machine learning, VACNet (Cox et al., 2019) analyzes players actions and decisions. Since the algorithms and source code of these commercial tools are unpublished, the details are unavailable for objective evaluation and comparison. To the best of our knowledge, anti-cheating solutions based on processing of the rendered frame, a much safer approach from the privacy point of view, are currently nonexistent. It is essential to notice that this approach is safe from the privacy point of view, as it only requires as input the rendered frame residing in the GPU buffer before showing it on display. Notice that frames do not need to be transmitted if the detection is performed on the local machine. Furthermore, the computational overhead is limited, as frame analysis can be performed with low priority and not necessarily in real-time.

In the context of visual cheating detection, it is crucial to estimate not only the probability that a given frame contains cheating information but also the confidence of such an estimate. We leverage confidence analysis to improve two aspects of our cheating detection system that are fundamental for its practical adoption: to prevent reporting legitimate gamers as cheaters; and to identify the need for DNN retraining, which may occur every time a cheating software is modified, or a new one introduced.

It has been indeed demonstrated that, for a DNN classifier, the output probabilities are poor indicators of its confidence (Sensoy et al., 2018). Procedures to estimate the DNN confidence have been consequently proposed. A simple method is based on dropout in the intermediate layers (Michelmore et al., 2018): by collecting sufficient output statistics for any given input, two metrics (the variation ratio and mutual information) that correlate well with confidence level can be computed. However, such a method is computationally demanding, and it is based on intuition more than a well-established theory. Taking into consideration the confidence output not only at inference time but also during training provides further advantages. Sensoy et al. describe a principled approach, where the loss function models prior and posterior multivariate classification distributions with Dirichlet distributions representing not only the target classes of the classifier but also an additional class for uncertain inputs. This leads to a DNN that outputs the probability that the input belongs to a given class and the level of confidence, improving the ability to detect queries that do not belong to the training distribution and the DNN robustness against adversarial examples (Sensoy et al., 2018).

Adversarial attacks can fool DNNs using images that appear innocuous to humans but produce a target output (Carlini et al., 2019; Xu et al., 2019). In the context of cheating detection, in addition to adding visual hacks to the rendered frames, cheating software may also add an adversarial pattern to fool our DNN detector. Although generating an adversarial perturbation that is imperceptible to the cheating gamer is not strictly needed, an attacker should anyway be careful not to corrupt the frame to the point where playing becomes uncomfortable, or the adversarial perturbation reduces the visibility (and thus the utility) of the visual hack.

Several attacking strategies have been proposed. Some of them, like the Fast Gradient Sign Method (FGSM (Goodfellow et al., 2015)), Projected Gradient Descent (PGD (Kurakin et al., 2017)), and its improved version with random start (Madry (Madry et al., 2018)) can be computed rather quickly, and thus meet one of the requirements to be potentially employed by a cheating software: the possibility to be computed on the fly and consequently applied to a cheating frame in real-time. More powerful attacks, like black-box attacks, are not practically feasible due to their computational complexity. FGSM and PGD belong to the so-called class of white-box attacks, where the attacker has access to the full DNN model. This may seem an insurmountable issue for an attacker, but researchers have demonstrated that adversarial attacks do transfer, i.e., attacks computed on a given DNN are often successful on a different DNN trained on the same dataset (Demontis et al., 2018). Furthermore, researchers also discovered universal attacks, i.e., perturbations that can be computed offline (thus with zero computation overhead) and applied to an entire set of images, with a discrete attack success rate (Moosavi-Dezfooli et al., 2017).

Together with attacking methods, the researchers also developed several defense techniques, but a solution that guarantees complete protection on general multi-class problems is not yet available. Because of its simplicity, adversarial training has become a popular method to prevent attacks and create robust DNNs (Goodfellow et al., 2015). It consists of adding adversarial images, generated with fast methods like FGSM, in the training dataset, so that the DNN learns to classify them correctly; however, the protection is not complete, and there is no mathematical guarantee of the achieved level of protection. A different defense method, IBP (Gowal et al., 2018), can be interpreted as a generalized form of adversarial training: instead of generating training adversarial images, an upper bound for the worst-case adversarial perturbation is propagated through the DNN, allowing to establish a bound on the maximum output change, and training performed on the worst-case scenario. IBP has the additional advantage of being capable of explicitly computing the bound and thus providing a guaranteed level of protection for a given magnitude of the input perturbation.

Existing defense techniques treat all the classes equally, while our application presents a different opportunity: we are only concerned with false negatives generated by adversarial attacks, while the second case (turning a non-cheating frame into a cheating one) has no practical importance for the attacker. Thus, we consider the case of asymmetric protection for the first time to the best of our knowledge. Similarly, and again for the first time to the best of our knowledge, we also study the interaction of the uncertainty loss (Sensoy et al., 2018) and IBP, which may be of interest beyond its application for anti-cheating.

To identify the best reduction layer in the global head and test the effect of the additional local head onto the global detection performances, we trained our DNN detector using the fully connected, max pooling, or average pooling layer in the global head, minimizing $CE_{global}$ or $CE_{combined}$ on data from $S_{A}^{train}\bigcup S_{B}^{train}\bigcup S_{C}^{train}$, and measure the accuracy of the DNN global head on each testing set separately. Data in Table 2 show that the detector’s overall accuracy generally improves when moving from the fully connected to the max-pooling to the average pooling layer. Furthermore, when average pooling is adopted in the global head, the accuracy increases when moving from minimization of the global cost function, $CE_{global}$, to the combined minimization of the global and local detection error. Overall, this table suggests that the highest accuracy is achieved using an average pooling layer in the global head and combined training of the local and global detector heads.

To better highlight the advantages and disadvantages of adopting a different reduction layer in the global head and the two cost functions, we computed the accuracy of the DNN detector, trained using data from the entire dataset, and tested on $S_{B}^{test}$ and $S_{C}^{test}$, measuring the accuracy as a function of the amount of cheating information present in the frames (full, medium, or minimum). The results in Table 3 show that, even when disentangling the cheating information level, the use of the combined cost function together with the average pooling layer in the global head leads to the overall better accuracy, thus confirming the previous results. This same Table highlights that, when a minimum amount of cheating information is added to the scene, the detection task becomes more challenging, as expected, but at the same time also the cheating information becomes less useful for the cheater because of the tiny size, low contrast and visibility of the added visual hack. Surprisingly, we also found that the accuracy is higher for a medium level of cheating information than the full cheating case. We speculate that the full cheating configuration may add cheating information that is possibly tiny, poorly relevant, or not always present in any frame, thus hard to be learned by the DNN detector that eventually treat it as noise, thus decreasing the overall accuracy. More investigation is needed to better clarify this aspect, which, however, does not significantly affect the performance achieved overall by the DNN detector.

Michelmore et al. (Michelmore et al., 2018) describe a method based on dropout to estimate a DNN output’s confidence. The rationale behind the method is that, even in the presence of noise, the network output should not change much if the confidence is high, and vice-versa. Following (Michelmore et al., 2018), we introduce a dropout component (with $p=0.15$) after the average pooling layer of the global head detector and process the same input image $T=64$ times to collect sufficient output statistics. Among the three metrics proposed in (Michelmore et al., 2018) that correlates with the confidence of the output, we experimentally found the best results when using the Variation Ratio, $VR[x]$, which expresses the output dispersion as:

where $x$ is the input, $y^{t}$ is the predicted class for the pass $t$, $c^{*}$ is the most frequently predicted class in $T$ passes. When confidence of the DNN is high, $VR[x]$ approaches zero, whereas it grows otherwise (to a maximum value of 0.5 for the two classes problem associated with our DNN detector).

Compared to the case of $VR[x]$, where the DNN confidence is computed on the fly for any input $x$, Sensoy et al. (Sensoy et al., 2018) introduce a more principled approach based on uncertainty loss that can be adopted not only to estimate the confidence at inference time, but also to improve training. Beyond estimating the output confidence, their approach leads to accuracy improvement and generates more regular decision boundaries that turn out to be useful also to defend against adversarial attacks (see next section). To estimate the confidence level, the authors of (Sensoy et al., 2018) introduce the concept of evidence of each output class. Given the logit vector of our DNN detector, $\{l_{0},l_{1}\}$, we define the evidences $\{e_{i}\}_{i=0,1}$ and alpha factors $\{\alpha_{i}\}_{i=0,1}$ as:

and derive the belief for each class, $\{b_{i}\}_{i=0,1}$ and the total uncertainty $u$ as:

There is a clear connection between the belief and the probabilities of each class estimated by a traditional softmax layer. In this framework an additional uncertainty class $u$ is added to represent the probability that the classification is indeed uncertain (notice that $\sum_{i}b_{i}+u=1$). Training of the DNN detector is performed by minimizing the uncertainty loss introduced in (Sensoy et al., 2018) for the global head detector. Among the three losses proposed in the paper, we found that the one based on MSE error (Eq. (5) in (Sensoy et al., 2018)) produces more stable results, and therefore we adopted it. For a detailed description of the cost function, we remind the interested reader to the original paper. In the following, we will indicate this uncertainty loss as $UL_{global}$.

When training the local and global heads together, to take advantage of the implicit spatial attention mechanism generated by the local head, it is necessary to guarantee that the gradient coming from the two heads have similar nature and magnitude. Otherwise, convergence may be badly affected. For this reason, when using $UL_{global}$ to train the global head, we use a Mean Square Error (MSE) loss (indicated as $MSE_{local}$) to train the local head instead of $CE_{local}$. The final cost function adopted to train the DNN detector with the uncertainty loss approach is then:

To estimate a representative baseline using the MSE loss, we also introduce the following loss: $MSE_{combined}=MSE_{global}+MSE_{local},$ where both the local and global heads are trained to minimize an MSE error with respect to the ground truth labels of the input frames.

The first application we envisioned for confidence analysis is to limit the probability of reporting a legit gamer as a cheater: in few words, a cheating frame is reported if and only if the confidence level is high. When no confidence criterion is adopted to detect cheating frames, i.e., when the DNN is trained using the $MSE_{combined}$ loss and confidence of the output is not estimated (first row of Table 4), approximately one-third of the cheating frames are correctly identified as containing cheating information (434 TP over a total of 434 TP + 1056 FN cheating frames); however, 96 FP non-cheating frames are incorrectly classified as cheating, which may lead to an unpleasant gaming experience for legit gamers, incorrectly reported as cheaters. The second row of the same Table shows the partial benefits of adopting the $VR[x]=0$ as a selection criterion (i.e., we discharge any frame whose variation ratio is not optimal): FP decreases to 51, which improves but does not solve the issue. The use of the uncertainty loss during training (third row) does provide a more significant improvement, as TP grows to 670, whereas at the same time FP is reduced to only 2, thus suggesting the uncertainty loss training does not only allows to estimate the uncertainty of the DNN output, but also lead to a much more robust classifier. It is then legitimate to ask how the VR criterion and the uncertainty loss interact: the fourth row in the Table demonstrates that these two criterions provides somehow complementary information about the confidence of the DNN output. When combined, one pays the cost of slightly reducing the number of TP (down to 519), while, on the other hand, reducing the FP to the optimal value of 0.

Overall, Table 4 demonstrates that confidence analysis is needed to create a detection system that does not suffer from the issue of incorrectly reporting legit cheaters and that the combination of $VR[x]$ and uncertainty loss represents a valid solution to this problem, at least on our dataset.

Re-training of the DNN detector is required anytime a change in the game or cheat features renders its output unreliable: we need a system that is capable of self-evaluation. To this scope, we extend the ideas presented in (Sensoy et al., 2018) to create a simple statistical tool that is experimentally capable of executing such a task on our dataset. More in detail, given the alpha factors $\{\alpha_{i}\}_{i=0,1}$, we define the Likelihood Ratio ($LR$) as:

$LR$ is always greater than or equal to one, and it indicates how likely a sample is to belong to the dominant class. It is also strictly correlated to the DNN confidence: as a matter of fact, a large evidence $e_{i}$ for one class leads to a large $\alpha_{i}$ value, small uncertainty $u$, and consequently large confidence and large LR. In other words, a highly confident DNN detector is expected to have a high LR, whereas, in case of degradation of its performances, LR decreases.

We leverage this simple and intuitive rule to automatically determine the need for re-training as follows. On the training dataset, we define three average likelihood ratios computed over all the instances of the predicted positive ($LR_{P}^{tra}$), negative ($LR_{N}^{tra}$), and all ($LR_{T}^{tra}$) classes. These are three target baselines for a DNN detector that operates with performances similar to those measured in training. Once deployed on the field, we measure the corresponding average likelihood ratios on the field: $LR_{P}^{fie}$, $LR_{N}^{fie}$, and $LR_{Tl}^{fie}$. We hypothesize that a significant degradation in any of the three likelihood ratios can be interpreted as a signal that DNN re-training is required.

To experimentally test the validity of such hypothesis, we trained four DNN detectors using the $UL_{combined}$ loss function and data either from $S_{A}^{train}$, $S_{B}^{train}$, $S_{C}^{train}$, or $S_{A}^{train}\bigcup S_{B}^{train}\bigcup S_{C}^{train}$. Then, we tested each detector on each set and measured the ratios between the average training LRs and those achieved on the field. Beyond reporting these ratios for each training-testing set pair, Table 5 reports the accuracy, sensitivity, and precision of each DNN on the testing sets. The sensitivity, defined as the $TP/(TP+FN)$ ratio, describes the percentage of cheating frames that the DNN detector identifies correctly. Ideally, we want the sensitivity to be close to one, but in the practical context of anti-cheating, we need it to be larger than a minimum value to guarantee that the detector indeed captures at least some of the cheating frames. The precision, defined as the $TP/(TP+FN)$ ratio, describes how many of the identified cheating frames do indeed contain some cheating information. In the context of anti-cheating, it is critical to keep the precision as close as possible to one. Otherwise, legit players risk being reported as cheaters. The Table shows a strong correlation between the decrease in the DNN performance quantified by a decrease in one of the likelihood ratios (as measured without resorting to any ground truth by the DNN on the field) and a degradation in the DNN sensitivity and precision. By adopting a simple, empirical criterion suggesting that re-training is needed if at least one of the field likelihood ratios is degraded by $50\%$ or more with respect to its training value, we can guarantee a minimum network precision of 0.9 on $S_{C}$, which is the most difficult among the three sets, and 0.96 in the case of training on the entire dataset. Interestingly enough, this simple criterion suggests that the DNN needs re-training anytime it is trained on a given set and tested on a different distribution, which appears reasonable and aligned with our expectations. It is important to notice that the re-training criterion reported here does not want to represent an optimal rule, as different thresholds on the LR degradation may be easily set to guarantee even higher performances. The network’s ability to detect when re-training is required will also help seamlessly deploy the network in the wild.

To generate adversarial attacks and test the performance of our DNN detector, we use DeepFool (Moosavi-Dezfooli et al., 2016) to generate universal attack (Moosavi-Dezfooli et al., 2017) from the entire set of training frames offline, while the FGSM and iterative (Madry) attacks are computed on the fly using Cleverhans  (Papernot et al., 2018). We generate only attacks relevant to the anti-cheating problem, i.e., that turn the cheating frame into a non-cheating frame for the DNN detector. The magnitude of the attacks varies from $\epsilon=0$ to $\epsilon=0.1$, for images normalized between 0 and 1. Notice that an $\epsilon=0.1$ attack is considered extremely strong in the context of adversarial attacks, and defending against such an attack is nowadays considered at least very hard in many practical multi-classes applications. Some adversarial images for $\epsilon=0.1$ and universal, FGSM and Madry attack are shown in Fig. 2. It is important noticing that even if larger $\epsilon$ values lead to successful attacks, they also corrupt the image potentially up to the point where the gaming experience is ruined (see, for instance, panel D in Fig. 2). Fig. 3 shows the effect of $\epsilon$ on image quality. We measure the degree of degradation of a frame corrupted by an adversarial attack through the Haar Wavelet-Based Perceptual Similarity Index (HaarPSI) (Reisenhofer et al., 2018; Kastryulin et al., 2019), which quantifies the perceptual quality. HaarPSI value is computed for the adversarial image against an uncorrupted original image. As expected, the image quality decays significantly with an increase in $\epsilon$. Therefore, a potential attacker must make a trade-off between the frame quality and the attack success rate.

Since we use the confidence analysis introduced in the previous section to determine if a cheating frame should be reported or not, we are interested in measuring how many of these true positives (and confident) are preserved under attack. Let then $TP$ and $TP_{\text{attack}}$ be the number of true positives on original frames and frames under adversarial attack. Since the effectiveness of the attack is measured by how many true positives are converted to false negatives by the attack, we use the ratio of $\frac{TP_{attack}}{TP}$ as a representative of the ability of the network to defend itself during an adversarial attack. In Fig. 4, we compare the $\frac{TP_{attack}}{TP}$ ratio for the DNN detector trained on the $MSE_{combined}$ loss function and thus without any form of protection against adversarial attack, with that achieved by the detector trained with the $UL_{combined}+IBP_{One-sided}$ cost function, which achieves protection by the combined effect of the uncertainty loss and IBP and guarantees the highest clean accuracy thanks to the asymmetric implementation of the IBP cost function. This figure demonstrates the importance of protecting the DNN detector against adversarial attacks, as a vanilla implementation of the detector is easily fooled even by small magnitude attacks. In contrast, the ratio of true positive preserved by the protected DNN allows it to operate as an effective cheating detector even under attack, and for large values of $\epsilon$ that already lead to the introduction of significant distortion in the frame (compare with Fig. 3).

To better understand the contribution of the loss function (and in particular of $UL_{global}$) and IBP and how they interact when the DNN is under an adversarial attack, we performed an ablation study and measured the TP, FN, FP, and TN statistics for the same DNN architecture trained with the different cost functions. For the ablation study, we set $\epsilon=0.0125$, which results in a reasonable trade-off between the attack’s strength and the frame quality, with a theoretical bound for defense provided by IBP above $50\%$. In all the cases, the $VR[x]>0$ criterion is applied to reject low confidence detections. Besides, we discharge frames with high uncertainty value $u$ as well. Results are reported in Table 6, where the first two rows show that, when IBP is not used in training, the theoretical protection bound is zero, suggesting that an attack is potentially possible on the entire dataset. The practically feasible attacks considered here (universal, FGSM, Madry) can indeed decrease the detector’s accuracy to $58\%$ in the worst case. In this situation, our DNN can still find many (389 TP) of the existing cheating frames, but this situation is not desirable as there is no theoretical robustness guarantee. The lack of a theoretical guarantee bound also affects the DNN trained with the $UL_{combined}$ loss (second row), which on the other hand, provides higher accuracy against the practical attacks, as already noticed in (Sensoy et al., 2018). The adoption of IBP without the uncertainty loss (third and fourth rows) increases the practical robustness of the DNN with respect to the vanilla $MSE_{combined}$ training, and an IBP bound can now be established to guarantee that more than $55\%$ of the frames in the dataset cannot be attacked. However, a side effect of IBP training is to increase the number of false positives (132) on clean data, a type of error that is incompatible with the practical deployment of an effective anti-cheating system. Training with one-side IBP reduces FP, but does not entirely solve the problem. The best results are achieved when both uncertainty loss and IBP are used: FP is minimized, and almost full protection against practical adversarial attacks is achieved simultaneously. Also, in this case, leveraging the problem’s symmetry with one-side IBP leads to the best compromise in terms of clean accuracy and level of robustness for the class of interest, with only one FP reported on the entire dataset, and 628 TP.

Overall, the results presented in these last sections demonstrate that the development of a reliable and precise enough anti-cheating detector requires a careful design that considers several aspects at the same time. We have demonstrated that the adoption of state of the art techniques for confidence estimation and adversarial defense, together with the leverage of the peculiar aspects of the cheating detection problem, allows creating a system that meets the practical requirements for its deployment on the field.