A Deep Reinforcement Learning Blind AI in DareFightingICE

Audio signal processing is one of the most important areas of Artificial Intelligence. In recent years, because deep learning has become more and more ubiquitous, it has been applied to audio processing and therefore has gained successes in applications such as speech recognition[8],[9] and text-to-speech[10], [11]. One common way to apply audio processing in deep learning is to convert the audio data into images and then process them as with other images. This can be done by generating spectrograms, which are 2D images representing sequences of spectra with time and frequency along two axes and with color representing the strength of a frequency component. A spectrogram can be obtained by applying Short-Time Fourier Transform (STFT) to the audio signal, and the STFT of a signal can be calculated using Fast Fourier Transform (FFT). A Mel-frequency scale can also be applied to spectrograms to create Mel-spectrogram, which is better for human perception and is used in [8] and [9]. In our work, as audio encoders for our blind AI, we compare three types of transformations that use a convolutional neural network (CNN) containing two 1D convolutional layers, a combination of FFT and a 2-layer fully connected network (FCN), and a combination of Mel-spectrogram and a CNN containing two 2D convolutional layers, respectively.

There have been a number of existing studies that focused on game playing AI using sound. Gaina and Stephenson[4] expanded the General Video Game AI framework to support sound and trained AIs to play the game using only sound as the input. Hegde et al.[5] extended the standard VizDoom framework [12] to provide the in-game sound to AIs and trained AIs in a series of increasingly complex scenarios to test the perception of sound. Results from these studies show the potential of research on an AI that learns to play games from sound. However, in these studies, AIs were trained with only one sound design and were not used to evaluate the effectiveness of sound designs in games. Therefore, our work, to the best of our knowledge, is the first time an AI is used to train with multiple sound designs and to evaluate their effectiveness.

The Proximal Policy Optimization (PPO) [13] algorithm has been a popular reinforcement learning approach in recent years. PPO provides a reliable Trust Region Policy Optimization based on previous policy gradient methods and outperforms traditional methods such as Q-learning. Its policy loss function is as follows:

In the equations above, $s_{t}$ and $a_{t}$ are the state and action at timestep $t$, respectively, $\pi_{\theta}(a_{t}|s_{t})$ and $\pi_{\theta_{old}}(a_{t}|s_{t})$ are the probability of $a_{t}$ given $s_{t}$ of the current policy and the previous policy, respectively. $V_{\theta}(s_{t})$ is the value function of state $s_{t}$, and $\epsilon$, $\lambda$, $\gamma$ are clipping, Generalized Advantage Estimate (GAE) and discount factor, respectively. In the Fighting Game AI competition, PPO had been used in a number of studies [14, 15, 16] and achieved a remarkable success, especially as can be seen from the 2021 champion and runner-up both using PPO as part of their AI¹¹1https://www.ice.ci.ritsumei.ac.jp/ ftgaic/index-R.html. In addition, PPO is outstanding in audio processing tasks, such as audio-based navigation in a multi-speaker environment [17] and semantic audio-visual navigation, where objects’ sounds are consistent with their semantic meaning [18]. In this work, we use PPO to train our blind AI.

Many state-of-the-art algorithms tested their performance in the Fighting Game AI Competition, including deep reinforcement learning (DRL). Kim et al. [14] created a fighting game AI agent using deep reinforcement learning with self-play and Monte Carlo Tree Search (MCTS), and they later proposed a reinforcement learning agent to tune the environment slightly by reusing latent space obtained from a different environment[16]. In 2020, Tang et al. [19] proposed a method that combined the Rolling Horizon Evolution Algorithm with an opponent model and won the competition that year. In 2021, Liang et al. [15] extended the said work by proposing PPO with an Elo-based selection mechanism in which strong historical AIs are more frequently chosen during training. However, all these AIs use frame data provided by the competition until 2021. From 2022, the DareFightingICE competition [6], using the titular game platform, has been launched where entry AIs are required to play the game from audio data only and have no access to frame data information. Therefore, our work is the first effort to train an AI to play a fighting game using only sound as the input.

Raw audio data provided by DareFightingICE are in the form of a vector ${\textit{{s}}}\in[-1,1]^{n}$ containing $n$ normalized audio samples. The original raw audio data size is 800 for each of the two channels (left and right), but each channel is padded with zeros so that it has 1024 samples for the sake of FFT in Java[6]. In our work, however, we choose to use only the 800 original samples for each channel. Motivated by the work of Hedge et al.[5], we propose and compare three encoders to process the audio data, before feeding them to a deep neural network, in the following. Figure  1 shows the architectures of these encoders, where all the parameters are based on previous work  [5], but empirically adapted for our work.

DRL refers to a growing set of powerful algorithms which use deep neural networks to learn in environments that have high dimensional states and actions. In our work, as stated earlier at the end of II-C, we use PPO whose architecture and reward are described in the following.

Here we propose two metrics for evaluating the effectiveness of a given sound design and/or a given audio encoder: $win_{ratio}$, and $avgHP_{diff}$. They are defined in a way that the more effective a sound design or an audio encoder is, the higher values these two metrics will be. We first train three blind AIs, each using a different audio encoder. The opponent AI in use is a weakened version of the MCTS sample AI in the preceding competition, MctsAi65 discussed in Khan et al. [6]. Each training lasts 900 game rounds. MctsAi65 is an AI whose execution time is reduced to 6.5 ms and uses frame data as the input. Because MctsAi65 was selected in [6] for human evaluation, as it is not too strong for visually impaired players to play, we choose it as the opponent AI.

We then evaluate the fighting performance of each trained blind AI by making it fight against the aforementioned opponent AI for 90 rounds. The ratio of the number of wins³³3In the game, the round winner is either the one with a non-zero HP while its opponent’s HP has reached zero or the one with the higher HP when the round-length limit of 60 s has reached. over 90 rounds, Eqn. (8), and the average HP difference at the end of a round between the trained AI and its opponent, Eqn. (9), are then calculated.