Adaptive Background Music for a Fighting Game: A Multi-Instrument Volume Modulation Approach

As the popularity of video games keeps growing, video game developers are trying to improve their games to leave a positive impression on the players by enhancing their experience [1]. These improvements are in many different forms like gameplay-related changes or visually pleasing games or games with very enjoyable music or sound effects. Regardless of the type of improvement, the ultimate goal is to enhance the overall experience of the players.

Our work is based on DareFightingICE Competition, which has two tracks: Sound Design Track and AI Track [2]. It ran at the 2022 IEEE Conference on Games (CoG 2022). The platform for the competition is DareFightingICE – an enhanced version of FightingICE [3] – with an enhanced sound design and a new AI interface that can give audio data to AIs.

This research focuses on video game music and how to use it to enhance players’ experience. An immersive gaming experience is greatly aided by BGM [5, 6]. In fighting games, BGM should change in response to the dynamic action, creating suspense and excitement as the players engage in combat. Fighting game BGM is typically implemented using pre-composed songs that loop, which fall short of accurately capturing the intensity of combat.

As a result, the target genre in this research is fighting games. In fighting games, players go against another player or a computer player in a one-versus-one fight, using different attacks and abilities to overcome the opponent. These fighting games are two-and-a-half-dimensional, which means that the players can only move in two dimensions (left or right, while in some fighting games the players can perform an attack or dodge that takes them out of 2D). In addition, because fighting games are fairly simple compared to other genres of video games, they are open to a larger audience. This research uses DareFightingICE.

The goal of this research is to create adaptive BGM by modifying the BGM of DareFightingICE, and evaluating the performance of the adaptive BGM by using a deep learning AI that only uses audio as input (Blind DL AI) [4]. The contributions of our work are as follows:

1. 1.

   One we are the first group to focus on adaptive music in fighting games.

2. 2.

   we are also the first group to use multiple instruments in the adaptation of a BGM.

3. 3.

   our research focuses on giving players information about the state of the game through the BGM.

Since our research touches on two topics which are audio in video games and adaptive audio in video games, this section is divided into two parts.

In this research, we propose an adaptive BGM that adapts to player actions and the players’ in-game position. The proposed adaptive BGM consists of five different instruments playing a classical music piece called “Air on G-String.” This music was selected for this research because we found the contrast of fast-paced actions with calming music to be a good combination, a technique used in some famous and popular movies such as X-Men: Days of the Future Past (QuickSilver-stopping-bullets scene). The five different instruments are the piano, the cello, the flute, the violin, and the ukulele. The BGM adjusts by altering the loudness of the instruments. This adjustment happens when the game elements, these instruments are connected to, change. The elements in use are both players’ health points (HP), energy points (EP), and the distance between the two players (PD). The HP is the number of hits a player can take before losing, and the EP is the energy in point that the players need to perform different attacks. The design of the proposed adaptive BGM is illustrated in Fig. 1.

In Fig. 1, there are five instruments playing together to compose the BGM, and every instrument is connected to a different game element. The violin and the piano are connected to player one’s HP and EP, respectively; and the flute and the ukulele are connected to player two’s HP and EP, respectively. Lastly, the cello is connected to PD.

In Fig. 2, the instruments connected to HP of both players are at maximum volume when HP is maximum, and the volume is turned down as the HP gets lower. The ones connected to players’ EP work in the same way, i.e., when EP is low the volume is low and when EP is high the volume is high. For the cello, the volume is at maximum when PD is close to zero meaning they are very close to each other, and it becomes lower the further away the players move from each other. The proposed adaptive BGM is designed this way to give useful information to both human and AI players.

The maximum HP for one player per round is 400, EP is 300, and the PD is 800 pixels horizontally. The HP and EP of both players and their PD are divided into different levels to decrease or increase the volume of the instruments gradually. The levels of not only instruments but also players’ HP, EP, and PD are empirically selected due to the lack of existing research when it comes to adaptive BGM using volume modulation.

The levels for HP are 400, 300, 250, 200, 150, 100, and 50, which are connected to the instrument’s volume levels of 75%, 60%, 55%, 40%, 35%, 25%, and 10% respectively. For EP of both players, the instruments’ volume levels are the same as HP, but EP levels are 300, 250, 200, 150, 100, and 50. For PD, the levels are 800, 600, 500, 400, 300, 60, and 0, which are connected to the instrument’s volume levels of 10%, 20%, 30%, 40%, 50%, 60%, and 75%. The reason for the abrupt change from 300 to 60 is for the players without vision to know without a doubt that the opponent player is near them.

Lastly, for the 2023 DareFightingICE Competition participants, if they want to modify or change the adaptive BGM¹¹1code of the adaptive BGM: http://tinyurl.com/Adapt-BGM. They just have to change the five instruments playing the BGM and if they want to use fewer than five instruments, they can decide which rules to ignore and which to select. Figure 3 shows a code snippet that they can modify. In Fig. 3, the play2() function takes five parameters as follows:

1. 1.

   The first parameter is the audio source of the instrument.

2. 2.

   The second parameter is the audio buffer that stores the audio file that will play on the audio source.

3. 3.

   The third parameter is the horizontal location in the game, where the audio source will play.

4. 4.

   The fourth parameter is the vertical location in the game.

5. 5.

   The last parameter is to check if the audio source should be looped or not.

For the evaluation of the adaptive BGM, we did an objective evaluation, the details for which are following.

For the objective evaluation, we use the aforementioned Blind DL AI. Our method for this evaluation is to train the Blind DL AI with and without the adaptive BGM and then compare the performance of the AIs. We hypothesize that if the performance of the Blind DL AI is better with the adaptive BGM then the adaptive BGM is giving useful information, and since the AI we are using can only take audio data as input we believe that players without vision will also be able to use the same information for their advantage.

To test our adaptive BGM we used the submissions to the 2022 DareFightingICE Competition Sound Design Track and train the Blind DL AI as mentioned above. We trained the Blind DL AI for 900 rounds on DareFightingICE 5.2 (1 game has 3 rounds) against MCTSAI65, a weaken version of a sample Monte-Carlo [18]. For DareFightingICE 6.0, we implement a new weak version of MCTS AI, which fixed the problem of MCTSAI65’s performance difference on different environments, and call it MCTSAI23i. This version of MCTS AI has similar performance as the previous MCTSAI65 against our Blind DL AI as shown in Table  I. We trained the Blind DL AI against the MCTSAI23i for 900 rounds for each sound design.

The difference between MCTSAI65 and MCTSAI23i is that for MCTSAI65 the MCTS execution time was changed to 6.5 ms and for MCTSAI23i we limit the number of iterations MCTS can do for each frame. We introduce this change because we found that MCTSAI65 performed differently in different environments.

For the subjective evaluation or evaluation of our adaptive BGM by human players, since the performance of Blind DL AI was similar to the performance of human players in previous work [2], we decided to only go with the objective evaluation. We believe that both players with and without vision will also be able to use the same information to their advantage.

We used the same environments as in the 2022 DareFightingICE Competition. More specifically, six computers were used that have the same specification, i.e., CPU: Intel(R) Xeon(R) W-2135 CPU@ 3.70GHz 3.70 GHz, RAM: 16 GB, GPU: NVIDIA Quadro P1000 4GB VRAM, and OS: Windows 10.

This paper presented a rule-based adaptive background music (BGM) system that consists of five different instruments playing a classical music piece called “Air on G-String.” The proposed adaptive BGM adapts by changing the volume of the instruments. Each instrument is connected to a different element of the game. The paper also showed that the performance of a deep reinforcement learning AI using only audio data as its input, called Blind DL AI, improved while playing with the adaptive BGM as compared to playing without it. We believe that both players with and without vision will also be able to use the same information given by the proposed adaptive BGM to their advantage, just like the Blind DL AI does.

In the future, we are planning to use a deep learning approach to the adaptation of BGM. This is because in the current work, we used the performance of the Blind DL AI when fighting against a version of a Monte-Carlo tree search AI to assess our adaptive BGM and we did not do any aesthetic evaluation of it. We believe that the adaptive BGM using deep learning would result in achieving a better performance from the Blind DL AI as compared to the rule-based, as we plan to use the Blind DL AI’s performance as a part of the reward function. As for aesthetic evaluation, since our current work is a rule-based adaptive BGM decided the rules taking into account the resulting BGM’s aesthetics; However, by using a deep learning approach we anticipate that the aesthetic evaluation of the BGM would be needed.