Using Graph-Aware Reinforcement Learning to Identify Winning Strategies in Diplomacy Games (Student Abstract)

The first step involved identifying the action space for the players in the Diplomacy games in terms of the rhetorical strategies that they follow to influence and persuade each other. However, only 60% of the CL-Aff Diplomacy dataset had high-quality labeled data. Hence, a weakly supervised approach was followed to predict the rhetorical strategies for the entire dataset. The training set comprising the labels from CL-Aff Diplomacy was used to train binary classifiers on the different rhetorical strategies, such as Friendship (F), Reasoning (R), Game Move (GM), and Share Information (SI). The best-performing classifiers (reported in Table 1) were used to predict labels for the entire dataset.¹¹1More results can be found at https://tinyurl.com/diplomacy-rl/

In the second step, we identified the winning player in each conversation as a function of their rhetorical strategies. We formulate this as an inverse RL problem and, more specifically, use score-based inverse reinforcement learning (SBIRL) (El Asri et al. 2016), which allows us to exploit the experiences of non-expert agents as well.

Each state $s$ of the state space $S$ is encoded using $\phi:S\rightarrow\mathbb{R}^{d}$. The encoding is done by picking a set of characteristic features corresponding to that state. Simpler operationalizations rely only on the player’s score at any given point. On the other hand, a graph-aware operationalization incorporated the player’s importance and influence in the textual communication network for the game through several centrality features. For the reward function, a linear parameterization was adopted: $r_{\theta}(s)=\theta^{\intercal}\phi(s)$ where $\theta$ is the set of parameters. Each thread $t$ of conversation is represented as:

We extracted the thread-level tuples $(t_{i},f_{i})$, where $t_{i}$ is the subsequence of states corresponding to the $i^{th}$ player (referred to as a subthread) and $f_{i}$ is the final score of the $i^{th}$ player at the end of the thread. For any given subthread and reward, the discounted sum of rewards can be written as

Where $\gamma$ is the discounting factor. We regress the final scores $f_{i}$ on the mappings $\mu(h_{i})$ and asymptotically minimise the risk based on the $\ell_{2}$-loss. A reward function estimator $r_{\theta_{n}}$ is derived after estimating $\theta_{n}$ by solving:

Once the annotation labels were obtained, key player strategies were identified. At the player level, Reasoning (R), Game move (GM), and Share Information (SI) all shared a strong pairwise Pearson correlation ($r\in(0.45,0.55)$), while each were strongly anti-correlated with Friendship (F) ($r\in(\textrm{-}0.61,\textrm{-}0.45)$), both with $p<0.001$. This suggests that players either acted on a Friendship or a Reasoning strategy and enabled us to label each utterance as an action under assumptions of mutual exclusion. When both labels were present, a voting mechanism with the Game move and Share Information was used to determine the ultimate label.

The winner of a chat thread is the player with the higher score at the end of the thread. The accuracy of the reward function is the fraction of times when the average estimated reward for the winner was greater than that of the loser.

First, we encoded only the difference between the game scores of the two players in our state feature vector. Next, given that each player is engaged in multiple conversations at the same time, we introduced multiple graph centrality features, such as authority score and eigen centrality, into the state representation²²2Calculated using igraph, https://igraph.org/r/. The results are reported in Table 2. We note that our graph-aware approach outperforms the context-agnostic approach, which relies solely on sociolinguistic cues and ignores interplay between players across multiple threads. Additionally, we restrict each player to their first $n$ utterances and perform the same exercise. Fig. 1 shows that the graph-aware approach achieves modest accuracy with the very first half dozen chat messages.