Game-Theoretical Analysis of Reviewer Rewards in Peer-Review Journal Systems: Analysis and Experimental Evaluation using Deep Reinforcement Learning

Game theory is a mathematical framework designed for understanding the interaction among rational agents [11, 12, 13]. A game in this context is a situation where a number of agents interact, each trying to optimize their own objectives.

In this setting, a strategy is a complete plan of action that defines what a player would do in every situation. A strategy profile $s\in\mathcal{S}$ is a vector that specifies a strategy for every player. Each player $i$’s utility function $U_{i}$ maps from the set of strategy profiles to the real numbers, representing the player’s preferences over the set of possible outcomes.

The Nash equilibrium, as defined in Definition 2.2, is a state of the game where no player can unilaterally deviate from their current strategy without decreasing their utility, given that all other players stick to their strategies. This concept is central in game theory and provides a solution concept for strategic interactions.

The payoff matrix is a straightforward representation of a two-player game where each player has a finite number of strategies [14, 15, 16]. This matrix offers an efficient method to examine the utilities for different strategy combinations.

This proposition defines a dominant strategy, which yields the highest utility for a player, regardless of the strategies chosen by other players.

This lemma establishes a connection between the dominant strategy and Nash equilibrium in a two-player game. If both players have dominant strategies, the strategy profile consisting of these dominant strategies is a Nash equilibrium.

This assumption is a basic tenet of game theory and is essential for our subsequent analysis. It posits that players will always act in a way that they believe will yield them the greatest utility.

Open-access (OA) journals have emerged as a significant development in the scholarly publishing industry [17, 18, 19]. These journals provide unrestricted online access to peer-reviewed scholarly research. In the OA model, rather than charging readers to access the content, the publication costs are usually covered by author fees or institutional funding.

The open-access publishing model has revolutionized the way knowledge is disseminated, allowing for a much wider spread of scholarly research. It is predicated on the philosophy of communal sharing of knowledge, making the fruits of academic endeavor available to all, free of charge. The financial burden is thus transferred from the reader to the author, ensuring that the paywalls, which may hinder the proliferation of knowledge, are eliminated.

An integral part of the academic publishing process, irrespective of the publishing model, is the peer-review process. This process is designed to maintain the quality and integrity of scholarly research. In the OA model, the reviewers play a crucial role in assessing the quality of papers.

Peer review acts as a gatekeeper, ensuring that only rigorous and high-quality research is disseminated [20, 21, 22]. The process involves the evaluation of an academic manuscript by experts in the same field as the paper. This stringent process helps to ensure that the research presented is of the highest standard and adds value to the field. The rigorous nature of this process is a critical factor in upholding the reputation and credibility of scientific research.

There are usually three options for the reviewers when assessing a paper: Accept, Revision, and Reject. An "Accept" decision means that the paper is ready for publication in its current form. A "Revision" decision indicates that the paper has potential but requires some changes before it is ready for publication. A "Reject" decision means that the paper is not suitable for publication in the journal.

The tripartite system of Accept, Revision, and Reject decisions provides a comprehensive framework for evaluating academic papers. An acceptance indicates a work of high quality, requiring no further alterations. A recommendation for revision, on the other hand, indicates that while the work holds promise, there are aspects that require refinement and improvement before it meets the standards of the journal. A rejection signifies that the work, in its current form, does not meet the necessary standards of quality and contribution to the field.

The introduction of a third reviewer in case of a disagreement between the initial reviewers ensures that the decision-making process is fair and unbiased. This mechanism ensures that a single reviewer cannot unduly influence the outcome, thus promoting the integrity of the review process. This procedure also underscores the importance of consensus in academic publishing.

In some OA journals, as a part of the incentive system, reviewers are rewarded with vouchers upon completion of their review tasks. These vouchers can be used for discounts on the open access publishing fees when the reviewers submit their own papers.

The voucher system serves as an incentive for reviewers, providing them with tangible recognition for their work. It is a critical aspect of the open-access model, as it helps to ensure a constant supply of reviewers. This system also encourages the reviewers to submit their own work to the journal, creating a cycle of submission and review that sustains the journal.

The temporal discounting of the voucher in case of a "Revision" decision introduces an interesting dynamic into the reviewer’s decision-making process. While the face value of the voucher remains the same, the delay in receiving it effectively reduces its value.

The quality of a paper is an inherent characteristic that can be assessed to varying degrees by different reviewers. We denote the true quality of a paper "p" as $S_{p}$, and we assume that each reviewer has a certain ability to assess this quality with some bias.

The quality score of a paper, though a subjective measure, provides an essential yardstick for evaluation. It encapsulates the perceived merit of the paper, including aspects such as originality, rigor, and contribution to the field. This score, though subjective, is a critical component in the decision-making process of the reviewers.

This assumption is critical to our model and reflects the real-world situation in the peer review process. Typically, reviewers do not have access to each other’s evaluations until after they have submitted their own. This ensures the independence and integrity of the review process.

The estimated quality score $\hat{S}^{i}_{p}$ encapsulates both the inherent quality of the paper $S_{p}$ and the personal bias of the reviewer $b^{i}_{p}$. The inherent quality $S_{p}$ is assumed to be a fixed attribute of the paper, known to all reviewers in an ideal world. However, the bias $b^{i}_{p}$ reflects the individual judgment of the reviewer and can vary widely among different reviewers. This variation introduces uncertainty into the review process, as reviewers may disagree about the value of $b^{i}_{p}$ and thus the quality of the paper $\hat{S}^{i}_{p}$.

Given the lack of common knowledge about $\hat{S}^{i}_{p}$, each reviewer must make their decision based on their individual perception of the paper’s quality, taking into consideration the potential decisions of the other reviewers. This forms the basis of the strategic interaction between reviewers in our model, as captured by the game $\mathcal{G}_{c}$. The individualistic nature of the $\hat{S}^{i}_{p}$ estimation is what drives the need for the Nash equilibrium analysis. Reviewers must not only consider their perception of the paper’s quality but also anticipate the reactions of the other reviewers to maximize their payoff.

It is generally accepted that there are certain thresholds in the quality score that determine whether a paper should be accepted, requires revision, or should be rejected. We denote these thresholds as $Th_{a}$ and $Th_{r}$, respectively.

The quality score thresholds provide a quantitative benchmark against which the merit of a paper can be gauged. The thresholds effectively segregate papers into distinct categories, allowing for a more streamlined review process. These thresholds, though arbitrary, serve to uphold the standards of the journal.

In the ensuing examination, we delve deeper into the implications of the postulated Assumption 2.8, which asserts that a reviewer’s primary objective is to maximize their voucher value. This assumption is made to provide a framework for analyzing the effect of the voucher system on the decision-making process of reviewers. It is pivotal to underscore that this assumption is a simplification of reality, as it abstracts away from other considerations that might influence a reviewer’s decision.

We take a moment to emphasize the role of our assumption, as specified in Assumption 2.8 and Remark 2.6, within the broader scope of our study. Our focus is on unveiling the strategic implications of the reward system, a task for which the assumption of reward maximization provides a tractable analytical lens. This assumption is not intended to be a definitive or comprehensive characterization of reviewers’ motivations, which, in reality, are multifaceted and may encompass a commitment to scholarly rigor, the advancement of their field, and upholding the integrity of the scientific method.

The underpinning assumption, Assumption 2.8, can be perceived as a behavioral model, an economic perspective on the decision-making process of reviewers. The assumption postulates that each reviewer aims to maximize their voucher value, i.e., reviewers are essentially trying to maximize the expected present value of the voucher they receive. It is worth pointing out that this assumption does not necessarily signify that reviewers prioritize their self-interest over the integrity of the review process. Instead, it provides an economic rationale for the reviewer’s decision-making process, which complements the existing understanding that the reviewer’s decision is primarily driven by their assessment of the paper’s quality.

In the light of this assumption, the decision-making process of a reviewer can be modeled as a form of strategic interaction where each reviewer makes their decision considering the potential decisions of the other reviewer, the quality of the paper, and the voucher system’s time-dependent nature. The reviewer’s decision-making process thus becomes an intricate dance between maintaining the integrity of the review process, assessing the quality of the paper, and maximizing the expected voucher value.

In the context of our study, the time-dependent nature of the voucher system introduces a dynamic that further complicates the reviewer’s decision-making process. It is important to note that, under Assumption 2.3, the "Revision" decision effectively discounts the value of the voucher due to the time delay in its receipt, even though the face value of the voucher is the same for all three decisions. Hence, when a reviewer is trying to maximize their voucher value, they might be incentivized to choose either "Accept" or "Reject" over "Revision".

We begin the analysis by modeling the current reviewer reward system as a game. Recall from Section 2 that a game is defined by a triplet $(\mathcal{P},\mathcal{S},\mathcal{U})$, where $\mathcal{P}$ is the set of players, $\mathcal{S}$ is the strategy space, and $\mathcal{U}$ is the set of utility functions.

In this game, each reviewer independently chooses a decision $d\in\mathcal{D}$ based on their evaluation of the paper’s quality. The utility of a reviewer is determined by the voucher value they receive, which depends on the decision they make and the time at which they receive the voucher (refer to Assumption 2.3 and Remark 2.3).

An example of the payoff matrix of this game, denoted $\mathcal{M}_{c}$, is as follows:

where the first number in each cell is the utility of reviewer 1, the second number is the utility of reviewer 2, and $d_{t}$ represents the time discount due to the delay in voucher receipt in the case of a "Revision" decision.

For instance, to calculate $d_{t}$, we consider the daily interest rate $r_{d}$ and the number of days of revision $n_{d}$. Specifically, $d_{t}$ can be computed as $d_{t}=(1+r_{d})^{n_{d}}$. This formulation captures the impact of delayed voucher receipt on the utilities of both reviewers in the "Revision" decision scenario.

The reviewers aim to maximize their utility, which, in this game, is equivalent to receiving the highest possible voucher value in the shortest possible time. They do this by estimating the other reviewer’s decision based on their own assessment of the paper’s quality, and choosing a decision that they believe is most likely to be in agreement with the other reviewer’s decision. This is formalized in the following proposition:

We now turn our attention to the analysis of Nash equilibriums for the game $\mathcal{G}_{c}$ defined in Definition 3.1. Nash equilibriums, as defined in Definition 2.2, are strategy profiles from which no player has an incentive to unilaterally deviate, given the strategies of the other players.

Let’s denote the set of strategies that maximize the expected utility of reviewer $i$ given their estimated quality score $\hat{S}^{i}_{p}$ as $S^{i}(\hat{S}^{i}_{p})$. Reviewer $i$’s optimal strategy $s^{*}_{i}$ is then a member of this set.

Proof of this lemma follows directly from Assumption 3.1 and the definition of a Nash equilibrium (Definition 2.2). In particular, if $s^{*}_{i}\in S^{i}(\hat{S}^{i}_{p})$, then by definition, there does not exist a strategy $s_{i}\in S_{i}$ such that $U_{i}(s_{i},s^{*}_{-i})>U_{i}(s^{*}_{i},s^{*}_{-i})$, which means that $(s^{*}_{1},s^{*}_{2})$ satisfies the condition for a Nash equilibrium.

Thus, our analysis shows that under the current reviewer reward system, there exists a Nash equilibrium for each paper. However, the alignment of reviewer decisions in these equilibriums could potentially compromise the quality of the review process.

Having analyzed the current system, we now turn our attention to the proposed reviewer reward system, which we model as a strategic game.

Similar to the current system, each reviewer independently chooses a decision $d\in\mathcal{D}$ based on their evaluation of the paper’s quality. The main difference in the proposed system lies in the utility functions: the voucher value, which is the reviewers’ utility, varies not only with their decision and the timing of voucher receipt but also with the agreement between the reviewers’ decisions.

An example of the payoff matrix for this game, denoted $\mathcal{M}_{p}$, is given by:

where the first number in each cell is the utility of reviewer 1 and the second number is the utility of reviewer 2.

Given the structure of the payoff matrix $\mathcal{M}_{p}$, the reviewers are incentivized to make their decisions not only based on their own assessments but also with an eye towards agreement with the other reviewer. This leads to the following proposition:

The underlying principles in evaluating the quality of the paper remain the same as in the previous game, as detailed in Assumption 2.6 and Definition 2.9. However, given the modified payoff matrix $\mathcal{M}_{p}$, the strategy for each reviewer changes.

In the proposed system, the reward function is symmetric, i.e., $R(d_{1},d_{2})=R(d_{2},d_{1})$ for all $d_{1},d_{2}\in\mathcal{D}$. This symmetry in rewards encourages cooperation between reviewers, driving them to reach a consensus on the quality of the paper. The proposed system thus fosters a more collaborative and quality-focused review process.

This assumption adds another layer of complexity to the reviewers’ decision-making process. Under this assumption, reviewer $i$ must not only consider their own evaluation of the paper, but also estimate the decision of the other reviewer, $i^{\prime}$, based on their own quality assessment.

Given that reviewer $i$ only has the paper as a source of information, the most reasonable estimation they can make about $\hat{S}^{i^{\prime}}{p}$ is that it equals their own estimated quality score, $\hat{S}^{i}_{p}$. This leads to the assumption that $\hat{S}^{i^{\prime}}_{p}\simeq\hat{S}^{i}_{p}$. In other words, in the absence of any other information, reviewer $i$ assumes that the other reviewer’s evaluation of the paper aligns with their own.

This assumption holds some basis in reality. When a reviewer is evaluating a paper, they may naturally assume that another reviewer, with a similar level of expertise and understanding, would evaluate the paper similarly. Indeed, this may not always be the case due to differing biases and perspectives, but in the absence of other information, this serves as a reasonable assumption.

Under this assumption, each reviewer tries to estimate the other’s decision based on their own assessment of the paper. Consequently, this encourages reviewers to more closely scrutinize the paper, as their reward is now dependent on not only their own decision but also on accurately predicting the other reviewer’s decision.

Therefore, in the proposed system, we see that the incentives are designed in such a way as to encourage both accurate evaluation of the paper and coordination between reviewers. This could potentially lead to an increase in the quality of the reviews, and ultimately, of the papers accepted by the journal.

We start by considering the Nash Equilibrium of the proposed game $\mathcal{G}_{p}$.

This definition follows directly from the standard definition of Nash equilibrium. In particular, a strategy profile is a Nash equilibrium if no player can unilaterally improve their utility by deviating from their current strategy.

The analysis of the Nash equilibria in the proposed game $\mathcal{G}_{p}$ reveals that the game encourages reviewers to make accurate estimations of the quality of a paper and to choose their decisions accordingly.

The two agents are modelled as neural networks with a single input, three outputs, and a hidden layer of size 256. The input to each agent is the estimated quality score $\hat{S}^{i}_{p}$ of a paper, while the output represents the probability distribution over the set of possible decisions $\mathcal{D}=\{\text{Accept},\text{Revision},\text{Reject}\}$. The structure of the agents’ neural networks is depicted in Figure 1.

In addition, each neural network employs a series of ten residual modules [23] to facilitate the training process. These modules are designed to learn identity functions that allow the input signals to bypass some layers and reach deeper parts of the network directly, effectively addressing the issue of vanishing gradients commonly observed in deep neural networks [24].

The agents’ neural networks are trained using the Adam optimizer [25] with a learning rate of $0.00001$, and the training process lasts for $100,000$ epochs. The reward at each epoch is calculated based on whether the decisions made by the agents are in agreement and whether these decisions are correct. Specifically, the agents receive a reward of $1.0$ if they both make the correct decision (either accepting or rejecting the paper), and a reward of $1.0-d_{t}$ if they make different decisions or both decide to request a revision. The quantity $d_{t}$ is a time discount factor set to $0.2$, which represents the delay in voucher receipt for a "Revision" decision.

In order to ensure that the agents’ decisions are not solely based on the maximization of immediate reward, an auxiliary loss term is included in the loss function. This term is calculated as the cross-entropy between the agents’ decisions and the true decision, and serves to guide the agents towards making decisions that are consistent with the actual quality of the papers.

The quality score $S_{p}$ and the biases $b^{i}_{p}$ are randomly generated for each epoch. The quality score is drawn from a uniform distribution between $0$ and $1$, while the biases are drawn from a uniform distribution between $-0.05$ and $0.05$. The thresholds $Th_{a}$ and $Th_{r}$ for making "Accept" and "Revision" decisions are set to $0.66$ and $0.33$, respectively, in order to have the same probabilities of "Accept", "Revision", and "Reject" for a random paper, in accordance with Assumptions 2.6 and 2.7.

In order to observe the evolution of the agents’ decision-making process during the training, we record the moving average of the rewards received by the agents, as well as the distribution of their decisions. The moving average reward is calculated over a sliding window of $100$ epochs, and the decision distribution is computed by counting the number of times each pair of decisions is made by the agents.

The agents’ architectures and the simulation framework remain the same for the proposed reviewer reward system as in the current system, as detailed in Subsection 5.1. However, the crucial distinction lies in the reward function, which we redefine in this context to better incentivize the reviewers’ decision-making process.

We employ this newly formulated reward function in our simulation, replacing the former reward function detailed in Subsection 5.2. The loss function defined in Equation 5 remains unaltered for this experiment, as it provides a stable and consistent means of optimizing the agents’ actions.

The results of the simulations, as showcased in Figures 2 and 3, offer valuable insights into the performance of the current and proposed reviewer reward systems, respectively. In this subsection, we delve into these outcomes to distill the key findings and implications.

After a sequence of 20,000 epochs, the agents stabilize towards a Nash equilibrium, as discerned from the fact that the rewards settle around the value of $0.95$ and do not exhibit significant variations henceforth. The existence of Nash equilibria is predominantly evident in the pairings of (Accept, Accept) and (Reject, Reject), as illustrated in Figure 2. In particular, the (Accept, Accept) combination is realized with a frequency of $31.40\%$, and the (Reject, Reject) pairing manifests itself with a frequency of $37.20\%$. Conversely, the (Revision, Revision) pairing is less common, with a likelihood of $10.60\%$. This pattern indicates that under the current reward system, reviewers have a propensity to adopt a binary decision-making approach, favoring either Accept or Reject verdicts.

In stark contrast, the proposed reward system demonstrates a more balanced distribution of decision pairings. As evidenced in Figure 3, Nash equilibria can be found in all decision pairings, including (Accept, Accept), (Revision, Revision), and (Reject, Reject). In terms of percentages, these pairings are realized with probabilities of $31.00\%$, $28.20\%$, and $25.60\%$, respectively. Given the thresholds $Th_{a}$ and $Th_{r}$ set at $0.66$ and $0.33$ for "Accept" and "Revision" decisions, the ideal probabilities for the respective judgments ought to be around $33.33\%$. Consequently, the proposed reward system is more conducive to achieving this desirable state, demonstrating a more balanced approach to the review process compared to the current system.

We begin by observing the deviation of decision pairing frequencies from the idealized probabilities under both systems. For the current system, the deviation for (Accept, Accept) pairing, denoted as $\Delta_{AA}^{c}$, can be calculated as $|31.40\%-33.33\%|=1.93\%$, and similarly, $\Delta_{RR}^{c}=3.87\%$, and $\Delta_{RevRev}^{c}=22.73\%$. On the other hand, the proposed system exhibits the deviations as $\Delta_{AA}^{p}=2.33\%$, $\Delta_{RR}^{p}=7.73\%$, and $\Delta_{RevRev}^{p}=5.13\%$. The significant reduction in deviation for the (Revision, Revision) pairing from $22.73\%$ to $5.13\%$ suggests a more balanced distribution of decision pairings under the proposed system.

A deeper analysis of these deviations reveals a noteworthy trend. Let $\Delta_{max}^{c}=\max\{\Delta_{AA}^{c},\Delta_{RR}^{c},\Delta_{RevRev}^{c}\}$ and $\Delta_{max}^{p}=\max\{\Delta_{AA}^{p},\Delta_{RR}^{p},\Delta_{RevRev}^{p}\}$ be the maximum deviations under the current and proposed systems, respectively. We observe that $\Delta_{max}^{c}>\Delta_{max}^{p}$, indicating that the proposed system brings the decision pairings closer to the idealized probabilities.

Another crucial observation pertains to the stability of the Nash equilibrium. By investigating the standard deviations of the rewards, denoted as $\sigma^{c}$ and $\sigma^{p}$ for the current and proposed systems respectively, we find that $\sigma^{c}>\sigma^{p}$. This suggests that the rewards are more stable under the proposed system, implying a more stable Nash equilibrium.