An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process

To study how review statistics vary by subject, and control for topic distribution in the analysis below, we define a rough categorization of papers to common ML topics. To keep things simple and interpretable, we define a short list of hand curated keywords for each topic, and identify a paper with that topic if it contains at least one of the relevant keywords. The topics used were theory, computer vision, natural language processing, adversarial ML, generative modelling, meta-learning, fairness, generalization, optimization, graphs, Bayesian methods, and Other.¹¹1See Appendix A.1 for details on the keywords that we chose. In total, 1605 papers from 2020 fell into the above categories, and 772 out of 1605 papers fell into multiple categories.

The reproducibility of review outcomes has been a hot-button issue, and many researchers are concerned that reviews are highly random. This issue was brought to the forefront by the well-known “NIPS experiment” (Lawrence & Cortes, 2014), in which papers were reviewed by two different groups.²²2The NeurIPS conference was officially called NIPS prior to 2018. This experiment yielded the observation that 43% of papers accepted to the conference would be accepted again if the conference review process were repeated. In this section, we conduct an analysis to quantify the randomness in the review process for papers submitted to ICLR from 2017 to 2020. Our analysis is post hoc, unlike the controlled experiment from NIPS 2014.

Monte-Carlo simulation To produce an interpretable metric of reproducibilty that accounts for variation in both reviewers and area chairs, we use Monte-Carlo simulations to estimate the NIPS experiment metric: if an accepted paper were reviewed anew, would it be accepted a second time?

Our simulation samples from review scores from all 2560 papers submitted to ICLR in 2020. We simulate area chair randomness using a logistic regression model that predicts the AC’s accept/reject decision as a function of a paper’s mean reviewer score. This model is fit using 2020 paper data, and the goodness of fit of this model is explored in Appendix A.6.

To estimate reproducibility, we treat the empirical distribution of mean paper scores as if it were the true distribution of “endogenous” paper quality. We simulate a paper by (i) drawing a mean/endogenous score from this distribution. We then (ii) examine all 2020 papers with similar (within 1 unit) mean score, and compute the differences between the review scores and mean score for each such paper. We (iii) sample 3 of these differences at random, and add them to the endogenous score for the simulated paper to produce 3 simulated reviewer scores. We then (iv) use the simulated reviewer scores as inputs to the 2020 logistic regression model to predict whether the paper is accepted. Finally, we (v) generate a second set of reviewer scores using the same endogenous score, and use the logistic regression to see whether the paper is accepted a second time.

We run separate simulations for years 2017 through 2020. We observe a downward trend in reproducibility, with scores decreasing from 75% in 2017, to 70% in 2018 and 2019, to 66% in 2020.

Reproducibility scores by topic appear in Figure 10 of the Appendix. We observe that Computer Vision papers have the lowest reproducibility score with 59%, while Generalization papers have the highest score with 70%, followed by Theory papers with 68%.

Analysis of Variance  A classical method for quantifying randomness in the review process is analysis of variance (ANOVA). In this very simple generative model, (i) each paper has an endogenous quality score that is sampled from a Gaussian distribution with learned mean/variance, and (ii) reviewer scores are sampled from a Gaussian distribution centered around the endogenous score, but with a learned variance. The ANOVA method generates unbiased estimates for the mean/variance parameters in the above model. An ANOVA using all of scores in 2020 finds that endogenous paper quality has a standard deviation of 2.95, while review scores have deviation 1.72. This can be interpreted to mean that the variation between reviewers is roughly 60% as large as the variation among papers. This does not account for the randomness of the area chair. Note the Gaussian model is a fairly rough approximation of the actual score distribution (ANOVA assumptions as discussed in Appendix A.2). Nonetheless, we report ANOVA results because this model is widely used and easily interpretable. Simulations in which papers and scores are drawn from an ANOVA model agree closely with simulations using empirical distributions (Appendix A.2).

A number of recent conferences have attempted to mitigate the perceived randomness of accept/reject decisions by increasing the number of reviewers per paper. Most notably, efforts were made at NeuRIPS 2020 to ensure that most papers received 5 reviews. Unfortunately, simulations suggest that increasing the number of reviewers is not particularly effective at mitigating randomness.

We apply the above reproducibility model to review scores in ICLR 2020 while increasing the number of simulated reviews per paper, and we plot results in Figure 2. While reproducibility scores increase with more reviewers, gains are marginal; increasing the number of reviewers from 2 to 5 boosts reproducibility by just 3%. Note that as the number of reviewers goes to infinity, the reproducibility score never exceeds 75%. This limiting behavior occurs because area chairs, who make final accept/reject decisions, vary in their standards for accepting papers. This uncertainty is captured by a logistic regression model in our simulation. As more reviewers are added, the high level of disagreement among area chairs remains constant, while the standard error in mean scores falls slowly. The result is paltry gains in reproducibility.

Our study suggests that program chairs should avoid assigning large numbers of reviewers per paper as a tactic to improve the review process. Rather, it is likely most beneficial to use a small number of reviews per paper (which gives reviewers more time for each review), and add ad-hoc reviewers to selected papers in cases where first-round reviews are uninformative.

Much of the conversation around the review process focuses on reproducibility and randomness. However, a perfectly reproducible process can still be quite bad if it reproducibly rejects high-impact and transformative papers. It is thought by some that the ML review process favors non-controversial papers with incremental theoretical results over papers with big new ideas.

In this section we analyze whether reviewer scores correlate with paper impact. We measure impact using citation rate, calculated by dividing citation count by the number of days since the paper was first published online. Though average scores for papers are roughly normally distributed, the distribution of citation rates is highly right-skewed and heavy tailed, with some papers receiving thousands of citations and others receiving none. To mitigate this issue, we use Spearman’s rank correlation coefficient as a metric to measure the strength of the nonlinear relationship between average review score and citation rate.

We assign each paper a “citation rank” (the number of papers with lower citation rate) and “score rank” (the number of papers with lower mean reviewer score), then calculate Spearman’s rank correlation coefficient between citation rank and score rank for all submitted papers. At first glance, the Spearman correlation (0.46, $p<0.001$) seems to indicate a moderate relationship between scores and citation impact. However, we find that most of this trend is easily explained by the fact that higher scoring papers are accepted to the conference, and gain citations from the public exposure at ICLR. This trend is clearly seen in Figure 3, in which there is a sharp divide between the citation rates of accepted and rejected papers.

To remove the effect of conference exposure, we calculate Spearman coefficients separately on accepted posters and rejected/retracted papers, obtaining Spearman correlations of 0.17 and 0.22, respectively. While these small correlations are still statistically significant ($p<0.001$) due to the large number of papers being analyzed, the effects are quite small; an inspection of Figure 3 does not reveal any visible relationship between scores and citation rates within each paper category.

Spotlight papers had a Spearman correlation of -0.043 ($p=0.66,n=107$), and oral papers had Spearman correlation -0.0064 ($p=0.97,n=48$).

It is likely that papers that appeared on the web long before the ICLR deadline were previously rejected from another conference. We find that papers that have been online for more than 3 months at the time of the ICLR submission date, in fact, have higher acceptance rates at ICLR as well as higher average scores. This fact is observed in each of the past 4 years (see Figure 4(a)). As these papers are likely to have been previously rejected, our findings suggest that resubmission should not be discouraged, as these papers are more likely to be accepted than fresh submissions.

Does ICLR give certain institutions preferential treatment? In this section, we see whether the rank of institution (as measured by CS Rankings) influences decisions. We focus on academic institutions because they fall within a widely known quantitative ranking system. We found that 85% of papers across all years (87% in 2020) had at least one academic author. In the event that a paper has authors from multiple institutions, we replicate the paper and assign a copy of it to each institution. Figure 11 in Appendix A.5 plots the average of all paper scores received by each institution, ordered by institution rank. We find no meaningful trend between institution rank and paper scores.

Area Chair Bias  It is difficult to detect institutional bias in reviews because higher scores among more prestigious institutions could be explained by either bias or differences in paper quality. It is easier to study institutional bias at the AC level, as we can analyze the accept/reject decisions an AC makes while controlling for paper scores (which serves as a proxy for paper quality).

To study institutional bias at the AC level, we use a logistic regression model to predict paper acceptance as a function of (i) the average reviewer score, and (ii) an indicator variable for whether a paper came from one of the top 10 most highly ranked institutions.

We found that, even after controlling for reviewer scores, being a top ten institution leads to a boost in the likelihood of getting accepted. This boost is equivalent to a 0.15 point increase in the average reviewer score ($p=0.028$).³³3We compute this effect by taking the ratio of the indicator coefficient and the average reviewer score coefficient, which represents the relative impact on the likelihood of a paper being accepted. Regression details can be found in Table 3 located in Appendix A.7.

Prestigious institutions are more likely than others to have famous researchers, and the acceptance bias could be attributed to the reputation of the researcher rather than the university. In Section 5.3 we control for last-author reputation and still find a significant effect for institution rank (Table 1).

We find that most of the institutional bias effect is explained by 3 universities: Carnegie Mellon, MIT, and Cornell. We repeat the same study, but instead of a top 10 indicator, we use a separate indicator for each top 10 school. Most institutions have large p-values, but CMU, MIT, and Cornell each have statistically significant boosts equating to 0.31, 0.31, and 0.58 points in the average reviewer score, respectively. See Tables 4 and 5 in Appendix A.7.

While 2020 area chairs came from a diverse range of universities, 26/115 of them had a Google or Deepmind affiliation, which lead us to investigate bias effects for large companies (Table 6, A.7). The 303 papers from Google/Deepmind have an insignificant boost (0.007 points, $p=0.932$) as did the 110 papers from Facebook (0.07 points, $p=0.630$). However, the 92 papers from Microsoft has a strong penalty (-0.50 points, $p=0.003$).

Is it better to post a paper on arXiv before the review process begins, or to stay anonymous? To explore the effects of arXiv, we introduce a new indicator variable that detects if a paper was posted online by a date one week after the ICLR submission deadline. We fit logistic regression models to predict paper acceptance as a function of (i) the average score given by the reviewers of a paper, and (ii) an indicator variable for papers that were visible on arXiv during the review process.

We fit models separately on 3 subsets of the 2020 papers: schools not in the top 10 (Table 8, A.7), top 10 schools excluding CMU/MIT/Cornell (Table 10, A.7), and papers from CMU, MIT, and Cornell (Table 9, A.7). We find that by having papers visible on arXiv during the review process, CMU, MIT, and Cornell as a group receive a boost of 0.67 points ($p=0.001$). Conversely, institutions not in the top 10 received a statistically insignificant boost (0.08 points, $p=0.235$), as did top 10 schools excluding CMU, MIT, and Cornell (0.05 points, $p=0.737$).

Overall, we found that papers appearing on arXiv tended to do better (after controlling for reviewer scores) across the board, including schools that did not make the top 50, in addition to Google, Facebook, and Microsoft. However, these latter effects are not statistically significant.

Does the reputation of an author impact the review process? We quantify the reputation of an author by computing log(citations/papers+1) for each author, where citations is the total citations for the author, and papers is their total number of papers (as indexed by Semantic Scholar). The raw ratio citations/papers has a highly skewed and heavy tailed distribution, and we find that the log transform of the ratio has a symmetric and nearly Gaussian distribution for last authors (see Figure 13, Appendix A.6).

There is a well-known achievement gap between women and men in the sciences. Women in engineering disciplines tend to have their papers accepted to high impact venues, and yet receive fewer citations (Ghiasi et al., 2015; Larivière et al., 2013; Lariviere, 2014; Rossiter, 1993). These citation disparities exist among senior researchers at ICLR. Over their whole career, male last authors from 2020 have on average 44 citations per publication while women have 33. This gap in total citations can be attributed in part to differences in author seniority; the 2019 Taulbee survey reports that women are more severely under-represented among senior faculty (15.7%) than among junior faculty (23.9%) (Zweben & Bizot, 2020), and more senior faculty have had more time to accumulate citations. To remove the effect of time, we focus in on only 2020 ICLR papers, where male last authors received on average 4.73 citations to date compared to 4.16 citations among women. This disparity flips for first authors; male first authors received just 4.4 citations vs 6.2 among women.

We find that women are under-represented at ICLR relative to their representation in computer science as a whole. In 2019, women made up 23.2% of all computer science PhD students (Zweben & Bizot, 2020) in the US. However, only 10.6% of publications at ICLR 2020 have a female first author. While women make up 22.6% of CS faculty in the US, they make up only 9.7% of last authorships at ICLR. Despite this, women first authors have a 13.3% chance of publishing with a women last author, which suggests that junior women researchers either prefer to be supervised by women, or achieve higher productivity when supervised by women. Similar trends occur in the experimental sciences (Gaule & Piacentini, 2018; Pezzoni et al., 2016; Kato & Song, 2018). Among papers from top-10 schools, women make up 12.3% of first authors and 13.4% of last authors.

As noted in Section 1, there is a wide range of acceptance rates among the different research topics at ICLR. Among papers that fall into a labeled topic (excluding other), male and female first author papers are accepted with rates 27.6% and 22.1%, respectively. One may suspect that this disparity is explained by differences in the research topic distribution of men and women, as shown in Figure 6(b). However, we find that differences in topic distribution are actually more favorable to women than men. If women kept their same topic distribution, but had the same per-topic acceptance rates as their male colleagues, we should expect women to achieve a rate of 27.8% among topic papers (slightly more than their male colleagues) – a number far greater than the 22.1% they actually achieve.

We test for area chair bias with a regression predicting paper acceptance as a function of review scores and gender. We did not find evidence for AC bias (Figure 11, Appendix A.7).