Gender Gap in Natural Language Processing Research:
Disparities in Authorship and Citations

Gender gaps are quantitative measures of the disparities in social, political, intellectual, cultural, or economic success due to one’s gender. Gender gaps can also refer to disparities in access to resources (such as healthcare and education), which in turn lead to disparities in success. We need to pay attention to gender gaps not only because they are inherently unfair but also because better gender balance leads to higher productivity, better health and well-being, greater economic benefits, better decision making, as well as political and economic stability Skjelsboek and Smith (2001); Woetzel et al. (2015); Hakura et al. (2016); Mehta et al. (2017); Gallego and Gutiérrez (2018).

Historically, gender has often been considered binary (male and female), immutable (cannot change), and physiological (mapped to biological sex). However, those views have been discredited Hyde et al. (2019); Richards et al. (2017); Darwin (2017); Lindsey (2015); Kessler and McKenna (1978). Gender is complex, and does not necessarily fall into binary male or female categories (e.g. nonbinary people), and also does not necessarily correspond to one’s assigned gender at birth.

Society has often viewed different gender groups differently, imposing unequal social and power structures Lindsey (2015). The World Economic Forum’s 2018 Global Gender Gap Report (which examined data from more than 144 countries) highlighted the gender gap between men and women in Artificial Intelligence as particularly alarming WEF (2018). It indicated that only 22% of the professionals in AI are women and that this low representation in a transformative field requires urgent action—otherwise, the AI gap has the potential to widen other gender gaps. Other studies have identified substantial gender gaps in science Håkanson (2005); Larivière et al. (2013); King et al. (2017); Andersen and Nielsen (2018).

Perez (2019) discusses, through numerous examples, how there is a considerable lack of disaggregated data for women and how that is directly leading to negative outcomes in all spheres of their lives, including health, income, safety, and the degree to which they succeed in their endeavors. This holds true even more for transgender people. Our work obtains disaggregated data for female Natural Language Processing (NLP) researchers and determines the degree of gender gap between female and male NLP researchers. (NLP is an interdisciplinary field that includes scholarly work on language and computation with influences from Artificial Intelligence, Computer Science, Linguistics, Psychology, and Social Sciences to name a few.) We hope future work will explore other gender gaps (e.g., between trans and cis people). Measuring gender gaps is a crucial step towards addressing them.

We examine tens of thousands of articles in the ACL Anthology (AA) (a digital repository of public domain NLP articles) for disparities in female authorship.¹¹1https://www.aclweb.org/anthology/ We also conduct experiments to determine whether female first authors are cited more or less than male first authors, using citation counts extracted from Google Scholar (GS).

We extracted and aligned information from the ACL Anthology and Google Scholar to create a dataset of tens of thousands of NLP papers and their citations as part of a broader project on analyzing NLP Literature.²²2Mohammad (2019) presents an overview of the many research directions pursued, using this data. Notably, Mohammad (2020a) explores questions such as: how well cited are papers of different types (journal articles, conference papers, demo papers, etc.)? how well cited are papers published in different time spans? how well cited are papers from different areas of research within NLP? etc. Mohammad (2020c) presents an interactive visualization tool that allows users to search for relevant related work in the ACL Anthology. We refer to this dataset as the NLP Scholar Dataset. We determined aggregate-level statistics for female and male researchers in the NLP Scholar dataset using an existing manually curated author–gender list as well as first names that are strongly associated with a gender.

Note that attempts to automatically infer gender of individuals can lead to harm Hamidi et al. (2018). Our work does not aim to infer gender of individual authors. We use name–gender association information to determine aggregate-level statistics for male and female researchers. Further, one may not know most researchers they cite, other than from reading their work. Thus perceived gender (from the name) can lead to unconscious effects, e.g., Dion et al. (2018) show that all male and mixed author teams cite fewer papers by female authors than all female teams. Further, seeing only a small number of female authors cited can demoralize young researchers entering the field.

We do not explore the reasons behind gender gaps. However, we will note that the reasons are often complex, intersectional, and difficult to disentangle. We hope that this work will increase awareness of gender gaps and inspire concrete steps to improve inclusiveness and fairness in research.

It should also be noted that even though this paper focuses on female–male disparities, there are many aspects to demographic diversity including: representation from transgender people; representation from various nationalities and race; representation by people who speak a diverse set of languages; diversity by income, age, physical abilities, etc. All of these factors impact the breadth of technologies we create, how useful they are, and whether they reach those that need it most.

Resources for the NLP Scholar project can be accessed through the project homepage.³³3http://saifmohammad.com/WebPages/nlpscholar.html

Pilcher (2017) shows that names function not only to identify individuals, but also to manage gender throughout one’s life. There is a strong cultural norm in various parts of the world to assign a first name to newborns as per their category of sex Pilcher (2017); Barry III and Harper (2014); Lieberson et al. (2000); Alford (1987). Pilcher (2017) argues that, throughout their life, the first name plays a role in repeatedly categorizing a person as being male or female. People may change their name and appearance to manage their gender Connell (2010); Pilcher (2017). People may choose a name that is not associated with male or female categorizations Connell (2010).

The strong normative tendency to use names to signal gender has led to a large body of work on automatically determining gender by one’s first name, not just for scientometric analysis discussed below, but also for language studies, social sciences, public health, and commerce. However, this can also lead to misgendering, which can cause significant pain and harm Hamidi et al. (2018). (Misgendering is when a person—or in this case, a machine—associates someone with a gender with which they do not identify.) Further, work that does not explicitly consider gender to be inclusive of trans people can reinforce stereotypes such as the dichotomy of gender.

We expect gender disparities to be different depending on the groups being compared: female–male, trans–cis, and so on. Our work does not aim to infer gender of individual authors. We obtain disaggregated statistics for women, specifically, to study the disparities between female and male NLP researchers. We discuss ethical considerations further in Section 6. See also Mihaljević et al. (2019) for a discussion on ethical considerations in using author name to estimate gender statistics in the Gender Gap in Science Project—a large ongoing project tracking gender gaps in Mathematical and Natural Sciences.⁴⁴4https://gender-gap-in-science.org

Most studies on gender and authorship have found substantial gender disparities in favor of male researchers. They include work on $\sim$1700 articles from journals of library and information science Håkanson (2005), on $\sim$12 million articles from the Web of Science (for Sociology, Political Science, Economics, Cardiology and Chemistry) Ghiasi et al. (2016); Andersen and Nielsen (2018), on $\sim$2 million mathematics articles Mihaljević-Brandt et al. (2016), on $\sim$1.6 million articles from PubMed life science and biomedical research Mishra et al. (2018), on $\sim$1.5 million articles from fifty disciplines published in JSTOR King et al. (2017), and on $\sim$0.5 million publications from US research universities Duch et al. (2012). There also exists some work that shows that in fields such as linguistics LSA (2017) and psychology Willyard (2011), female and male participation is either close to parity or tilted in favor of women.

In NLP research, Schluter (2018) showed that there are barriers in the paths of women researchers, delaying their attainment of mentorship status (as estimated through last author position in papers). Anderson et al. (2012) examine papers from 1980 to 2008 to track the ebb and flow of topics within NLP, and the influence of researchers from outside NLP on NLP. Vogel and Jurafsky (2012) examined about 13,000 papers from 1980 to 2008 to determine basic authorship statistics by women and men. Gender statistics were determined by a combination of automatic and manual means. The automatic method relied on lists of baby names from various languages. They found that female authorship has been steadily increasing from 1980 to 2008. Our work examines a much larger set of NLP papers (1965–2019), re-examines some of the questions raised in Vogel and Jurafsky (2012), and explores several new questions, especially on first author gender and disparities in citation.

We extracted and aligned information from the ACL Anthology (AA) and Google Scholar (GS) to create a dataset of tens of thousands of NLP papers and their citations. We aligned the information across AA and GS using the paper title, year of publication, and first author last name. Details about the dataset, as well as an analysis of the volume of research in NLP over the years, are available in Mohammad (2020b). We summarize key information below.

We use the datasets described in §3.1 (especially AA* and AA**) to answer a series of questions on female authorship. Since we do not have full self-reported information for all authors, these should be treated as estimates.

First author is a privileged position in the author list that is usually reserved for the researcher that has done the most work and writing.¹⁴¹⁴14A small number of papers have more than one first author. This work did not track that. In NLP, first authors are also often students. Thus we are especially interested in gender gaps that effect them. The last author position is often reserved for the most senior or mentoring researcher. We explore last author disparities only briefly (in Q1).

To put these numbers in context, the percentage of female scientists worldwide (considering all areas of research) has been estimated to be around 30%. The reported percentages for many computer science sub-fields are much lower.¹⁵¹⁵15https://unesdoc.unesco.org/ark:/48223/pf0000235155 The percentages are much higher for certain other fields such as psychology Willyard (2011) and linguistics LSA (2017).

To obtain further insights, we also repeat some of the experiments described above for unigrams in paper titles. We found that FFA rates are relatively high in non-English European language research such as papers on Russian, Portuguese, French, and Italian. FFA rates are also relatively high for work on prosody, readability, discourse, dialogue, paraphrasing, and individual parts of speech such as adjectives and verbs. FFA rates are particularly low for papers on theoretical aspects of statistical modelling, and for terms such as WMT, parsing, markov, recurrent, and discriminative.

Research articles can have impact in a number of ways—pushing the state of the art, answering crucial questions, finding practical solutions that directly help people, etc. However, individual measures of research impact are limited in scope—they measure only some kinds of contributions. The most commonly used metrics of research impact are derived from citations including: number of citations, average citations, h-index, and impact factor Bornmann and Daniel (2009). Despite their limitations, citation metrics have substantial impact on a researcher’s scientific career; often through a combination of funding, the ability to attract talented students and collaborators, job prospects, and other opportunities in the wider research community. Thus, disparities in citations (citation gaps) across demographic attributes such as gender, race, and location have direct real-world adverse implications. This often also results in the demoralization of researchers and marginalization of their work—thus negatively impacting the whole field.

Therefore, we examine gender disparities in citations in NLP. We use a subset of the 32,985 AA^′ papers (§3.2) that were published from 1965 to 2016 for the analysis (to allow for at least 2.5 years for the papers to collect citations). There are 26,949 such papers.

It is also interesting to note that the gender-unknown category has almost bridged the gap with the male category in terms of average citations. Further, the proportion of the gender-unknown authors has steadily increased over the years—arguably, an indication of better representation of authors from around the world in recent years.¹⁸¹⁸18Our method is expected to have a lower coverage of names from outside North America and Europe because USSA and PUBMED databases historically have had fewer names from outside North America and Europe.

If women chose to work in areas that happen to attract less citations by virtue of the area, then we would not expect to see citation gaps in so many areas. Recall also that we already showed that FFA% is not correlated with rank of popularity of an area (Q4). Thus it is unlikely that the choice of area of research is behind the gender gap.

Analysis focused on women and men leaves out non-binary people.¹⁹¹⁹19Note that as per widely cited definitions, nonbinary people are considered transgender, but most transgender people are not non-binary. Also, trans people often use a name that is more associated with their gender identity. Additionally, not disaggregating cis and trans people means that the statistics are largely reflective of the more populous cis class. We hope future work will obtain disaggregated information for various genders. However, careful attention must be paid as some gender classes might include too few NLP researchers to ensure anonymity even with aggregate-level analysis.

The use of female- and male-gender associated names to infer population level statistics for women and men excludes people that do not have such names and people from some cultures where names are not as strongly associated with gender.

Names are not immutable (they can be changed to indicate or not indicate gender) and people can choose to keep their birth name or change it (providing autonomy). However, changing names can be quite difficult. Also, names do not capture gender fluidity or contextual gender.

A more inclusive way of obtaining gender information is through self-reported surveys. However, challenges persist in terms of how to design effective and inclusive questionnaires Bauer et al. (2017); Group (2014). Further, even with self-report textboxes that give the respondent the primacy and autonomy to express gender, downstream research often ignores such data or combines information in ways beyond the control of the respondent.²⁰²⁰20https://reallifemag.com/counting-the-countless/ Also, as is the case here, it is not easy to obtain self-reported historical information.

Social category detection can potentially lead to harm, for example, depriving people of opportunities simply because of their race or gender. However, one can also see the benefits of NLP techniques and social category detection in public health (e.g., developing targeted initiatives to improve health outcomes of vulnerable populations), as well as in psychology and social science (e.g., to better understand the unique challenges of belonging to a social category).

A larger list of ethical considerations associated with the NLP Scholar project is available through the project webpage.²¹²¹21http://saifmohammad.com/WebPages/nlpscholar.html Mihaljević et al. (2019) also discusses the ethical considerations in using author names to infer gender statistics in the Gender Gap in Science Project.²²²²22https://gender-gap-in-science.org

We analyzed the ACL Anthology to show that only $\sim$30% have female authors, $\sim$29% have female first authors, and $\sim$25% have female last authors. Strikingly, even though some gains were made in the early years of NLP, overall FFA% has not improved since the mid 2000s. Even though there are some areas where FFA% is close to parity with male first authorship, most areas have a substantial gap in the numbers for male and female authorship. We found no correlation between popularity of research area and FFA%. We also showed how FFA% varied by paper type, venue, academic age, and area of research. We used citation counts extracted from Google Scholar to show that, on average, male first authors are cited markedly more than female first authors, even when controlling for experience and area of work. Thus, in NLP, gender gaps exist both in authorship and citations.

This paper did not explore the reasons behind the gender gaps. However, the inequities that impact the number of women pursuing scientific research Roos (2008); Foschi (2004); Buchmann (2009) and biases that impact citation patterns unfairly Brouns (2007); Feller (2004); Gupta et al. (2005) are well-documented. These factors play a substantial role in creating the gender gap, as opposed to differences in innate ability or differences in quality of work produced by these two genders. If anything, past research has shown that self-selection in the face of inequities and adversity leads to more competitive, capable, and confident cohorts Nekby et al. (2008); Hardies et al. (2013).

Many thanks to Rebecca Knowles, Ellen Riloff, Tara Small, Isar Nejadgholi, Dan Jurafsky, Rada Mihalcea, Isabelle Augenstein, Eric Joanis, Michael Strube, Shubhanshu Mishra, and Cyril Goutte for the tremendously helpful discussions. Many thanks to Cassidy Rae Henry, Luca Soldaini, Su Lin Blodgett, Graeme Hirst, Brendan T. O’Connor, Lucía Santamaría, Lyle Ungar, Emma Manning, and Peter Turney for discussions on the ethical considerations involved with this work.