Discrimination and Constraints: Evidence from The Voice

An international television singing competition franchise initially created by Dutch television producer John de Mol and singer Roel van Velzen. One hundred forty-five countries have adopted the format and begun airing their versions since 2010. The show’s format features five stages of competition: producers’ auditions, blind auditions, battle rounds, knockouts (since 2012), and live performance shows.

Each season employs a panel of four coaches who critique the artists’ performances and guide their teams of selected artists through the remainder of the season. They also compete to ensure that their act wins the competition, thus making them the winning coach. While these constraints mean that the assignment of coaches to the season is not literally random, the more relevant claim for this research design is an entirely arbitrary assignment of coaches to the artists.

Each season begins with the ”Blind Auditions,” where coaches form their team of artists they mentor through the remainder of the season. The coaches’ chairs are faced toward the audience during artists’ performances; those interested in an artist press their button, which turns their chair towards the artist and illuminates the bottom of the chair to read ”I want you.” Each coach has the length of the auditioner’s performance (about one minute) to decide if he or she wants that singer on his or her team; if two or more coaches want the same singer (as happens frequently), the singer has the final choice of coach. After the performance, an artist either defaults to the only coach who turned around or selects the coach if more than one coach expresses interest. The coaches not seeing the artists is a setting in which coaches’ encounters are demonstrably exogenous with respect to artists’ performances (White, 2022).

I used the blind audition part of the show as a natural experiment in this work. The data on a blind audition was compiled from historical Voice TV shows on Wikipedia. I web-scraped the results of the blind audition for four countries: the UK, France, Germany, and Australia. Each country held approximately ten seasons each year, starting from the year 2012. Each country’s Voice Wiki page is divided by season, each season is divided by the episodes, and the order of the performance splits each episode. The page includes the singer and coach’s information, the singer’s age, the order of performance, and the song.

From the Wiki page, I constructed the outcome variable based on the coaches who pressed the button if the coach chose the specific artist. Thus, I can build the result of the four coaches for each artist. Similarly, the page provides the actual order of the performance. Since the primary goal of the analysis is to identify gender-specific preferences, I needed to determine the gender of the artist.

I infer the gender of the artists based on their first names. I used an online application programming interface (API) called Genderize.io. The application has pre-trained machine learning algorithms that are used to predict gender. The API gives a confidence level in probabilistic terms and the exact name count in their already used datasets. Utilizing a threshold of 90% probability, I am able to attach a gender to approximately 97% of the whole dataset. For the remaining 3% of the first names, I hardcoded by checking the performance on the YouTube webpage. The gender of the coaches can easily be identified as all of them are well-established artists.

I also used the genre of the song the artist performs. Using song information from the Wikipedia page, I extracted the genre of the music from the Spotify API. The API gives multiple genres that correspond to the given song. By doing frequency distribution, I determined the most frequent genres and came up with the eleven unique genres.

Summary statistics are shown at the coach level in Table 1. Column (1) includes all coaches in the sample, while Columns (2) and (3) disaggregate to coaches who are female or male, respectively. The data has 11,972 observations, 58 unique coaches, and 3,009 unique artists. 31.6 percent of the coaches were female, though there is considerable heterogeneity across the team gender composition and failure rates for female and male artists. Similarly, about 46.3 percent of artists are men, providing decent statistical power for an own-gender bias analysis.

My primary analysis includes all the coaches’ choices, with approximately 40 percent of the artists being chosen in the blind audition round. Female artists are chosen about 21.6 percent of the time. The difference between female coaches and male coaches choosing female artists is a 1 percent difference. This pattern of having a 1% difference continues to hold for the male artists who are chosen 18 percent of the time. Thus, a naive comparison of the raw data for female and male gender would yield a conclusion of no gender bias. Since the preference of female and male artists across female and male coaches is similar, the selection difference lies at about one percent. To justify this premise, I examine the order of each performance at the artist level and how TV show-related specification affects decision-making.

Table 1 also provides summary statistics for TV show-related specifications, considering them heterogeneous in the analysis. Each coach and artist has a unique specification, such as the order of their performance, their preference towards a specific coach, and the genre of the song. Ultimately, 16.8 percent of the artists ended up on a team. From that team configuration, we can determine the gender composition of the team, which is the number of women and men on the team during the blind audition stage. The average difference between the number of males and females for each order on the team is -0.533, meaning that the number of female artists is 0.5 people higher on average compared with the male artists. Additional information is obtained when two or more coaches pursue the same artist, providing the artist with the final choice. I estimated the failure rate for these cases, and each coach has about a 0.27 failure rate for female artists and 0.24 for male artists. This is the rate at which a coach wanted an artist of a particular gender on their team, and the artist did not choose the coach. Both of these variables have considerable differences across the gender of the coach, which enriches our study to identify own-gender bias in the selection process.

As discussed and shown in the Section 1, the institutional setting provides for the exogenous assignment of gender encounters between coach and artist. Table 2 examines this exogeneity assumption by estimating differences in the observed characteristics of male and female artists. The first column shows mean characteristics for all artists, while the second and third columns present average outcomes for male and female artists. The fourth column reports the difference between the two means. Comparing overall raw means is consistent with the assignment procedure described above. There is no significant difference between male and female artists in any show-related specifications, such as being chosen by coaches, selecting a team, being chosen by a male coach, and the order of their performance. This supports our identification strategy: in a given show setting, there is little evidence to suggest that male artists’ performances systematically differ from those performed by female ones. However, I also find some statistically significant differences. For example, male artists are 5.2 percent more likely to perform pop genre songs. Similarly, the share of classic style and the share of rock style is lower, while the share of country style is higher among male artists. All these significant observables will be estimated in the setting as confounding factors.

The identification strategy used in this study is Difference in Differences with Dummy Outcomes (Price and Wolfers, 2010), which focuses on coach and artist gender interaction. Specifically, we estimate the following linear probability model:

where the outcome of interest, $Choice_{i,c}$ is the binary variable equal to one when the coach chooses the artist in performance, where $i$ denotes an artist selected by the coach $c$ . Similarly, $MaleCoach_{c}$ is one when the Coach is male and zero otherwise. $MaleArtist_{c}$ is one when the artist is male and 0 otherwise. The primary variable of interest is the interaction between male coaches and male contestants. $\beta_{2}$ captures the difference between male artists chosen by female coaches relative to female artists chosen by female coaches, allowing for artists to differ in their overall propensities to be selected. $\beta_{1}$ coefficient captures differences in selection rates of female artists by male coaches relative to female artists by female coaches, allowing coaches to differ in their overall propensities to select the artist. $\beta_{3}$ is the difference in differences coefficient of interest and indicates the degree of own-gender bias. Notably, the difference in differences estimator provides only a symmetric and relative measure of gender bias. So the coefficient captures the comparison of the difference in selection rates of female artists by male coaches relative to female artists by female coaches to the difference in selection rates of male artists by male coaches relative to male artists by female coaches. If there is no gender bias, then $\beta_{3}$ = 0. A positive sign indicates same-gender leniency (or other-gender harshness) in blind audition selection.

The richness of the data allows for extending this analysis to control for the various confounding factors such as contestant-specific characteristics and coach fixed effect, genre, and order of the performance. In the model, I include $Coach_{c}$ coach-specific fixed effect, which captures time-invariant coach characteristics. $X_{i,c}$ is the set of control variables at the performance level, including the genre of song, performance order, the number of male relative to female artists in the coach’s team, the coach’s failure rate for male and female artists during the audition as well as fixed effect for the country of the TV show. I also cluster standard errors for all estimations at the coach level, as this seems ex-ante to be the appropriate cluster for most autocorrelation concerns.

The regression model in the difference-in-differences framework does not capture potential heterogeneous effects across covariates. There could be a positive effect on some groups in the covariates but a negative or no effect on other groups. For example, the coaches in the initial stage of the blind audition could elicit own-gender bias, but in the later stage, opposite-gender bias. The standard regression model will provide a weighted average of effect size, even if an effect varies across a population represented by different groups. One plausible solution is to estimate heterogeneous treatment effects. Instead of just estimating one effect, I estimate a distribution of effect size for a given unit with a given set of attributes and what their impact might be. To analyze this heterogeneous effect of own-gender bias, I incorporated the first difference approach into the causal forest approach. I will discuss the performance of the proposed causal forest approach and how to apply it to my empirical example.

The causal forest approach is consistent and asymptotically Gaussian and provides an estimator for their asymptotic variance that allows us to construct valid confidence intervals (Athey and Wager, 2019). Causal forest utilizes a forest-based approach to calculate a similarity weight and then applies the local generalized method of moments to estimate based on a weighted set of neighbors. The authors also show that using the centered instead of the original outcome and treatment allows a forest to focus its splits on the features that affect the treatment effect instead of the primary outcome. Intuitively, a causal forest consists of regression trees (Athey and Imbens, 2016; Breiman et al., 1984), which estimate conditional average treatment effects (CATE) at the leaves of the trees instead of predicting the outcome variables.

I utilize a two-step approach by incorporating the first-difference approach into the causal forest approach, the similar two-step approach in the standard difference-in-difference method. In the first step, I estimate the first difference, which is the change in the selection of individual contestants from a male coach to a female coach. This step teases out the systematic differences between coaches in female and male coach groups. In the second step, I use the outcome variables obtained from the first step and apply the causal forest approach for heterogeneous treatment-effect analysis. This step teases out the effect of contestant gender trends that exist in both women and men.

My methods rely heavily on the causal forest approach, a machine learning method that can estimate heterogeneous causal effect functions under the above assumptions. The main strength associated with the causal forest is that it provides a strategy for learning patterns of heterogeneity from the data. This method requires little researcher discretion compared to traditional subgroup analyses. For instance, one does not have to decide which cutoffs to use to analyze heterogeneity in effects along continuous covariates, nor do we require to define the functional condition of the heterogeneity; the causal forest algorithm is designed to automate this search and find the most appropriate model that best characterizes the heterogeneous effect in the data (Bonander and Svensson, 2021).

My identification strategy is to leverage the exogenous assignment of coaches to the artist and use a difference-in-differences estimator across these coach-artist interactions to estimate the degree of gender bias in selection. The baseline difference in differences results shows that female coaches are more likely to choose male artists, and male coaches are more likely to select female artists, opposite gender bias (Figure 1). A simple calculation suggests an opposite gender bias of 4.2 percentage points.

In this paper, we examine whether the gender of the coach and artist matters when it comes to selecting the performance in the blind audition stage in the Voice TV show. Results provide strong evidence that gender matters and I show that coaches choose other gender artists 12 percent more when they hear similar performances in the audition. The results are robust with many variations, such as coach and artist-specific covariates. Under the interpretation of opposite-gender bias, the results would be due to male and/or female coaches being either too lenient to opposite-gender artists, being too tough on own-gender artists, or both.

Relatedly, a potential mechanism through which opposite-gender interaction cause statistically significant gender bias could be employers hiring for female-dominated occupations showing a preference for male applicants (Weichselbaumer, 2004; Protsch, 2021). However, Weichselbaumer (2004) and Protsch (2021)’s research also found that employers preferred male applicants in male-dominated occupations. Still, because of data limitations, they do not observe the gender of the recruiter. Others who do not observe what I observe in my work found that employers prefer male applicants in male-dominated occupations and female applicants in female-dominated occupations (Campero and Fernandez, 2018; Benard and Correll, 2010; Glick et al., 1988; Ridgeway and Correll, 2004). These works provide robust evidence of gender bias in hiring. Still, we cannot distinguish whether it is an opposite-gender or own-gender bias hiring since the gender of the decision-makers is unobservable or decisions are made collectively. In my setting, I observe the gender of the coach and the coach’s individual choice in selecting the artists. Therefore I incorporate this gender composition feature into my setting to address whether the coach changes their behavior based on the gender composition in the team. For example, If one coach’s team is dominated by one gender, would the coach shift behavior during the audition and select an artist from another gender, a minority gender?

I conduct a heterogeneity analysis assessing the effect of different gender compositions in the team on the likelihood of being selected. To do so, I took advantage of the fact that the sample I used includes both selection of male and female coaches in the blind audition stages and if the artist ended up being in the team of the selected coach. Additionally, the other advantage of the sample is substantial variations in gender composition in each team, including both male-dominated and female-dominated settings. Typically, we observe only one type of setting, where the firm is female-dominated or male-dominated. Leveraging this variation in the gender composition, I estimate the heterogeneous effect of gender bias for male and female coaches. I found that male artists have an advantage over female artists in a male-dominated team when the coach is male. Likewise, women have an advantage over men in a female-dominated team. When the coach is female, we can see opposite scenarios. Female coaches tend to choose women over men in male-dominated teams and men over women in the female-dominated team. This difference is partially statistically significant, but the magnitude remains consistent.

The literature provides robust support for a hypothesis on group formation dynamics, which is different for male and female leaders. One study of bias in other contexts suggests that men might not respond favorably to the presence of gender diversity, particularly in domains that men have historically dominated. Also, gender-related issues were most likely to be raised in groups where women are a minority or where the female is a leader (Crocker and McGraw, 1984). In addition, there is substantial evidence that women in team formation care about equality, men care about efficiency, and men consider male or female dominance in the team as more efficient. More directly, Kuhn and Villeval (2014) found that women respond differently to alternative rules for team formation in a manner consistent with advantageous inequity aversion. In contrast, men show greater responsiveness to efficiency gains associated with team production, meaning that when forming a team, men make their selections based on who will be the most efficient rather than on the grounds of gender equality. Overall, our findings indicate that gender composition and the gender of the decision-maker are important determinants in the hiring process. While it is difficult to know if these findings extend to other industries, these results corroborate the debate over gender discrimination in the hiring process.

In conclusion, the results add evidence to a growing literature testing own or opposite gender bias in decision-making. This finding provides insights into the extent to which situational factors work to mitigate or exacerbate gender disparities in the hiring process. I show strong evidence of the heterogeneity in gender bias from gender composition in the team. This evidence suggests that the selection process is primarily rooted in the female and male recruiters’ behavioral preferences of inequality or efficiency. In addition, the findings offer a new perspective to enrich past research on gender discrimination, shedding light on the instances of gender bias variation by the gender of the decision maker and team gender composition.