Simpson’s Paradox and Lagging Progress in Completion Trends of Underrepresented Students in Computer Science

In comparing CS with other areas of study, we are interested not only in the gap between completions for students from underrepresented groups, but also in the trends over time.

Figure 1 depicts national level trends in CS completions as they compare to other areas of study (Non-CS). Figure 1(a) presents completion trends computed as absolute counts while Figure 1(b) completion trends with percentages. Absolute counts are computed as the sum of all completions for a given year and demographic group. It can be observed that when computed as absolute counts, completions across all underrepresented groups exhibit strong positive trends for all demographic groups in CS as well as other areas of study.

Used as a measure of participation or success, absolute counts do not lead to fair comparisons. In Figure 1(a), CS completions are compared to total completions for all other areas of study. The wide gap between the CS line (orange) and the other line (blue) is by definition; hence, the vertical axes of all panels in Figure 1(a) are log-scaled. Figure 1(b) presents completion trends as percentages, computed as the sum of completions by students from a demographic group in a given year divided by the sum of all completions that year. It is now the case in Figure 1(b) that the gap between the trend lines for the two groups is meaningful. It is also the case that the trend lines individually exhibit more interesting patterns in stark contrast to absolute counts in Figure 1(a).

When measured using absolute counts, CS exhibits positive trends for all three underrepresented groups as shown in Figure 1(a). However, when using relative counts, the CS trends weaken, and in the case of Black students, a reversal occurs. We conclude that even though CS is seeing an increase in the absolute count of completions overall, the share of underrepresented students in these completions is not increasing at the same rate, if at all. In fact, as shown in Table 1, the share of Black students in CS completions experienced a sharp decline $r_{CS\wedge Black}=-0.93$ during the same time when their share in other areas of study remained relatively stable $r_{other\wedge Black}=-0.06$.

National Trends. Figure 1(b) presents strong evidence of the large gap between CS and other areas of study when it comes to participation and success of students from underrepresented groups. For most underrepresented groups, these gaps are only becoming wider with time. This should be alarming to the CS community.

In 2011, women accounted for only 19% of the graduating class in CS compared to other areas of study (58%; Figure 1b). While representation of women in the graduating class remained relatively stable at 59% in non-CS fields in 2019, women accounted for only 21% of the graduating class in CS. For black students in CS, the trends are even more bleak. While CS has much work to do with female student representation, the trend in CS has greater positive strength than other areas of study ($r_{CS\wedge Women}=0.91>0.85=r_{other\wedge Women}$ summarized in Table 1). For black students, completions in CS regressed sharply from 10% to 8% of the graduating class. During the same period, the share of Black completions in other areas of study remained relatively stable at around 9% of the graduating class. For reference, 14.2% of the total US population is Black or African American alone, as per the US Census Bureau (Bureau, 2020).

For Hispanic or Latino students, trends in both CS and other areas of study are strongly positive, indicating good progress. However, the positive trend is stronger in other areas of study as compared to in CS $r_{other}=1.0>0.97=r_{CS}$ (Table 1). This indicates that CS is lagging behind and not able to keep up with other disciplines in its progress for Hispanic or Latino student representation and success. If these trends (albeit positive) persist, the gap between CS and other areas of study would only widen.

For Women, we observe in Figure 2 that the frequency distribution is a bell-shaped curve with the peak just to the right of zero. The mean difference (dashed line) for Women is $0.07$ indicating that for an average institution, the completion trends for Women is marginally better in CS than other areas of study. For Black students, the frequency distribution is similarly bell-shaped, but with a peak to the left of zero. However, the mean difference (dashed line) for Black students is $-0.06$ indicating that for an average institution, CS is lagging behind other disciplines with respect to completion trends of Black students.

For Hispanic or Latino students, the frequency distribution is also bell-shaped but the peak is far to the left of zero with mean difference of $-0.45$. This suggests that for an average institution in the United States, CS is lagging far behind other areas of study in terms of completions by Hispanic or Latino students. We can also observe this result in the rightmost panel of Figure 2 in which most of the distribution is blue representing the fact that $r_{CS}<r_{other}$ for an overwhelming majority of institutions.

In subsections 3.1 and 3.2, we compared completion trends in CS to other areas of study for Women, Black and Hispanic or Latino students. Our analysis considered data from a nationwide scale (dotted line in Figure 2) as well as at an institutional scale (dashed line in Figure 2). In some cases, we observe a congruence at both scales and for others, an alarming incongruence.

For Women, the trends at national and institutional scales are congruent: at the national level $r_{CS}-r_{other}=0.06$ whereas, for an average institution, $\mu_{r_{CS}-r_{other}}=0.07$, a marginal difference. However, for Black and Hispanic or Latino students, the trends observed at national level are incongruous with trends observed at institutional level. We informally observe this incongruity by noting the wide gaps between dotted and dashed lines in the center and right panels of Figure 2. Numerically, these gaps are also reflected in divergent national and institutional values in columns 3 and 4 of Table 1. These data are a clear example of Simpson’s Paradox: when a trend appears in several groups of data but disappears or reverses when the groups are combined.

It is of particular interest that the effect of this paradox on CS completion trends is different for Black students than for Hispanic or Latino students. For Black students in CS, the national trend is strongly negative ($r_{CS}-r_{other}=-0.87$) but at the institutional level, the average trend is relatively neutral ($\mu_{~{}r_{CS}-r_{other}}=-0.06$). In contrast, for Hispanic or Latino students, the completion trend observed at the national level is relatively flat ($r_{CS}-r_{other}=-0.03$) whereas the completion trend observed at the institutional level is strongly negative ($\mu_{~{}r_{CS}-r_{other}}=-0.45$).

For all three demographic groups in Figure 3, the majority of the institutions fall in the quadrant $1$ (top right) where $r_{CS}>0$ and $r_{other}>0$. The KDE distribution for Women is nearly uniformly distributed with peaks in the top-right of quadrant $1$. This aligns with the mean difference of $0.07$ for Women students observed in Figure 2. The region of the plot corresponding to $r_{CS}<-0.5$ is relatively sparse indicating that with respect to CS, very few institutions exhibit strong negative trends. This is consistent with Figure 1 where completions in other areas of study have stabilized at the saturation point of around $60\%$ but CS still has a lot of ground to cover.

For Black students (center panel), Figure 3 shows a near bimodal distribution with most institutions in either quadrant $1$ or quadrant $3$, with quadrants $2$ and $4$ being relatively sparse. This indicates that institutions with a positive trend in CS completions also exhibit a positive trend in other areas of study and vice versa. Reconciling the institutional-level scatter-KDE plot with the national level trends we observed in Figure 1(b) suggests that institutions with $r_{CS}<0$ for Black students must have a disproportionately large influence to cause a sharply declining trend at the national level. We consider this further in subsection 3.4.

For Hispanic or Latino students, the KDE plot in Figure 3 (right panel) is far more concentrated than the distributions for Women and Black students. This grouping of institutions in quadrants $1$ and $4$ means the variance along the $r_{CS}$ vertical axis is far greater than the variance along the $r_{other}$ horizontal axis. Thus, for Hispanic or Latino students, an overwhelming majority of institutions have a strong positive trend for completions in other areas of study while the trends of CS completions varies relatively more from institution to institution. Reconciling the institutional-level scatter-KDE plot with the national level trends we observed in Figure 2 suggests that institutions with $r_{CS}<0$ for Hispanic or Latino students must also have a disproportionately large influence to cause a positive trend at the national level but a neutral trend at the institutional level.

In the top panel of Figure 5 we observe that for every underrepresented demographic group, most institutions fall within quadrant $1$: $r_{CS}>0$ and $r_{other}>0$. It is therefore the case that most institutions exhibit a positive trend in both CS completions and completions in other areas of study. However, the bottom panel of Figure 5 shows that for Black students, the institutions where $r_{CS}<0$ (i.e., quadrants $3$ and $4$) also have the largest graduating classes. This explains why the national-level trends are so different from the institutional-level trends for Black students: the few institutions that have relatively large CS graduating classes are the ones that exhibit a negative trend in CS completions. This contrasts with most institutions that have relatively smaller graduating classes, but exhibit a positive trend in CS completions. The results are similar for Hispanic or Latino students in Figure 5 where $r_{CS}<0$ category institutions have a large number of total completions (graduates). However, because $r_{CS}>0\wedge r_{other}>0$ category institutions have the largest size for Hispanic or Latino students, the institution level trends are neutral or flat as opposed to negative for Black students.

Thus far, our discussion has focused on completion trends in CS and other areas of study that are strictly temporal. We augment our analyses with the added dimension of space to identify any spatiotemporal trends in completions for underrepresented students. Figure 6 shows hexagon plots over the United States for each demographic group color-coded by completion trends in CS and other areas of study. White hexagons correspond to areas of the United States with either no institution or institutions with no CS graduates belonging to the given demographic group. The color of each hexagon is determined by the mode of $r$-category of institutions within the region. The four categories along with their corresponding color are given in the color bars to the right and are consistent with the $r$-categories colors from Figure 5.

We observe that completion trends are spatially localized. For Women, negative trends in CS and other areas of study are observed away from the East and West coasts with a negative cluster near the Central West region. For Black students, the negative completion trends in CS and other areas of study seem to strongly concentrate in the South. In particular, we observe a large cluster of negative completion trends in the Southeast. These negative trends are alarming since the majority of the Black population in the United States lives in these regions, as per the US Census Bureau (Bureau, 2020). A strong but relatively smaller negative cluster can also be observed near Southern California and adjacent areas. For Hispanic or Latino, the US census (Bureau, 2020) shows that the population is concentrated in the Southwest but in contrast to Black students we do not observe any regions of concentrated negative trends.